Robotic control method and debris picking system for sorting debris from coal

Abstract

This application provides a robotic arm control method and a debris picking system for sorting debris in coal. The method includes: acquiring a first detection frame, a first segmentation mask, a target direction, and a first distance; filtering the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask; determining a first target quantity, stacking complexity, debris aspect ratio, and a second distance; determining a first threshold; calculating the distance of each debris based on the first segmentation mask and determining a second threshold based on the distance of each debris; and determining a robotic arm gripping mode to grip the debris based on the first target quantity, second target quantity, debris aspect ratio, second threshold, stacking complexity, first threshold, first distance, and second distance. This method solves the technical problems of low efficiency and poor accuracy in existing automatic sorting technologies.

Description

Technical Field

[0001] This invention relates to the field of mineral processing technology, and more specifically, to a robotic arm control method, robotic arm control device, computer-readable storage medium, and debris picking system for sorting impurities in coal. Background Technology

[0002] In the field of coal, gangue, and debris sorting, multi-robot systems are specialized equipment designed specifically for coal and gangue sorting. Depending on the robot structure, multi-robot systems can be categorized into articulated, Cartesian coordinate, and parallel types. These systems are primarily responsible for handling the sorting of multiple dynamic targets during coal, gangue, and debris sorting to prevent large pieces of gangue or hard debris from entering subsequent washing processes and causing equipment failure. Currently, there is a lack of fast, accurate, and reliable multi-target, multi-robotic arm collaborative sorting strategies in coal, gangue, and debris sorting. In existing online identification, positioning, and sorting processes, the sorting robot needs to assign tasks based on the detection results of each piece of debris. However, when geological conditions or working face conditions change, leading to an increase in hard debris in the coal, a one-to-one debris sorting strategy becomes inefficient and fails to meet production demands. Therefore, a robotic arm picking strategy for common debris in coal is proposed, which can effectively improve the sorting effect of various common debris in coal under complex scenarios. Summary of the Invention

[0003] The main objective of this application is to provide a robotic arm control method, robotic arm control device, computer-readable storage medium, and debris picking system for sorting debris in coal, so as to at least solve the problems of low efficiency and poor accuracy in the prior art of automatic sorting.

[0004] To achieve the above objectives, according to one aspect of this application, a robotic arm control method for sorting impurities in coal is provided. The method includes: acquiring a first detection box, a first segmentation mask, a target direction, and a first distance; wherein the first detection box is a target detection box corresponding to the impurity in a detection image; the first segmentation mask is an instance segmentation mask corresponding to the impurity; the first distance is the minimum safe working distance between the robotic arms; and the target direction is the coal flow direction; filtering the first detection box and the first segmentation mask based on the target direction to obtain a second detection box and a second segmentation mask; and determining a first target quantity based at least on the second detection box and the second segmentation mask. The criteria include: stacking complexity, clutter aspect ratio, and a second distance, where the second distance is the minimum distance between the clutter objects; the first target quantity is the number of clutter objects; the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same direction as the target and overlap with the image border; a first threshold is determined based on the first distance; the distance between each clutter object is calculated based on the first segmentation mask; and a second threshold is determined based on the distance between each clutter object. The first threshold is a stacking complexity threshold, and the second threshold is a clutter aspect ratio threshold; a second target quantity is obtained, and if the first target quantity is greater than the second target quantity and the clutter aspect ratio is greater than the second threshold... In the following situations, the robotic arm is controlled to grasp in a two-handed grasping mode based on the first distance, and the second target quantity is the total number of robotic arms; when the first target quantity is greater than the second target quantity and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a one-handed grasping mode based on the first distance and the second distance; when the first target quantity is less than or equal to the second target quantity and the stacking complexity is greater than the first threshold, if the third target quantity is less than or equal to the second target quantity, the first threshold is adjusted until the third target quantity is greater than the second target quantity, and if the third target quantity is greater than the second target quantity, the first threshold is adjusted until the third target quantity is greater than the second target quantity. The second target quantity is determined by controlling the robotic arm to grasp objects in either the two-handed grasping mode or the one-handed grasping mode, based at least on the ratio of the fourth target quantity to the second target quantity, the first distance, and the second distance. If the first target quantity is less than or equal to the second target quantity and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp objects in a two-handed grasping mode based on the first distance and the second distance. If the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp objects in a one-handed grasping mode based on the first distance and the second distance.

[0005] Optionally, determining the stacking complexity based on the second detection box and the second segmentation mask includes: determining the corresponding geometric center based on the second detection box to obtain a first target point; determining the corresponding coordinates based on the first target point to obtain first target coordinates; determining the intersection-union ratio (IU) of any one of the second detection boxes with other second detection boxes to obtain multiple target IU ratios; determining the minimum distance based on the first target coordinates corresponding to the second detection box and the first target coordinates corresponding to other second detection boxes to obtain a third distance; substituting the target IU ratios and the third distance corresponding to each first target point into a first target formula to obtain multiple candidate stacking complexities; and determining the maximum value of the multiple candidate stacking complexities as the stacking complexity.

[0006] Optionally, determining the aspect ratio of the debris based at least on the second detection frame and the second segmentation mask includes: calculating the area of ​​the debris based on the second detection frame to obtain a first area; calculating the area of ​​the debris based on the second segmentation mask to obtain a second area; obtaining a target length and a target width, wherein the target length is the length of the second detection frame and the target width is the width of the second detection frame; and substituting the first area, the second area, the target length, and the target width into a second target formula to obtain the aspect ratio of the debris.

[0007] Optionally, determining the second distance based at least on the second detection box and the second segmentation mask includes: calculating the zeroth moment and the first moment of the corresponding debris according to the second segmentation mask to obtain a first target moment and a second target moment; substituting the first target moment and the second target moment into the second target formula to determine the centroid coordinates of the debris to obtain second target coordinates; determining a third detection box according to the second target coordinates and the second detection box, wherein the third detection box is the second detection box with the smallest distance to the second target coordinates; determining a third target coordinate and a fourth target coordinate based on the first target coordinate of the third detection box, wherein the third target coordinate and the fourth target coordinate are the centroid coordinates of the two debris objects with the smallest distance to the first target coordinate; calculating the set distance based on the second target coordinate and the third target coordinate and the fourth target coordinate respectively, and determining the minimum value as the second distance.

[0008] Optionally, when the number of the first target is greater than the number of the second target and the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp in a two-hand grasping mode based on the first distance, including: extracting the target skeleton using a skeleton extraction algorithm based on the second segmentation mask; determining the point with the smallest distance between the target skeleton and the second target coordinates as the second target point; drawing a circle with the second target point as the center and the first distance as the diameter, and determining the intersection of the circle and the target skeleton as the grasping point; connecting the grasping points, and determining the direction perpendicular to the line as the grasping direction; controlling the robotic arm to grasp based on the grasping points and the grasping direction.

[0009] Optionally, controlling the robotic arm to grasp in a single-handed grasping mode based on the first distance and the second distance includes: when the first distance is greater than or equal to the second distance, drawing a circle with the second target coordinate as the center and the first distance as the diameter, determining any intersection point of the circle with the target skeleton as a grasping point, connecting the grasping points with a line, determining the direction perpendicular to the line as the grasping direction, and controlling the robotic arm to grasp based on the grasping point and the grasping direction; when the first distance is less than the second distance, drawing a circle with the third target coordinate or the fourth target coordinate as the center and the first distance as the diameter, determining any intersection point of the circle with the target skeleton as a grasping point, connecting the grasping points with a line, determining the direction perpendicular to the line as the grasping direction, and controlling the robotic arm to grasp based on the grasping point and the grasping direction.

[0010] Optionally, if the number of the third target is greater than the number of the second target, the robotic arm is controlled to grasp in either the two-handed grasping mode or the one-handed grasping mode based at least on the ratio of the number of the fourth target to the number of the second target, the first distance, and the second distance. This includes: if the ratio of the first target data to the number of the second target is less than a third threshold, and if the debris is a hard debris, the robotic arm is controlled to grasp in the two-handed grasping mode based on the first distance; if the debris is not a hard debris, the robotic arm is controlled to grasp in the one-handed grasping mode based on the first distance and the second distance. If the ratio of the first target data to the number of the second target is greater than or equal to the third threshold, and if the debris is a hard debris and the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp in the two-handed grasping mode based on the first distance; if the debris is a hard debris and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in the one-handed grasping mode based on the first distance and the second distance; if the debris is not a hard debris, no grasping is performed.

[0011] According to another aspect of this application, a robotic arm control device for sorting impurities in coal is provided. The device includes: a first acquisition unit, configured to acquire a first detection frame, a first segmentation mask, a target direction, and a first distance, wherein the first detection frame is a target detection frame corresponding to the impurity in a detection image, the first segmentation mask is an instance segmentation mask corresponding to the impurity, the first distance is the minimum safe working distance between the robotic arms, and the target direction is the coal flow direction; and a first calculation unit, configured to filter the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, and to determine at least the second detection frame and the second segmentation mask a first target quantity and a stacking complexity. The system comprises: a noise level, a noise aspect ratio, and a second distance, wherein the second distance is the minimum distance between the noise objects; the first target quantity is the number of noise objects; the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same direction as the target and overlap with the image border; a second calculation unit, configured to determine a first threshold based on the first distance, calculate the distance of each noise object based on the first segmentation mask, and determine a second threshold based on the distance of each noise object, wherein the first threshold is a stacking complexity threshold and the second threshold is a noise aspect ratio threshold; and a second acquisition unit, configured to acquire the second target quantity, wherein the first target quantity is greater than the second target quantity and the noise aspect ratio is greater than the second threshold. In this case, the robotic arm is controlled to grasp in a two-handed grasping mode according to the first distance, and the second target number is the total number of robotic arms; the first control unit is used to control the robotic arm to grasp in a one-handed grasping mode according to the first distance and the second distance when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold; the second control unit is used to adjust the first threshold until the third target number is greater than the second target number when the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number. The quantity is greater than the second target quantity. The third control unit is configured to control the robotic arm to grasp in either the two-handed grasping mode or the one-handed grasping mode, based on the ratio of the fourth target quantity to the second target quantity, the first distance, and the second distance, if the first target quantity is less than or equal to the second target quantity and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, to control the robotic arm to grasp in a two-handed grasping mode based on the first distance and the second distance; and if the aspect ratio of the debris is less than or equal to the second threshold, to control the robotic arm to grasp in a one-handed grasping mode based on the first distance and the second distance.

[0012] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the execution of any of the methods described in the device where the computer-readable storage medium is located.

[0013] According to another aspect of this application, a miscellaneous goods picking system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any one of the methods described.

[0014] Applying the technical solution of this application, in the above-mentioned robotic arm control method for selecting impurities in coal, the method includes: firstly, acquiring a first detection box, a first segmentation mask, a target direction, and a first distance, wherein the first detection box is the target detection box corresponding to the impurity in the detection image, the first segmentation mask is the instance segmentation mask corresponding to the impurity, the first distance is the minimum safe working distance between the robotic arms, and the target direction is the coal flow direction; then, filtering the first detection box and the first segmentation mask based on the target direction to obtain a second detection box and a second segmentation mask, and determining at least the first target quantity, stacking complexity, and other parameters based on the second detection box and the second segmentation mask. The clutter aspect ratio and a second distance are defined, where the second distance is the minimum distance between the clutter objects, the first target quantity is the number of clutter objects, and the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same direction as the target and overlap with the image border. Then, a first threshold is determined based on the first distance, the distance between each clutter object is calculated based on the first segmentation mask, and a second threshold is determined based on the distance between each clutter object. The first threshold is a stacking complexity threshold, and the second threshold is a clutter aspect ratio threshold. Next, the second target quantity is obtained. If the first target quantity is greater than the second target quantity and the clutter aspect ratio is greater than the second threshold, then... The robotic arm is controlled to grasp objects in a two-handed grasping mode based on the first distance mentioned above, and the second target quantity is the total number of robotic arms. Then, if the first target quantity is greater than the second target quantity and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp objects in a one-handed grasping mode based on the first and second distances. Then, if the first target quantity is less than or equal to the second target quantity and the stacking complexity is greater than the first threshold, and if the third target quantity is less than or equal to the second target quantity, the first threshold is adjusted until the third target quantity is greater than the second target quantity. If the third target quantity is greater than the second target quantity, the robotic arm is controlled to grasp objects in a one-handed grasping mode based on the first and second distances. The number of two targets is determined by at least the ratio of the number of the fourth target to the number of the second target, the first distance, and the second distance, controlling the robotic arm to grasp in either a two-handed grasping mode or a one-handed grasping mode. Finally, if the number of the first target is less than or equal to the number of the second target and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp in a two-handed grasping mode based on the first distance and the second distance; if the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a one-handed grasping mode based on the first distance and the second distance.This application filters and removes redundant results based on the detection results of the category and location of debris. Then, based on the debris, the shape of the debris and scene constraints, such as the number of debris, the number of robotic arms, and the scene stacking situation, it determines whether to adopt a single-hand gripping strategy or a two-hand gripping strategy. This enables the automatic screening of debris that has a significant impact on subsequent work processes, and solves the technical problems of low efficiency and poor accuracy in automatic sorting in the prior art. Attached Figure Description

[0015] Figure 1 A hardware block diagram of a mobile terminal for a robotic arm control method for sorting impurities in coal, provided in an embodiment of this application, is shown.

[0016] Figure 2 A flowchart illustrating a robotic arm control method for sorting impurities in coal, according to an embodiment of this application, is shown.

[0017] Figure 3 A structural block diagram of a robotic arm control device for sorting impurities in coal, according to an embodiment of this application, is shown.

[0018] The above figures include the following reference numerals:

[0019] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] As described in the background section, existing technologies cannot effectively sort out impurities in coal due to changes in geological conditions during coal mining and technical adjustments to the working plane. To address the issues of low efficiency and poor precision in automatic sorting, embodiments of this application provide a robotic arm control method, a robotic arm control device, a computer-readable storage medium, and an impurity sorting system for selecting impurities in coal.

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0025] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a robotic arm control method for sorting impurities in coal, according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0026] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the device information display method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0027] This embodiment provides a robotic arm control method for sorting impurities in coal, which runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0028] Figure 2 This is a flowchart of a robotic arm control method for sorting impurities in coal according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0029] Step S201: Obtain a first detection box, a first segmentation mask, a target direction, and a first distance. The first detection box is the target detection box corresponding to the debris in the detection image. The first segmentation mask is the instance segmentation mask corresponding to the debris. The first distance is the minimum safe working distance between the robotic arms. The target direction is the coal flow direction.

[0030] Specifically, the debris sorting system inputs the target detection box corresponding to the debris in the detection image, the instance segmentation mask corresponding to the debris, and the given direction of coal flow movement. Minimum safe distance L between the robot and the robotic arm MINParameters such as these.

[0031] Step S202: Based on the target direction, the first detection box and the first segmentation mask are filtered to obtain a second detection box and a second segmentation mask. At least based on the second detection box and the second segmentation mask, the number of first targets, stacking complexity, aspect ratio of clutter, and second distance are determined. The second distance is the minimum distance between the clutter, the number of first targets is the number of clutter, and the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same target direction and overlap with the image border.

[0032] Specifically, based on the coal flow direction, the target detection boxes and instance segmentation masks corresponding to the debris in the detection image are filtered. Instance segmentation masks and corresponding detection boxes corresponding to debris that intersect with the upper and lower edges of the debris detection image along the coal flow direction are removed. The removed results are returned, resulting in the second detection box and the second segmentation mask. Each detection result is iterated to determine the number of debris N, i.e., the number of the first target, the stacking complexity C, the aspect ratio F of the debris, and the minimum distance l between debris. m That is, the second distance mentioned above.

[0033] Step S203: Determine a first threshold based on the first distance, calculate the distance of each of the above-mentioned objects based on the first segmentation mask, and determine a second threshold based on the distance of each of the above-mentioned objects. The first threshold is a stacking complexity threshold, and the second threshold is a clutter aspect ratio threshold.

[0034] Specifically, the backbone f(x,y) of the debris segmentation mask is extracted using a backbone extraction algorithm, and then combined with the sorting scenario debugging results and the number of available robotic arms N. MANIPULATOR The stacking complexity threshold C is determined based on the minimum safe working distance between the robotic arms. THRESHOLD The distance between detected debris is calculated based on the instance segmentation mask corresponding to the debris, and the aspect ratio threshold F of the debris is determined based on the distance between the debris. THRESHOLD .

[0035] Step S204: Obtain the second target quantity. If the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, control the robotic arm to grasp in a two-hand grasping mode according to the first distance. The second target quantity is the total number of robotic arms.

[0036] Specifically, obtain the current total number N of robotic arms. MANIPULATOR Determine whether the number N of detected debris is greater than the total number N of the current robotic arms. MANIPULATOR When the amount of debris N is greater than the number of available robotic arms N MANIPULATOR That is, N>N MANIPULATORIn cases where the number of robotic arms is insufficient, a priority picking strategy is adopted, and it is determined whether the aspect ratio F of the debris is greater than the aforementioned second threshold F. THRESHOLD If F > F THRESHOLD If so, a "dual robotic arm object grasping strategy" is adopted. Based on the minimum safe working distance between the robotic arms, the robotic arms are controlled to grasp the object in a two-hand grasping mode. The index number, grasping point, and downward direction of the object are returned to the computer to complete the instruction registration.

[0037] Step S205: When the number of the first target is greater than the number of the second target and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance.

[0038] Specifically, when the amount of debris N is greater than the number of available robotic arms N... MANIPULATOR That is, N>N MANIPULATOR In cases where the number of robotic arms is insufficient, a priority picking strategy is adopted, and it is determined whether the aspect ratio F of the debris is greater than the aforementioned second threshold F. THRESHOLD If F≤F THRESHOLD If the "single robotic arm object grasping strategy" is adopted, the robotic arm is controlled to grasp the object in a two-hand grasping mode according to the minimum safe working distance between the robotic arms and the minimum distance between the objects. The index number of the object, the grasping point (loc), and the downward direction (drc) are returned to the computer to complete the instruction registration.

[0039] Step S206: If the number of the first target is less than or equal to the number of the second target and the stacking complexity is greater than the first threshold, and if the number of the third target is less than or equal to the number of the second target, adjust the first threshold until the number of the third target is greater than the number of the second target. If the number of the third target is greater than the number of the second target, control the robotic arm to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the number of the fourth target to the number of the second target, the first distance and the second distance.

[0040] Specifically, when the number of miscellaneous items N is less than or equal to the number of available robotic arms N MANIPULATOR That is, N>N MANIPULATOR A priority picking strategy is adopted based on the sufficient number of robotic arms, and the stacking complexity is greater than the first threshold, i.e., C>C. THRESHOLD In the case of high-stack scenarios, a debris sorting strategy is adopted. If the number of debris N in the aforementioned third target stacking area is... ′ The total number N of robotic arms is less than or equal to MANIPULATOR N ′ ≤N MANIPULATOR Then increase the preset stacking complexity threshold C.THRESHOLD Until the condition is met, the number of miscellaneous items in the stacked area N is reached. ′ Greater than the number of robotic arms N MANIPULATOR Then, the sorting robot arm is readjusted. If the number of debris N in the stacking area of ​​the third target quantity is... ′ Greater than the total number N of robotic arms MANIPULATOR N ′ ≤N MANIPULATOR If the number of debris is reassessed, a "single robot grabbing strategy" or a "dual robot grabbing strategy" is adopted based on the ratio of the number of debris to the number of available robot arms, the minimum safe working distance between robot arms, and the minimum distance between debris. The robot arms are controlled to grab debris in a two-handed or one-handed grabbing mode, and the index number (index), grabbing point (loc), and downward direction (drc) of the debris are returned to the computer to complete the instruction registration.

[0041] Step S207: If the number of the first target is less than or equal to the number of the second target and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp in a two-hand grasping mode according to the first distance and the second distance. If the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a one-hand grasping mode according to the first distance and the second distance.

[0042] Specifically, when the number of miscellaneous items N is less than or equal to the number of available robotic arms N MANIPULATOR Furthermore, the stacking complexity is less than or equal to the first threshold, i.e., C ≤ C THRESHOLD In the case of [missing information], a low-stack scenario debris sorting strategy is adopted. If the aspect ratio of the aforementioned debris is greater than the second threshold mentioned above, i.e., F>F[missing information], [missing information] will be used. THRESHOLD In the case of a dual-manipulator object grasping strategy, the manipulators are controlled to grasp objects in a two-hand grasping mode based on the minimum safe working distance between the manipulators and the minimum distance between the objects. If the aspect ratio of the object is less than or equal to the second threshold, i.e., F≤F THRESHOLD In this case, a single-handed robotic arm debris-grabbing strategy is implemented, controlling the robotic arm to grasp debris in a single-handed grasping mode based on the minimum safe working distance between robotic arms and the minimum distance between debris.

[0043] In this embodiment, firstly, a first detection box, a first segmentation mask, a target direction, and a first distance are obtained. The first detection box is the target detection box corresponding to the debris in the detection image. The first segmentation mask is the instance segmentation mask corresponding to the debris. The first distance is the minimum safe working distance between the robotic arms. The target direction is the coal flow direction. Then, based on the target direction, the first detection box and the first segmentation mask are filtered to obtain a second detection box and a second segmentation mask. At least based on the second detection box and the second segmentation mask, the first target quantity, stacking complexity, debris aspect ratio, and second distance are determined. The second distance is the distance between the debris. The minimum distance is determined by the following criteria: the number of the first target is the number of clutter objects; the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same direction as the target and overlap with the image border; then, a first threshold is determined based on the first distance, the distance of each clutter object is calculated based on the first segmentation mask, and a second threshold is determined based on the distance of each clutter object, where the first threshold is a stacking complexity threshold and the second threshold is a clutter aspect ratio threshold; then, the number of the second target is obtained, and if the number of the first target is greater than the number of the second target and the aspect ratio of the clutter is greater than the second threshold, the robotic arm is controlled to move in a dual-directional manner based on the first distance. The robotic arm performs a grasping operation in a hand-grabbing mode, where the number of the second target is the total number of robotic arms. Then, if the number of the first target is greater than the number of the second target and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to perform a single-hand grasping operation based on the first and second distances. Next, if the number of the first target is less than or equal to the number of the second target and the stacking complexity is greater than the first threshold, and if the number of the third target is less than or equal to the number of the second target, the first threshold is adjusted until the number of the third target is greater than the number of the second target. If the number of the third target is greater than the number of the second target, at least... Based on the ratio of the number of the fourth target to the number of the second target, the first distance, and the second distance, the robotic arm is controlled to grasp in either a two-handed grasping mode or a one-handed grasping mode. Finally, if the number of the first target is less than or equal to the number of the second target and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp in a two-handed grasping mode based on the first distance and the second distance; if the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a one-handed grasping mode based on the first distance and the second distance.This application filters and removes redundant results based on the detection results of the category and location of debris. Then, based on the debris, the shape of the debris and scene constraints, such as the number of debris, the number of robotic arms, and the scene stacking situation, it determines whether to adopt a single-hand gripping strategy or a two-hand gripping strategy. This enables the automatic screening of debris that has a significant impact on subsequent work processes, and solves the technical problems of low efficiency and poor accuracy in automatic sorting in the prior art.

[0044] To determine the stacking complexity, in one optional implementation, step S202 above includes:

[0045] Step S20201: Determine the corresponding geometric center based on the second detection frame to obtain the first target point; determine the corresponding coordinates based on the first target point to obtain the first target coordinates.

[0046] Specifically, the debris x is calculated based on the second detection frame described above. a Find the geometric center of the detection box, return the geometric center position (i,j), obtain the first target point, determine the coordinates of its corresponding position, and obtain the coordinates of the first target.

[0047] Step S20202: Determine the intersection-union ratio (IUU) of any one of the second detection boxes with other second detection boxes to obtain multiple target IUUs; determine the minimum distance based on the coordinates of the first target corresponding to the second detection box and the coordinates of the first target corresponding to other second detection boxes to obtain the third distance.

[0048] Specifically, based on any miscellaneous object x a The second detection frame described above determines the intersection-union ratio (IoU) of the detection frames of other debris. (i,j) The intersection-over-union ratio (CUI) of multiple target objects within the object detection frame is obtained, based on an object x. a The coordinates of the geometric center position (i,j) of the detection frame and the geometric center position coordinates of other debris detection frames are used to calculate the minimum distance between the center points of the debris. Determine the third distance mentioned above.

[0049] Step S20203: Substitute the target intersection-union ratio and the third distance corresponding to each of the first target points into the first target formula to obtain multiple alternative stacking complexities;

[0050] Specifically, based on the target intersection-union ratio (IoU) corresponding to the first target point of each debris... (i,j) and the third distance Using the first objective formula Calculate the stacking complexity of each item with other items.

[0051] Step S20204: The maximum value of the multiple candidate stacking complexities is determined as the stacking complexity.

[0052] Specifically, the stacking complexity of each miscellaneous item with other miscellaneous items is calculated separately, and the highest stacking complexity among the miscellaneous items is determined as the aforementioned stacking complexity.

[0053] To determine the aspect ratio of the debris, in one optional implementation, step S202 includes:

[0054] Step S20211: Calculate the area of ​​the debris based on the second detection frame to obtain a first area; calculate the area of ​​the debris based on the second segmentation mask to obtain a second area.

[0055] Specifically, the first area A is calculated based on the second detection frame described above. box The area A of the debris is calculated based on the second segmentation mask mentioned above. mask .

[0056] Step S20212: Obtain the target length and target width, where the target length is the length of the second detection box and the target width is the width of the second detection box.

[0057] Specifically, the target length h used to calculate the aspect ratio of the debris is determined based on the length and width of the second detection frame. box and target width w box .

[0058] Step S20213: Substitute the first area, the second area, the target length, and the target width into the second target formula to obtain the aspect ratio of the debris.

[0059] Specifically, according to miscellaneous items x a Calculate the shape factor from the detection bounding box and segmentation mask results. The aspect ratio of the above-mentioned miscellaneous items was obtained.

[0060] In order to determine the second distance, in one alternative implementation, step S202 above includes:

[0061] Step S20221: Calculate the zero-order moment and first-order moment of the corresponding debris according to the second segmentation mask to obtain the first target moment and the second target moment;

[0062] Specifically, based on the selected second segmentation mask, the zeroth and first moments of the irregular debris are calculated. The zeroth moment is obtained through M. 00 =∑ I ∑ J V(i,j) is calculated, where V(i,j) is the gray value of point (i,j), to obtain the first target moment. The first moment is obtained through M. 10 =∑ I ∑ J i·V(i,j), M01 =∑ I ∑ J The second objective moment is obtained by calculating j·V(i,j), where i and j are the x and y coordinates of each pixel, respectively. This definition is essentially the sum of the products of the x and y coordinates of all pixels and the pixel value.

[0063] Step S20222: Substitute the first target moment and the second target moment into the second target formula to determine the center of gravity coordinates of the debris and obtain the second target coordinates;

[0064] Specifically, using the above-mentioned second objective formula Calculate the centroid coordinates of the segmentation mask for irregular debris instances to obtain the coordinates of the second target.

[0065] Step S20223: Determine a third detection box based on the second target coordinates and the second detection box. The third detection box is the second detection box that has the smallest distance to the second target coordinates.

[0066] Specifically, based on the centroid coordinates of the irregular debris calculated according to the second objective formula, a third detection box with the smallest distance from the second coordinate is determined, which is used to determine information about other debris.

[0067] Step S20224: Based on the first target coordinates of the third detection frame, determine the third target coordinates and the fourth target coordinates. The third target coordinates and the fourth target coordinates are the centroid coordinates of the two objects that are closest to the first target coordinates.

[0068] Specifically, the centroid coordinates of the two other objects with the smallest distance are determined based on the third detection frame and the first target coordinates of the object.

[0069] Step S20225: Calculate the set distance based on the second target coordinates, the third target coordinates, and the fourth target coordinates, and determine the minimum value as the second distance.

[0070] Specifically, take the distance x from the clutter. a The two objects x corresponding to the geometric center of the nearest detection box m x n Information. Next, the problem is transformed into finding a point (g) in analytic geometry. i ,g j The problem involves finding the distances to the other two points, and then calculating the corresponding distances for each point.

[0071] In order to determine the gripping pattern of the robotic arm, in one optional implementation, step S204 above includes:

[0072] Step S2041: Extract the target backbone using the backbone extraction algorithm based on the second segmentation mask described above;

[0073] Specifically, take the miscellaneous items x a The centroid (i,j) is used to mark the boundary of the image according to the second segmentation mask mentioned above. Then, redundant points are removed from the marked boundary. The backbone extraction algorithm extracts the backbone f(x,y) of the segmentation mask to obtain the target backbone.

[0074] Step S2042: Determine the point with the smallest distance between the above target backbone and the above second target coordinates as the second target point;

[0075] Specifically, the target backbone f(x,y) is calculated to the level of the debris x. a The point closest to the centroid (i,j) is denoted as (i nearest ,j nearest ), which was determined as the second target point.

[0076] Step S2043: Draw a circle with the second target point as the center and the first distance as the diameter, and determine the intersection of the circle and the target skeleton as the grab point;

[0077] Specifically, with the aforementioned second target point (i) nearest ,j nearest L is the minimum safe working distance between robotic arms, centered at 0. MIN Draw a circle with diameter f(x,y) and intersect it at points p1 and p2, or at only one point p. Choose any point as the gripping point of a single robotic arm.

[0078] Step S2044: Connect the above-mentioned grabbing points with lines, and determine the direction perpendicular to the lines as the grabbing direction;

[0079] Specifically, connecting the above intersection points p1 and p2, we obtain the line l. p1,p2 The perpendicular direction of the connecting line is determined as the gripping direction of the robotic arm.

[0080] Step S2045: Control the robotic arm to perform grasping based on the above-mentioned grasping point and grasping direction.

[0081] Specifically, the robotic arm is controlled to perform grasping in either a single-handed or double-handed grasping mode based on the grasping point and grasping direction.

[0082] In order to determine the gripping pattern of the robotic arm, in one optional implementation, step S205 above includes:

[0083] Step S2051: When the first distance is greater than or equal to the second distance, draw a circle with the second target coordinates as the center and the first distance as the diameter, and determine any intersection point of the circle with the target skeleton as the gripping point. Connect the gripping points with a line, and determine the direction perpendicular to the line as the gripping direction. Control the robot arm to perform gripping based on the gripping points and the gripping direction.

[0084] Specifically, determining the minimum safe working distance L between robotic arms. MIN Is it greater than the minimum distance between the debris? Minimum safe working distance L between robotic arms MIN Greater than or equal to the minimum distance between the debris In the case of the aforementioned debris's center of gravity coordinates as the center, the minimum safe working distance L between the robotic arms is... MIN Draw a circle with diameter f(x,y) as the diameter. Select any point where the circle intersects the target skeleton f(x,y) as the gripping point of a single robot arm. Connect the intersection points to obtain a line. Determine the gripping direction of the robot arm as the direction perpendicular to the line.

[0085] Step S2052: When the first distance is less than the second distance, draw a circle with the third or fourth target coordinates as the center and the first distance as the diameter. Determine any intersection point of the circle with the target skeleton as the gripping point. Connect the gripping points with a line and determine the direction perpendicular to the line as the gripping direction. Control the robotic arm to perform gripping based on the gripping points and the gripping direction.

[0086] Specifically, the minimum safe working distance L between robotic arms MIN Less than the minimum distance between the debris In the case of the aforementioned third target coordinates or the aforementioned fourth target coordinates as the center, the minimum safe working distance L between the robotic arms is... MIN Draw a circle with diameter f(x,y) as the diameter. Select any point where the circle intersects the target skeleton f(x,y) as the gripping point of a single robot arm. Connect the intersection points to obtain a line. Determine the gripping direction of the robot arm as the direction perpendicular to the line.

[0087] In order to determine the gripping pattern of the robotic arm, in one optional implementation, step S206 above includes:

[0088] Step S2061: If the ratio of the first target data to the second target quantity is less than the third threshold, if the debris is a hard debris, the robotic arm is controlled to grasp it in the two-hand grasping mode according to the first distance; if the debris is not a hard debris, the robotic arm is controlled to grasp it in the one-hand grasping mode according to the first distance and the second distance.

[0089] Specifically, if the ratio of the number of debris to the total number of robotic arms is less than a third threshold, and if the debris is hard, a dual-robotic arm debris grasping strategy is adopted. Based on the minimum safe working distance between the robotic arms, the robotic arms are controlled to grasp the debris in the dual-arm grasping mode. If the debris is not hard, a single-robotic arm debris grasping strategy is adopted. Based on the minimum safe working distance between the robotic arms and the distance between the debris, the robotic arms are controlled to grasp the debris in the single-arm grasping mode.

[0090] Step S2062: If the ratio of the first target data to the second target quantity is greater than or equal to the third threshold, and if the debris is a hard debris and the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp it in the two-hand grasping mode according to the first distance. If the debris is a hard debris and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp it in the one-hand grasping mode according to the first distance and the second distance. If the debris is not a hard debris, it is not grasped.

[0091] Specifically, if the ratio of the number of debris to the total number of robotic arms is greater than or equal to the third threshold, and if the debris is hard and its aspect ratio is greater than the second threshold F>F... THRESHOLD If the debris is hard and its aspect ratio is less than or equal to the second threshold F≤F, then a dual-manipulator debris-grabbing strategy is adopted. Based on the minimum safe working distance between the manipulators, the manipulators are controlled to grasp the debris in the aforementioned two-hand grasping mode. THRESHOLD If the single robotic arm is not a hard object, it will not be grasped and will directly move to the next object.

[0092] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0093] This application also provides a robotic arm control device for sorting impurities in coal. It should be noted that the robotic arm control device for sorting impurities in coal in this application embodiment can be used to execute the robotic arm control method for sorting impurities in coal provided in this application embodiment. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0094] The following describes the robotic arm control device for sorting impurities in coal provided in the embodiments of this application.

[0095] Figure 3 This is a structural block diagram of a robotic arm control device for sorting impurities in coal according to an embodiment of this application. Figure 3 As shown, the device includes:

[0096] The first acquisition unit 10 is used to acquire a first detection box, a first segmentation mask, a target direction, and a first distance. The first detection box is the target detection box corresponding to the debris in the detection image. The first segmentation mask is the instance segmentation mask corresponding to the debris. The first distance is the minimum safe working distance between the robotic arms. The target direction is the coal flow direction.

[0097] Specifically, the debris sorting system inputs the target detection box corresponding to the debris in the detection image, the instance segmentation mask corresponding to the debris, and the given direction of coal flow movement. Minimum safe distance L between the robot and the robotic arm MIN Parameters such as these.

[0098] The first calculation unit 20 is used to filter the first detection box and the first segmentation mask based on the target direction to obtain a second detection box and a second segmentation mask, and to determine the number of first targets, stacking complexity, aspect ratio of clutter and second distance based at least on the second detection box and the second segmentation mask. The second distance is the minimum distance between the clutter, the number of first targets is the number of clutter, and the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same target direction and overlap with the image border.

[0099] Specifically, based on the coal flow direction, the target detection boxes and instance segmentation masks corresponding to the debris in the detection image are filtered. Instance segmentation masks and corresponding detection boxes corresponding to debris that intersect with the upper and lower edges of the debris detection image along the coal flow direction are removed. The removed results are returned, resulting in the second detection box and the second segmentation mask. Each detection result is iterated to determine the number of debris N, i.e., the number of the first target, the stacking complexity C, the aspect ratio F of the debris, and the minimum distance l between debris. m That is, the second distance mentioned above.

[0100] The second calculation unit 30 is used to determine a first threshold based on the first distance, calculate the distance of each of the above-mentioned objects based on the first segmentation mask, and determine a second threshold based on the distance of each of the above-mentioned objects. The first threshold is a stacking complexity threshold, and the second threshold is a clutter aspect ratio threshold.

[0101] Specifically, the backbone f(x,y) of the debris segmentation mask is extracted using a backbone extraction algorithm, and then combined with the sorting scenario debugging results and the number of available robotic arms N. MANIPULATOR The stacking complexity threshold C is determined based on the minimum safe working distance between the robotic arms. THRESHOLD The distance between detected debris is calculated based on the instance segmentation mask corresponding to the debris, and the aspect ratio threshold F of the debris is determined based on the distance between the debris. THRESHOLD .

[0102] The second acquisition unit 40 is used to acquire the second target quantity. When the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, the robot arm is controlled to grasp in a two-hand grasping mode according to the first distance. The second target quantity is the total number of the robot arm.

[0103] Specifically, obtain the current total number N of robotic arms. MANIPULATOR Determine whether the number N of detected debris is greater than the total number N of the current robotic arms. MANIPULATOR When the amount of debris N is greater than the number of available robotic arms N MANIPULATOR That is, N>N MANIPULATOR In cases where the number of robotic arms is insufficient, a priority picking strategy is adopted, and it is determined whether the aspect ratio F of the debris is greater than the aforementioned second threshold F. THRESHOLD If F > F THRESHOLD If so, a "dual robotic arm object grasping strategy" is adopted. Based on the minimum safe working distance between the robotic arms, the robotic arms are controlled to grasp the object in a two-hand grasping mode. The index number, grasping point, and downward direction of the object are returned to the computer to complete the instruction registration.

[0104] The first control unit 50 is configured to control the robotic arm to perform a single-handed grasping mode based on the first distance and the second distance when the number of the first target is greater than the number of the second target and the aspect ratio of the debris is less than or equal to the second threshold.

[0105] Specifically, when the amount of debris N is greater than the number of available robotic arms N... MANIPULATOR That is, N>N MANIPULATOR In cases where the number of robotic arms is insufficient, a priority picking strategy is adopted, and it is determined whether the aspect ratio F of the debris is greater than the aforementioned second threshold F. THRESHOLD If F≤F THRESHOLD If the "single robotic arm object grasping strategy" is adopted, the robotic arm is controlled to grasp the object in a two-hand grasping mode according to the minimum safe working distance between the robotic arms and the minimum distance between the objects. The index number of the object, the grasping point (loc), and the downward direction (drc) are returned to the computer to complete the instruction registration.

[0106] The second control unit 60 is configured to, when the number of the first target is less than or equal to the number of the second target and the stacking complexity is greater than the first threshold, adjust the first threshold until the number of the third target is greater than the number of the second target, and if the number of the third target is greater than the number of the second target, control the robotic arm to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the number of the fourth target to the number of the second target, the first distance and the second distance;

[0107] Specifically, when the number of miscellaneous items N is less than or equal to the number of available robotic arms N MANIPULATOR That is, N>N MANIPULATOR A priority picking strategy is adopted based on the sufficient number of robotic arms, and the stacking complexity is greater than the first threshold, i.e., C>C. THRESHOLD In the case of high-stack scenarios, a debris sorting strategy is adopted. If the number of debris N in the aforementioned third target stacking area is... ′ The total number N of robotic arms is less than or equal to MANIPULATOR N ′ ≤N MANIPULATOR Then increase the preset stacking complexity threshold C. THRESHOLD Until the condition is met, the number of miscellaneous items in the stacked area N is reached. ′ Greater than the number of robotic arms N MANIPULATOR Then, the sorting robot arm is readjusted. If the number of debris N in the stacking area of ​​the third target quantity is... ′ Greater than the total number N of robotic arms MANIPULATOR N ′ ≤N MANIPULATORIf the number of debris is reassessed, a "single robot grabbing strategy" or a "dual robot grabbing strategy" is adopted based on the ratio of the number of debris to the number of available robot arms, the minimum safe working distance between robot arms, and the minimum distance between debris. The robot arms are controlled to grab debris in a two-handed or one-handed grabbing mode, and the index number (index), grabbing point (loc), and downward direction (drc) of the debris are returned to the computer to complete the instruction registration.

[0108] The third control unit 70 is configured to, when the number of the first target is less than or equal to the number of the second target and the stacking complexity is less than or equal to the first threshold, and when the aspect ratio of the debris is greater than the second threshold, control the robotic arm to grasp in a two-handed grasping mode based on the first distance and the second distance; and when the aspect ratio of the debris is less than or equal to the second threshold, control the robotic arm to grasp in a one-handed grasping mode based on the first distance and the second distance.

[0109] Specifically, when the number of miscellaneous items N is less than or equal to the number of available robotic arms N MANIPULATOR Furthermore, the stacking complexity is less than or equal to the first threshold, i.e., C ≤ C THRESHOLD In the case of [missing information], a low-stack scenario debris sorting strategy is adopted. If the aspect ratio of the aforementioned debris is greater than the second threshold mentioned above, i.e., F>F[missing information], [missing information] will be used. THRESHOLD In the case of a dual-manipulator object grasping strategy, the manipulators are controlled to grasp objects in a two-hand grasping mode based on the minimum safe working distance between the manipulators and the minimum distance between the objects. If the aspect ratio of the object is less than or equal to the second threshold, i.e., F≤F THRESHOLD In this case, a single-handed robotic arm debris-grabbing strategy is implemented, controlling the robotic arm to grasp debris in a single-handed grasping mode based on the minimum safe working distance between robotic arms and the minimum distance between debris.

[0110] In this embodiment, the first acquisition unit acquires a first detection box, a first segmentation mask, a target direction, and a first distance. The first detection box is the target detection box corresponding to the debris in the detection image. The first segmentation mask is the instance segmentation mask corresponding to the debris. The first distance is the minimum safe working distance between the robotic arms. The target direction is the coal flow direction. The first calculation unit filters the first detection box and the first segmentation mask based on the target direction to obtain a second detection box and a second segmentation mask. At least based on the second detection box and the second segmentation mask, it determines the first target quantity, stacking complexity, debris aspect ratio, and second distance. The second distance is the distance between the debris and the target target. The minimum distance between objects is determined by the first target quantity, which is the number of objects. The second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same direction as the target and overlap with the image border. The second calculation unit determines a first threshold based on the first distance, calculates the distance of each object based on the first segmentation mask, and determines a second threshold based on the distance of each object. The first threshold is a stacking complexity threshold, and the second threshold is a clutter aspect ratio threshold. The second acquisition unit acquires the second target quantity. If the first target quantity is greater than the second target quantity and the clutter aspect ratio is greater than the second threshold, the robot arm is controlled based on the first distance. The robotic arm performs a two-handed grasping motion, where the second target quantity is the total number of robotic arms. If the first target quantity is greater than the second target quantity and the aspect ratio of the debris is less than or equal to the second threshold, the first control unit controls the robotic arm to perform a one-handed grasping motion based on the first and second distances. If the first target quantity is less than or equal to the second target quantity and the stacking complexity is greater than the first threshold, and the third target quantity is less than or equal to the second target quantity, the second control unit adjusts the first threshold until the third target quantity is greater than the second target quantity. The third control unit controls the robotic arm to grasp objects in either a two-handed grasping mode or a one-handed grasping mode, based at least on the ratio of the number of the fourth target to the number of the second target, the first distance, and the second distance. If the number of the first target is less than or equal to the number of the second target and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the control unit controls the robotic arm to grasp objects in a two-handed grasping mode based on the first distance and the second distance. If the aspect ratio of the debris is less than or equal to the second threshold, the control unit controls the robotic arm to grasp objects in a one-handed grasping mode based on the first distance and the second distance.This application filters and removes redundant results based on the detection results of the category and location of debris. Then, based on the debris, the shape of the debris and scene constraints, such as the number of debris, the number of robotic arms, and the scene stacking situation, it determines whether to adopt a single-hand gripping strategy or a two-hand gripping strategy. This enables the automatic screening of debris that has a significant impact on subsequent work processes, and solves the technical problems of low efficiency and poor accuracy in automatic sorting in the prior art.

[0111] The aforementioned robotic arm control device for sorting impurities in coal includes a processor and a memory. All the aforementioned units are stored as program units in the memory, and the processor executes these program units to achieve the corresponding functions. All the aforementioned modules reside in the same processor; alternatively, the modules may be located in different processors in any combination.

[0112] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the accuracy of automatic sorting can be improved by adjusting kernel parameters.

[0113] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0114] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the robotic arm control method for sorting impurities in coal.

[0115] Specifically, the control methods for robotic arms used to sort impurities from coal include:

[0116] Step S201: Obtain a first detection box, a first segmentation mask, a target direction, and a first distance. The first detection box is the target detection box corresponding to the debris in the detection image. The first segmentation mask is the instance segmentation mask corresponding to the debris. The first distance is the minimum safe working distance between the robotic arms. The target direction is the coal flow direction.

[0117] Step S202: Based on the target direction, the first detection box and the first segmentation mask are filtered to obtain a second detection box and a second segmentation mask. At least based on the second detection box and the second segmentation mask, the number of first targets, stacking complexity, aspect ratio of clutter, and second distance are determined. The second distance is the minimum distance between the clutter, the number of first targets is the number of clutter, and the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same target direction and overlap with the image border.

[0118] Step S203: Determine a first threshold based on the first distance, calculate the distance of each of the above-mentioned objects based on the first segmentation mask, and determine a second threshold based on the distance of each of the above-mentioned objects. The first threshold is a stacking complexity threshold, and the second threshold is a clutter aspect ratio threshold.

[0119] Step S204: Obtain the second target quantity. If the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, control the robotic arm to grasp in a two-hand grasping mode according to the first distance. The second target quantity is the total number of robotic arms.

[0120] Step S205: When the number of the first target is greater than the number of the second target and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance.

[0121] Step S206: If the number of the first target is less than or equal to the number of the second target and the stacking complexity is greater than the first threshold, and if the number of the third target is less than or equal to the number of the second target, adjust the first threshold until the number of the third target is greater than the number of the second target. If the number of the third target is greater than the number of the second target, control the robotic arm to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the number of the fourth target to the number of the second target, the first distance and the second distance.

[0122] Step S207: If the number of the first target is less than or equal to the number of the second target and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp in a two-hand grasping mode according to the first distance and the second distance. If the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a one-hand grasping mode according to the first distance and the second distance.

[0123] This invention provides a processor for running a program, wherein the program executes the aforementioned robotic arm control method for sorting impurities in coal.

[0124] Specifically, the control methods for robotic arms used to sort impurities from coal include:

[0125] Step S201: Obtain a first detection box, a first segmentation mask, a target direction, and a first distance. The first detection box is the target detection box corresponding to the debris in the detection image. The first segmentation mask is the instance segmentation mask corresponding to the debris. The first distance is the minimum safe working distance between the robotic arms. The target direction is the coal flow direction.

[0126] Step S202: Based on the target direction, the first detection box and the first segmentation mask are filtered to obtain a second detection box and a second segmentation mask. At least based on the second detection box and the second segmentation mask, the number of first targets, stacking complexity, aspect ratio of clutter, and second distance are determined. The second distance is the minimum distance between the clutter, the number of first targets is the number of clutter, and the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same target direction and overlap with the image border.

[0127] Step S203: Determine a first threshold based on the first distance, calculate the distance of each of the above-mentioned objects based on the first segmentation mask, and determine a second threshold based on the distance of each of the above-mentioned objects. The first threshold is a stacking complexity threshold, and the second threshold is a clutter aspect ratio threshold.

[0128] Step S204: Obtain the second target quantity. If the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, control the robotic arm to grasp in a two-hand grasping mode according to the first distance. The second target quantity is the total number of robotic arms.

[0129] Step S205: When the number of the first target is greater than the number of the second target and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance.

[0130] Step S206: If the number of the first target is less than or equal to the number of the second target and the stacking complexity is greater than the first threshold, and if the number of the third target is less than or equal to the number of the second target, adjust the first threshold until the number of the third target is greater than the number of the second target. If the number of the third target is greater than the number of the second target, control the robotic arm to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the number of the fourth target to the number of the second target, the first distance and the second distance.

[0131] Step S207: If the number of the first target is less than or equal to the number of the second target and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp in a two-hand grasping mode according to the first distance and the second distance. If the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a one-hand grasping mode according to the first distance and the second distance.

[0132] This invention provides a debris sorting system, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0133] Step S201: Obtain a first detection box, a first segmentation mask, a target direction, and a first distance. The first detection box is the target detection box corresponding to the debris in the detection image. The first segmentation mask is the instance segmentation mask corresponding to the debris. The first distance is the minimum safe working distance between the robotic arms. The target direction is the coal flow direction.

[0134] Step S202: Based on the target direction, the first detection box and the first segmentation mask are filtered to obtain a second detection box and a second segmentation mask. At least based on the second detection box and the second segmentation mask, the number of first targets, stacking complexity, aspect ratio of clutter, and second distance are determined. The second distance is the minimum distance between the clutter, the number of first targets is the number of clutter, and the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same target direction and overlap with the image border.

[0135] Step S203: Determine a first threshold based on the first distance, calculate the distance of each of the above-mentioned objects based on the first segmentation mask, and determine a second threshold based on the distance of each of the above-mentioned objects. The first threshold is a stacking complexity threshold, and the second threshold is a clutter aspect ratio threshold.

[0136] Step S204: Obtain the second target quantity. If the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, control the robotic arm to grasp in a two-hand grasping mode according to the first distance. The second target quantity is the total number of robotic arms.

[0137] Step S205: When the number of the first target is greater than the number of the second target and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance.

[0138] Step S206: If the number of the first target is less than or equal to the number of the second target and the stacking complexity is greater than the first threshold, and if the number of the third target is less than or equal to the number of the second target, adjust the first threshold until the number of the third target is greater than the number of the second target. If the number of the third target is greater than the number of the second target, control the robotic arm to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the number of the fourth target to the number of the second target, the first distance and the second distance.

[0139] Step S207: If the number of the first target is less than or equal to the number of the second target and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp in a two-hand grasping mode according to the first distance and the second distance. If the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a one-hand grasping mode according to the first distance and the second distance.

[0140] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0141] Step S201: Obtain a first detection box, a first segmentation mask, a target direction, and a first distance. The first detection box is the target detection box corresponding to the debris in the detection image. The first segmentation mask is the instance segmentation mask corresponding to the debris. The first distance is the minimum safe working distance between the robotic arms. The target direction is the coal flow direction.

[0142] Step S202: Based on the target direction, the first detection box and the first segmentation mask are filtered to obtain a second detection box and a second segmentation mask. At least based on the second detection box and the second segmentation mask, the number of first targets, stacking complexity, aspect ratio of clutter, and second distance are determined. The second distance is the minimum distance between the clutter, the number of first targets is the number of clutter, and the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same target direction and overlap with the image border.

[0143] Step S203: Determine a first threshold based on the first distance, calculate the distance of each of the above-mentioned objects based on the first segmentation mask, and determine a second threshold based on the distance of each of the above-mentioned objects. The first threshold is a stacking complexity threshold, and the second threshold is a clutter aspect ratio threshold.

[0144] Step S204: Obtain the second target quantity. If the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, control the robotic arm to grasp in a two-hand grasping mode according to the first distance. The second target quantity is the total number of robotic arms.

[0145] Step S205: When the number of the first target is greater than the number of the second target and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance.

[0146] Step S206: If the number of the first target is less than or equal to the number of the second target and the stacking complexity is greater than the first threshold, and if the number of the third target is less than or equal to the number of the second target, adjust the first threshold until the number of the third target is greater than the number of the second target. If the number of the third target is greater than the number of the second target, control the robotic arm to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the number of the fourth target to the number of the second target, the first distance and the second distance.

[0147] Step S207: If the number of the first target is less than or equal to the number of the second target and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp in a two-hand grasping mode according to the first distance and the second distance. If the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a one-hand grasping mode according to the first distance and the second distance.

[0148] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0149] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0154] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0155] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0156] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0157] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0158] 1) The robotic arm control method for sorting impurities in coal according to this application firstly acquires a first detection box, a first segmentation mask, a target direction, and a first distance. The first detection box is the target detection box corresponding to the impurity in the detection image, the first segmentation mask is the instance segmentation mask corresponding to the impurity, the first distance is the minimum safe working distance between the robotic arms, and the target direction is the coal flow direction. Then, based on the target direction, the first detection box and the first segmentation mask are filtered to obtain a second detection box and a second segmentation mask. At least based on the second detection box and the second segmentation mask, the first target quantity, stacking complexity, impurity aspect ratio, and second distance are determined. The second distance is the minimum distance between the aforementioned objects, the first target quantity is the number of objects, and the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same direction as the target and overlap with the image border. Then, a first threshold is determined based on the first distance, the distance between each of the aforementioned objects is calculated based on the first segmentation mask, and a second threshold is determined based on the distance between each of the aforementioned objects. The first threshold is a stacking complexity threshold, and the second threshold is a clutter aspect ratio threshold. Then, the second target quantity is obtained. If the first target quantity is greater than the second target quantity and the clutter aspect ratio is greater than the second threshold, the second threshold is determined based on the first distance. The robotic arms are controlled to grasp objects in a two-handed grasping mode, where the number of the second target is the total number of robotic arms. Then, if the number of the first target is greater than the number of the second target and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arms are controlled to grasp objects in a one-handed grasping mode based on the first and second distances. Then, if the number of the first target is less than or equal to the number of the second target and the stacking complexity is greater than the first threshold, and if the number of the third target is less than or equal to the number of the second target, the first threshold is adjusted until the number of the third target is greater than the number of the second target. If the number of the third target is greater than the number of the second target... The quantity is controlled by at least the ratio of the fourth target quantity to the second target quantity, the first distance, and the second distance, controlling the robotic arm to grasp in either a two-handed grasping mode or a one-handed grasping mode. Finally, if the first target quantity is less than or equal to the second target quantity and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp in a two-handed grasping mode based on the first distance and the second distance; if the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a one-handed grasping mode based on the first distance and the second distance.This application filters and removes redundant results based on the detection results of the category and location of debris. Then, based on the debris, the shape of the debris and scene constraints, such as the number of debris, the number of robotic arms, and the scene stacking situation, it determines whether to adopt a single-hand gripping strategy or a two-hand gripping strategy. This enables the automatic screening of debris that has a significant impact on subsequent work processes, and solves the technical problems of low efficiency and poor accuracy in automatic sorting in the prior art.

[0159] 2) The robotic arm control device for sorting impurities in coal according to this application comprises: a first acquisition unit acquiring a first detection frame, a first segmentation mask, a target direction, and a first distance; the first detection frame being the target detection frame corresponding to the impurity in the detection image; the first segmentation mask being the instance segmentation mask corresponding to the impurity; the first distance being the minimum safe working distance between the robotic arms; and the target direction being the coal flow direction; a first calculation unit filtering the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask; and determining at least the first target quantity, stacking complexity, impurity aspect ratio, and second distance based on the second detection frame and the second segmentation mask. The second distance is the minimum distance between the aforementioned objects, the first target quantity is the number of objects, the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same direction as the target and overlap with the image border; the second calculation unit determines a first threshold based on the first distance, calculates the distance of each of the aforementioned objects based on the first segmentation mask, and determines a second threshold based on the distance of each of the aforementioned objects, the first threshold is a stacking complexity threshold, and the second threshold is a clutter aspect ratio threshold; the second acquisition unit acquires the second target quantity, and when the first target quantity is greater than the second target quantity and the clutter aspect ratio is greater than the second threshold, it determines the second threshold based on the first distance. The first control unit controls the robotic arms to grasp objects in a two-handed grasping mode, where the second target quantity is the total number of robotic arms. If the first target quantity is greater than the second target quantity and the aspect ratio of the debris is less than or equal to the second threshold, the first control unit controls the robotic arms to grasp objects in a one-handed grasping mode based on the first and second distances. If the first target quantity is less than or equal to the second target quantity and the stacking complexity is greater than the first threshold, and the third target quantity is less than or equal to the second target quantity, the second control unit adjusts the first threshold until the third target quantity is greater than the second target quantity. The target quantity is determined by controlling the robotic arm to grasp objects in either a two-handed grasping mode or a one-handed grasping mode, based at least on the ratio of the fourth target quantity to the second target quantity, the first distance, and the second distance. If the first target quantity is less than or equal to the second target quantity and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the third control unit controls the robotic arm to grasp objects in a two-handed grasping mode based on the first distance and the second distance. If the aspect ratio of the debris is less than or equal to the second threshold, the third control unit controls the robotic arm to grasp objects in a one-handed grasping mode based on the first distance and the second distance.This application filters and removes redundant results based on the detection results of the category and location of debris. Then, based on the debris, the shape of the debris and scene constraints, such as the number of debris, the number of robotic arms, and the scene stacking situation, it determines whether to adopt a single-hand gripping strategy or a two-hand gripping strategy. This enables the automatic screening of debris that has a significant impact on subsequent work processes, and solves the technical problems of low efficiency and poor accuracy in automatic sorting in the prior art.

[0160] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for controlling a robotic arm used for sorting impurities in coal, characterized in that, include: The first detection box, the first segmentation mask, the target direction, and the first distance are obtained. The first detection box is the target detection box corresponding to the debris in the detection image. The first segmentation mask is the instance segmentation mask corresponding to the debris. The first distance is the minimum safe working distance between the robotic arms. The target direction is the coal flow direction. Based on the target direction, the first detection box and the first segmentation mask are filtered to obtain a second detection box and a second segmentation mask. At least based on the second detection box and the second segmentation mask, the first target quantity, stacking complexity, clutter aspect ratio and second distance are determined. The second distance is the minimum distance between the clutter. The first target quantity is the number of clutter. The second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same target direction and overlap with the image border. A first threshold is determined based on the first distance, the distance of each of the debris is calculated based on the first segmentation mask, and a second threshold is determined based on the distance of each of the debris. The first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold. Obtain a second target quantity. If the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, control the robotic arm to grasp in a two-hand grasping mode according to the first distance. The second target quantity is the total number of robotic arms. When the number of the first target is greater than the number of the second target and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a single-handed grasping mode according to the first distance and the second distance. If the first target quantity is less than or equal to the second target quantity and the stacking complexity is greater than the first threshold, and if the third target quantity is less than or equal to the second target quantity, the first threshold is adjusted until the third target quantity is greater than the second target quantity. If the third target quantity is greater than the second target quantity, the robotic arm is controlled to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the fourth target quantity to the second target quantity, the first distance and the second distance. If the number of the first target is less than or equal to the number of the second target and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp in a two-hand grasping mode according to the first distance and the second distance. If the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp in a one-hand grasping mode according to the first distance and the second distance.

2. The method according to claim 1, characterized in that, Determining the stacking complexity based on the second detection box and the second segmentation mask includes: The corresponding geometric center is determined based on the second detection frame to obtain the first target point, and the corresponding coordinates are determined based on the first target point to obtain the first target coordinates; Based on any one of the second detection boxes, the intersection-union ratio (IUU) with other second detection boxes is determined to obtain multiple target IUUs. Based on the first target coordinates corresponding to the second detection box and the first target coordinates corresponding to other second detection boxes, the minimum distance is determined to obtain the third distance. Substituting the target intersection-union ratio and the third distance corresponding to each of the first target points into the first target formula, we obtain multiple alternative stacking complexities; The maximum value of the multiple candidate stacking complexities is determined as the stacking complexity.

3. The method according to claim 1, characterized in that, Determining the aspect ratio of the clutter based at least on the second detection frame and the second segmentation mask includes: The area of ​​the debris is calculated based on the second detection frame to obtain a first area, and the area of ​​the debris is calculated based on the second segmentation mask to obtain a second area; Obtain the target length and target width, wherein the target length is the length of the second detection box and the target width is the width of the second detection box; Substituting the first area, the second area, the target length, and the target width into the second target formula yields the aspect ratio of the debris.

4. The method according to claim 2, characterized in that, Determining the second distance based at least on the second detection box and the second segmentation mask includes: The zeroth and first moments of the corresponding impurities are calculated based on the second segmentation mask to obtain the first target moment and the second target moment. Substitute the first target moment and the second target moment into the second target formula to determine the centroid coordinates of the debris, and obtain the second target coordinates; A third detection box is determined based on the second target coordinates and the second detection box, wherein the third detection box is the second detection box that has the smallest distance from the second target coordinates; Based on the first target coordinates of the third detection frame, the coordinates of the third target and the fourth target are determined, wherein the third target coordinates and the fourth target coordinates are the centroid coordinates of the two objects that are closest to the first target coordinates; The set distances are calculated based on the second target coordinates, the third target coordinates, and the fourth target coordinates, and the minimum value is determined as the second distance.

5. The method according to claim 4, characterized in that, When the number of the first target objects is greater than the number of the second target objects and the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp the debris in a two-handed grasping mode according to the first distance, including: The target backbone is obtained by extracting the backbone using the backbone extraction algorithm based on the second segmentation mask; The point with the smallest distance between the target backbone and the second target coordinates is determined as the second target point; With the second target point as the center and the first distance as the diameter, draw a circle, and determine the intersection of the circle and the target skeleton as the grab point; Based on the grasping points, a line is drawn, and the direction perpendicular to the line is determined as the grasping direction; The robotic arm is controlled to perform grasping based on the grasping point and the grasping direction.

6. The method according to claim 5, characterized in that, Controlling the robotic arm to grasp in a single-handed grasping mode based on the first distance and the second distance includes: When the first distance is greater than or equal to the second distance, a circle is drawn with the second target coordinates as the center and the first distance as the diameter. Any intersection point of the circle and the target skeleton is determined as the gripping point. A line is drawn based on the gripping point, and the direction perpendicular to the line is determined as the gripping direction. The robotic arm is controlled to perform gripping based on the gripping point and the gripping direction. When the first distance is less than the second distance, a circle is drawn with the third target coordinate or the fourth target coordinate as the center and the first distance as the diameter. Any intersection point of the circle and the target skeleton is determined as the gripping point. A line is drawn based on the gripping point, and the direction perpendicular to the line is determined as the gripping direction. The robotic arm is controlled to perform gripping based on the gripping point and the gripping direction.

7. The method according to claim 6, characterized in that, If the number of the third target is greater than the number of the second target, the robotic arm is controlled to grasp in either the two-handed grasping mode or the one-handed grasping mode, based at least on the ratio of the number of the fourth target to the number of the second target, the first distance, and the second distance. If the ratio of the first target data to the second target quantity is less than the third threshold, and if the debris is a hard debris, the robotic arm is controlled to grasp it in the two-hand grasping mode according to the first distance; if the debris is not a hard debris, the robotic arm is controlled to grasp it in the one-hand grasping mode according to the first distance and the second distance. If the ratio of the first target data to the second target quantity is greater than or equal to the third threshold, and if the debris is a hard debris and the aspect ratio of the debris is greater than the second threshold, the robotic arm is controlled to grasp it in the two-hand grasping mode according to the first distance. If the debris is a hard debris and the aspect ratio of the debris is less than or equal to the second threshold, the robotic arm is controlled to grasp it in the one-hand grasping mode according to the first distance and the second distance. If the debris is not a hard debris, it is not grasped.

8. A robotic arm control device for sorting impurities in coal, characterized in that, The device includes: The first acquisition unit is used to acquire a first detection box, a first segmentation mask, a target direction, and a first distance. The first detection box is the target detection box corresponding to the debris in the detection image. The first segmentation mask is the instance segmentation mask corresponding to the debris. The first distance is the minimum safe working distance between the robotic arms. The target direction is the coal flow direction. A first calculation unit is configured to filter the first detection box and the first segmentation mask based on the target direction to obtain a second detection box and a second segmentation mask, and to determine at least the second detection box and the second segmentation mask a first target quantity, a stacking complexity, a clutter aspect ratio, and a second distance, wherein the second distance is the minimum distance between the clutter, the first target quantity is the number of clutter, and the second detection box and the second segmentation mask are the first detection box and the first segmentation mask that are in the same target direction and overlap with the image border; The second calculation unit is used to determine a first threshold based on the first distance, calculate the distance of each of the debris based on the first segmentation mask, and determine a second threshold based on the distance of each of the debris. The first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold. The second acquisition unit is used to acquire the second target quantity. When the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, the robot arm is controlled to grasp in a two-hand grasping mode according to the first distance. The second target quantity is the total number of robot arms. A first control unit is configured to control the robotic arm to perform a single-handed grasping mode based on the first distance and the second distance when the number of the first target is greater than the number of the second target and the aspect ratio of the debris is less than or equal to the second threshold. The second control unit is configured to, when the first target quantity is less than or equal to the second target quantity and the stacking complexity is greater than the first threshold, adjust the first threshold until the third target quantity is greater than the second target quantity if the third target quantity is less than or equal to the second target quantity, and if the third target quantity is greater than the second target quantity, control the robotic arm to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the fourth target quantity to the second target quantity, the first distance and the second distance; The third control unit is configured to, when the first target quantity is less than or equal to the second target quantity and the stacking complexity is less than or equal to the first threshold, control the robotic arm to grasp in a two-handed grasping mode based on the first distance and the second distance if the aspect ratio of the debris is greater than the second threshold, and control the robotic arm to grasp in a one-handed grasping mode based on the first distance and the second distance if the aspect ratio of the debris is less than or equal to the second threshold.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 7.

10. A debris picking system, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising methods for performing any one of claims 1 to 7.

Images