Manipulator control method for selecting impurities in coal and impurity selecting system
Through the robot control method, the grabbing strategy is automatically adjusted according to the detected debris and the scene situation, which improves the efficiency and accuracy of gangue miscellaneous sorting, and solves the problem of low and poor automatic sorting efficiency in the existing technology.
Patent Information
- Application Number
- CN202411161257.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-22
AI Technical Summary
The automatic sorting efficiency and poor accuracy during the existing coal gangue miscellaneous sorting process, especially when the mining geological conditions change, it is difficult to effectively deal with hard debris in coal.
A robot control method is adopted to calculate the number of debris, stacking complexity and aspect ratio by obtaining the detection frame, segmentation mask and target direction, determine the manipulator grabbing mode, adopt a two-handed or one-handed grabbing strategy, and use a debris based on the debris. The number and scenario conditions automatically adjust the crawling strategy.
It improves the sorting efficiency and accuracy of debris in coal, adapts to debris sorting in complex scenarios, reduces the dependence on the number of robots, and enhances support for subsequent operation processes.
Smart Images

Figure CN119017382A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of mineral processing, and in particular to a manipulator control method, a manipulator control device, a computer-readable storage medium and a debris picking system for picking debris in coal. Background Art
[0002] In the field of coal, gangue and debris sorting, multi-robot system is a special equipment specially designed for gangue sorting. According to the different robot structures, multi-robot systems can be divided into articulated type, rectangular coordinate type, parallel type, etc. These systems are mainly responsible for handling the sorting of multiple dynamic targets in the process of gangue sorting to prevent large gangue or hard debris from entering the subsequent washing process and causing equipment failure. At present, there is a lack of fast, accurate and reliable multi-target multi-robot collaborative sorting strategy in the process of gangue sorting. In the existing online identification, positioning and sorting process, the sorting robot needs to assign tasks to each piece of debris detection result. However, when the mining geological conditions or construction working surface conditions change, the hard debris in the coal increases, and the one-to-one debris sorting strategy is inefficient and difficult to meet production needs. Therefore, a robot picking strategy for common debris in coal is proposed, which can effectively improve the sorting effect of various common debris in coal under complex scenarios. Summary of the invention
[0003] The main purpose of the present application is to provide a robot control method, a robot control device, a computer-readable storage medium and a debris picking system for picking debris in coal, so as to at least solve the problem of low efficiency and poor precision of automatic sorting in the prior art.
[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a manipulator control method for picking up debris in coal is provided, the method comprising: obtaining a first detection frame, a first segmentation mask, a target direction and a first distance, the first detection frame being a target detection frame corresponding to the debris in the detection image, the first segmentation mask being an instance segmentation mask corresponding to the debris, the first distance being a minimum safe working distance between manipulators, and the target direction being a flow direction of coal; screening the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, and determining the number of first targets at least based on the second detection frame and the second segmentation mask. , stacking complexity, debris aspect ratio and second distance, the second distance is the minimum distance between the debris, the first target number is the number of debris, the second detection frame and the second segmentation mask are the first detection frame and the first segmentation mask that are in the same direction as the target and overlap with the image border; determine a first threshold according to the first distance, calculate the distance of each of the debris according to the first segmentation mask and determine a second threshold according to the distance of each of the debris, the first threshold is the stacking complexity threshold, and the second threshold is the debris aspect ratio threshold; obtain the second target number, and when the first target number is greater than the second target number and the debris aspect ratio is greater than the second threshold In the case where the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance; in the case where the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number, the first threshold is adjusted until the third target number is greater than the second target number, and if the third target number is greater than the second target number the second target number, at least according to the ratio of the fourth target number to the second target number, the first distance and the second distance, the manipulator is controlled to grasp in the two-hand grasping mode or the one-hand grasping mode; when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, if the length-to-width ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in the two-hand grasping mode according to the first distance and the second distance; if the length-to-width ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in the one-hand grasping mode according to the first distance and the second distance.
[0005] Optionally, the stacking complexity is determined based on the second detection frame and the second segmentation mask, including: determining the corresponding geometric center according to the second detection frame to obtain the first target point, determining the corresponding coordinates according to the first target point to obtain the first target coordinates; determining the intersection and union ratio with any second detection frame to obtain multiple target intersection and union ratios, and determining the minimum distance according to the first target coordinates corresponding to the second detection frame and the first target coordinates corresponding to other second detection frames to obtain a third distance; substituting the target intersection and union ratio and the third distance corresponding to each of the first target points into the first target formula respectively to obtain multiple alternative stacking complexities; and determining the maximum value of the multiple alternative stacking complexities as the stacking complexity.
[0006] Optionally, determining the debris aspect ratio based at least on the second detection frame and the second segmentation mask includes: calculating the area of the debris according to the second detection frame to obtain a first area, and calculating the area of the debris according to the second segmentation mask to obtain a second area; obtaining a target length and a target width, wherein the target length is the length of the second detection frame, and the target width is the width of the second detection frame; and substituting the first area, the second area, the target length, and the target width into a second target formula to obtain the debris aspect ratio.
[0007] Optionally, determining the second distance based at least on the second detection frame and the second segmentation mask includes: calculating the corresponding zero-order moment and first-order moment of the debris according to the second segmentation mask, respectively, to obtain the first target moment and the second target moment; substituting the first target moment and the second target moment into the second target formula to determine the center of gravity coordinates of the debris to obtain the second target coordinates; determining a third detection frame according to the second target coordinates and the second detection frame, the third detection frame being the second detection frame with the smallest distance from the second target coordinates; determining third target coordinates and fourth target coordinates based on the first target coordinates of the third detection frame, the third target coordinates and the fourth target coordinates being the center of gravity coordinates of the two debris with the smallest distance from the first target coordinates; calculating the set distance based on the second target coordinates and the third target coordinates and the fourth target coordinates, respectively, and determining the minimum value as the second distance.
[0008] Optionally, when the number of the first targets is greater than the number of the second targets and the aspect ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in a two-handed grasping mode according to the first distance, including: extracting the target backbone through a backbone extraction algorithm according to the second segmentation mask; determining the point with the smallest distance between the target backbone and the second target coordinates as the second target point; making a circle with the second target point as the center and the first distance as the diameter, and determining the intersection of the circle and the target backbone as the grasping point; drawing a line based on the grasping point, and determining the direction perpendicular to the line as the grasping direction; and controlling the manipulator to grasp based on the grasping point and the grasping direction.
[0009] Optionally, the manipulator is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance, including: when the first distance is greater than or equal to the second distance, a circle is drawn with the second target coordinate as the center and the first distance as the diameter, any intersection of the circle and the target backbone is determined as a grasping point, a line is connected based on the grasping point, a direction perpendicular to the line is determined as the grasping direction, and the manipulator is controlled to grasp based on the grasping point and the grasping direction; when the first distance is less than the second distance, a circle is drawn with the third target coordinate or the fourth target coordinate as the center and the first distance as the diameter, any intersection of the circle and the target backbone is determined as a grasping point, a line is connected based on the grasping point, a direction perpendicular to the line is determined as the grasping direction, and the manipulator is controlled to grasp based on the grasping point and the grasping direction.
[0010] Optionally, if the third target number is greater than the second target number, the manipulator is controlled to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the fourth target number to the second target number, the first distance, and the second distance, including: when the ratio of the first target data to the second target number is less than a third threshold, if the debris is hard debris, the manipulator is controlled to grasp in the two-hand grasping mode according to the first distance, and if the debris is not the hard debris, the manipulator is controlled to grasp in the one-hand grasping mode according to the first distance and the second distance; when the ratio of the first target data to the second target number is greater than or equal to the third threshold, if the debris is the hard debris and the length-to-width ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in the two-hand grasping mode according to the first distance, if the debris is the hard debris and the length-to-width ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in the one-hand grasping mode according to the first distance and the second distance, and if the debris is not the hard debris, no grasping is performed.
[0011] According to another aspect of the present application, a manipulator control device for picking up debris in coal is provided, the device comprising: a first acquisition unit, used to acquire a first detection frame, a first segmentation mask, a target direction and a first distance, the first detection frame being a target detection frame corresponding to the debris in the detection image, the first segmentation mask being an instance segmentation mask corresponding to the debris, the first distance being a minimum safe working distance between manipulators, and the target direction being a flow direction of coal; a first calculation unit, used to screen the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, and to determine the first target quantity, the stacking complexity and the number of first targets at least based on the second detection frame and the second segmentation mask; the first detection frame and the second segmentation mask are the first detection frame and the first segmentation mask that are in the same direction as the target and overlap with the image border; a second calculation unit is used to determine a first threshold according to the first distance, calculate the distance of each of the debris according to the first segmentation mask, and determine a second threshold according to the distance of each of the debris, the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold; a second acquisition unit is used to acquire the second target number, and when the first target number is greater than the second target number and the debris aspect ratio is greater than the second threshold In the case where the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in a two-hand grasping mode according to the first distance, and the second target number is the total number of the manipulators; a first control unit is used to control the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold; a second control unit is used to adjust the first threshold if the third target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, until the third target number is greater than the second target number, if the third target number greater than the second target number, at least according to the ratio of the fourth target number to the second target number, the first distance and the second distance, controlling the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode; a third control unit, for, when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, if the length-to-width ratio of the debris is greater than the second threshold, controlling the manipulator to grasp in the two-hand grasping mode according to the first distance and the second distance, and if the length-to-width ratio of the debris is less than or equal to the second threshold, controlling the manipulator to grasp in the one-hand grasping mode according to the first distance and the second distance.
[0012] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.
[0013] According to another aspect of the present application, a debris picking system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.
[0014] Applying the technical solution of the present application, in the above-mentioned manipulator control method for picking up debris in coal, the above-mentioned method includes: first, obtaining a first detection frame, a first segmentation mask, a target direction and a first distance, the above-mentioned first detection frame is a target detection frame corresponding to the debris in the detection image, the above-mentioned first segmentation mask is an instance segmentation mask corresponding to the above-mentioned debris, the above-mentioned first distance is the minimum safe working distance between manipulators, and the above-mentioned target direction is the flow direction of coal; then, based on the above-mentioned target direction, the above-mentioned first detection frame and the above-mentioned first segmentation mask are screened to obtain a second detection frame and a second segmentation mask, and at least based on the above-mentioned second detection frame and the above-mentioned second segmentation mask, the first target quantity, stacking complexity, The debris aspect ratio and the second distance, the second distance is the minimum distance between the debris, the first target number is the number of debris, the second detection frame and the second segmentation mask are the first detection frame and the first segmentation mask that are in the same direction as the target and overlap with the image border; then, a first threshold is determined according to the first distance, the distance of each of the debris is calculated according to the first segmentation mask, and a second threshold is determined according to the distance of each of the debris, the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold; then, the second target number is obtained, and when the first target number is greater than the second target number and the debris aspect ratio is greater than the second threshold, based on According to the above-mentioned first distance, the above-mentioned manipulator is controlled to grasp in a two-hand grasping mode, and the above-mentioned second target number is the total number of the manipulator; thereafter, when the above-mentioned first target number is greater than the above-mentioned second target number and the above-mentioned aspect ratio of the debris is less than or equal to the above-mentioned second threshold value, the above-mentioned manipulator is controlled to grasp in a single-hand grasping mode according to the above-mentioned first distance and the above-mentioned second distance; thereafter, when the above-mentioned first target number is less than or equal to the above-mentioned second target number and the above-mentioned stacking complexity is greater than the above-mentioned first threshold value, if the third target number is less than or equal to the above-mentioned second target number, the above-mentioned first threshold value is adjusted until the above-mentioned third target number is greater than the above-mentioned second target number, if the above-mentioned third target number is greater than the above-mentioned first threshold value The first and second distances are used to control the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the fourth target number to the second target number, the first distance and the second distance; finally, when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, if the aspect ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in the two-hand grasping mode according to the first distance and the second distance; if the aspect ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in the one-hand grasping mode according to the first distance and the second distance.The present application screens and removes redundant results based on the category and position detection results of the debris, and then determines the use of single-handed grasping strategy and two-handed grasping strategy based on the debris, the shape of the debris and the scene restrictions, such as the number of debris, the number of manipulators, and the scene stacking situation, to achieve automatic screening of debris that has a greater impact on subsequent work processes, solving the technical problem of low efficiency and poor precision of automatic sorting in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 A hardware structure block diagram of a mobile terminal according to a manipulator control method for picking debris in coal provided in an embodiment of the present application is shown;
[0016] Figure 2 A schematic flow chart of a manipulator control method for selecting debris in coal provided according to an embodiment of the present application is shown;
[0017] Figure 3 A structural block diagram of a manipulator control device for picking debris in coal provided according to an embodiment of the present application is shown.
[0018] The above drawings include the following reference numerals:
[0019] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION
[0020] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0023] As introduced in the background technology, the prior art cannot effectively sort the debris in the coal due to changes in coal flow debris caused by changes in geological conditions of coal mining and technical adjustments to the working plane. In order to solve the problem of low efficiency and poor precision of automatic sorting, the embodiments of the present application provide a manipulator control method, a manipulator control device, a computer-readable storage medium and a debris picking system for picking debris in coal.
[0024] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0025] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a manipulator control method for selecting impurities in coal according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0026] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0027] In this embodiment, a robot control method for picking debris in coal that runs on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0028] Figure 2 FIG. 1 is a flow chart of a manipulator control method for selecting debris in coal according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:
[0029] Step S201, obtaining a first detection frame, a first segmentation mask, a target direction and a first distance, wherein the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between manipulators, and the target direction is a flow direction of the coal;
[0030] Specifically, the debris picking system inputs the target detection frame corresponding to the debris in the detection image, the instance segmentation mask corresponding to the debris, and the movement direction of the given coal flow. Minimum safety distance L between the robot and the robot MINAnd other parameters.
[0031] Step S202, screening the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, determining the first target quantity, stacking complexity, debris aspect ratio, and second distance based at least on the second detection frame and the second segmentation mask, the second distance being the minimum distance between the debris, the first target quantity being the quantity of the debris, and the second detection frame and the second segmentation mask being the first detection frame and the first segmentation mask having the same direction as the target and overlapping with the image border;
[0032] Specifically, according to the flow direction of the coal, the target detection frame corresponding to the debris in the detection image and the instance segmentation mask corresponding to the debris are screened, and the instance segmentation mask and the corresponding detection frame corresponding to the debris that intersect the upper and lower edges of the debris detection image along the flow direction of the coal are removed. The removed result is returned to obtain the above-mentioned second detection frame and second segmentation mask. Each detection result is traversed to determine the number of debris N, that is, the first target number, the stacking complexity C, the debris aspect ratio F and the minimum distance l between debris. m , which is the second distance mentioned above.
[0033] Step S203, determining a first threshold according to the first distance, calculating the distance of each of the debris according to the first segmentation mask, and determining a second threshold according to the distance of each of the debris, wherein the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold;
[0034] Specifically, the backbone extraction algorithm is used to extract the backbone f(x, y) of the debris segmentation mask, combined with the sorting scene debugging results and the number of available manipulators N MANIPULATOR , the stacking complexity threshold C is determined based on the minimum safe working distance between manipulators THRESHOLD , calculate the distance between detected debris according to the instance segmentation mask corresponding to the debris, and determine the debris aspect ratio threshold F according to the distance between the debris THRESHOLD .
[0035] Step S204, obtaining a second target number, and when the first target number is greater than the second target number and the aspect ratio of the debris is greater than the second threshold, controlling the manipulator to grasp in a two-hand grasping mode according to the first distance, and the second target number is the total number of the manipulators;
[0036] Specifically, get the total number of current manipulators N MANIPULATOR , determine whether the number of detected debris N is greater than the total number of current manipulators N MANIPULATOR , when the number of debris N is greater than the number of available manipulators N MANIPULATOR , that is, N>N MANIPULATORIn the case of insufficient number of manipulators, the priority picking strategy is adopted to determine whether the length-to-width ratio F of the debris is greater than the second threshold F THRESHOLD , if F>F THRESHOLD , then adopt the "dual manipulator debris grasping strategy", control the above manipulators to grasp in a two-handed grasping mode according to the minimum safe working distance between the manipulators, and return the index number (index), grasping point (loc), and hand direction (drc) of the debris to the computer to complete the instruction registration.
[0037] Step S205, when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold, controlling the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance;
[0038] Specifically, when the number of debris N is greater than the number of available manipulators N MANIPULATOR , that is, N>N MANIPULATOR In the case of insufficient number of manipulators, the priority picking strategy is adopted to determine whether the length-to-width ratio F of the debris is greater than the second threshold F THRESHOLD , if F≤F THRESHOLD , then adopt the "single robot debris grasping strategy", control the above robot to grasp in two-handed grasping mode according to the minimum safe working distance between the robots and the minimum distance between the debris, and return the index number (index), grasping point (loc), and hand direction (drc) of the debris to the computer to complete the instruction registration.
[0039] Step S206, when the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number, adjust the first threshold until the third target number is greater than the second target number, and if the third target number is greater than the second target number, control the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the fourth target number to the second target number, the first distance and the second distance;
[0040] Specifically, when the number of debris N is less than or equal to the number of available manipulators N MANIPULATOR , that is, N>N MANIPULATOR , adopt the strategy of giving priority to picking when the number of manipulators is sufficient, and the stacking complexity is greater than the first threshold, that is, C>C THRESHOLD , in the case of high stacking scene debris sorting strategy, if the number of debris in the stacking area of the third target number N ′ Less than or equal to the total number of manipulators N MANIPULATOR , that is, N ′ ≤N MANIPULATOR , then increase the preset stacking complexity threshold CTHRESHOLD Until the condition is met, the number of debris in the stacking area N ′ Greater than the number of manipulators N MANIPULATOR , and then re-debug the sorting robot. If the third target number of debris in the stacking area is N ′ Greater than the total number of manipulators N MANIPULATOR , that is, N ′ ≤N MANIPULATOR , then according to the re-judgment of the number of debris at this time, according to the ratio of the number of debris to the number of available manipulators, the minimum safe working distance between the manipulators, and the minimum distance between the debris, the "single manipulator debris grasping strategy" or "double manipulator debris grasping strategy" is adopted accordingly, and the manipulator is controlled to grasp in two-hand grasping mode or one-hand grasping mode, and the index number (index), grasping point (loc), and lower hand direction (drc) of the debris are returned to the computer to complete the instruction registration.
[0041] Step S207, when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, if the aspect ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in a two-hand grasping mode according to the first distance and the second distance; if the aspect ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance.
[0042] Specifically, when the number of debris N is less than or equal to the number of available manipulators N MANIPULATOR , and the stacking complexity is less than or equal to the first threshold, that is, C≤C THRESHOLD , the low stacking scene debris sorting strategy is adopted. If the aspect ratio of the debris is greater than the second threshold, that is, F>F THRESHOLD In the case of , the dual-manipulator debris grasping strategy is executed, and the above-mentioned manipulators are controlled to grasp in a two-hand grasping mode according to the minimum safe working distance between the manipulators and the minimum distance between the debris. If the aspect ratio of the above-mentioned debris is less than or equal to the above-mentioned second threshold, that is, F≤F THRESHOLD In this case, a single-handed manipulator debris grasping strategy is executed, and the above-mentioned manipulator is controlled to grasp in a single-handed grasping mode according to the minimum safe working distance between the manipulators and the minimum distance between the debris.
[0043] Through this embodiment, first, a first detection frame, a first segmentation mask, a target direction and a first distance are obtained, wherein the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between manipulators, and the target direction is a flow direction of coal; then, based on the target direction, the first detection frame and the first segmentation mask are screened to obtain a second detection frame and a second segmentation mask, and the first target quantity, stacking complexity, debris aspect ratio and second distance are determined at least based on the second detection frame and the second segmentation mask, wherein the second distance is the distance between the debris. The minimum distance, the first target number is the number of debris, the second detection frame and the second segmentation mask are the first detection frame and the first segmentation mask that are in the same direction as the target and overlap with the image frame; then, a first threshold is determined according to the first distance, the distance of each of the debris is calculated according to the first segmentation mask, and a second threshold is determined according to the distance of each of the debris, the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold; then, the second target number is obtained, and when the first target number is greater than the second target number and the debris aspect ratio is greater than the second threshold, the manipulator is controlled according to the first distance to double The robot grasps in the single-hand grasping mode, and the second target number is the total number of the robot; then, when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold, the robot is controlled to grasp in the single-hand grasping mode according to the first distance and the second distance; then, when the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number, adjust the first threshold until the third target number is greater than the second target number, and if the third target number is greater than the second target number, at least According to the ratio of the fourth target number to the second target number, the first distance and the second distance, the manipulator is controlled to grasp in the two-hand grasping mode or the one-hand grasping mode; finally, when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, if the length-to-width ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in the two-hand grasping mode according to the first distance and the second distance; if the length-to-width ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in the one-hand grasping mode according to the first distance and the second distance.The present application screens and removes redundant results based on the category and position detection results of the debris, and then determines the use of single-handed grasping strategy and two-handed grasping strategy based on the debris, the shape of the debris and the scene restrictions, such as the number of debris, the number of manipulators, and the scene stacking situation, to achieve automatic screening of debris that has a greater impact on subsequent work processes, solving the technical problem of low efficiency and poor precision of automatic sorting in the prior art.
[0044] In order to determine the stacking complexity, in an optional implementation, the above step S202 includes:
[0045] Step S20201, determining the corresponding geometric center according to the second detection frame to obtain a first target point, and determining the corresponding coordinates according to the first target point to obtain first target coordinates;
[0046] Specifically, the debris x is calculated based on the second detection frame a The geometric center of the detection box is returned, the geometric center position (i, j) is returned, the first target point is obtained, the coordinates of the corresponding position are determined, and the first target coordinates are obtained.
[0047] Step S20202, determining an intersection-and-union ratio with other second detection frames according to any of the second detection frames to obtain a plurality of target intersection-and-union ratios, determining a minimum distance according to the first target coordinates corresponding to the second detection frame and the first target coordinates corresponding to other second detection frames to obtain a third distance;
[0048] Specifically, according to any debris x a The above second detection frame determines the intersection and union (IoU) ratio with the above detection frames of other debris (i,j) , get the intersection and union ratio of multiple target debris in the debris detection frame, according to a debris x a The minimum distance between the center points of the debris is calculated by the geometric center position (i, j) coordinates of the detection frame of the other debris detection frames and the geometric center position coordinates of the other debris detection frames. The third distance mentioned above is determined.
[0049] Step S20203, substituting the target intersection-and-union ratio and the third distance corresponding to each of the first target points into the first target formula to obtain a plurality of candidate stacking complexities;
[0050] Specifically, according to the target intersection over union (IoU) corresponding to the first target point of each debris (i,j) and the third distance Using the first objective formula Calculate the stacking complexity of each clutter item with respect to other clutter items.
[0051] Step S20204: determine the maximum value of the multiple candidate stacking complexities as the stacking complexity.
[0052] Specifically, the stacking complexity between each sundry object and other sundries is calculated respectively, and the highest stacking complexity between the sundries is taken to be determined as the above stacking complexity.
[0053] In order to determine the aspect ratio of the debris, in an optional implementation, the above step S202 includes:
[0054] Step S20211, calculating the area of the debris according to the second detection frame to obtain a first area, and calculating the area of the debris according to the second segmentation mask to obtain a second area;
[0055] Specifically, the first area A is calculated based on the second detection frame. box , calculate the debris area A according to the second segmentation mask mask .
[0056] Step S20212, obtaining a target length and a target width, wherein the target length is the length of the second detection frame, and the target width is the width of the second detection frame;
[0057] Specifically, the target length h used to calculate the aspect ratio of the debris is determined according to the length and width of the second detection frame. box and the target width w box .
[0058] Step S20213, substituting the first area, the second area, the target length and the target width into the second target formula to obtain the debris aspect ratio.
[0059] Specifically, according to the sundries x a The detection box and segmentation mask results are used to calculate the shape factor The above-mentioned aspect ratio of debris is obtained.
[0060] In order to determine the second distance, in an optional implementation, the above step S202 includes:
[0061] Step S20221, calculating the zero-order moment and the first-order moment of the corresponding debris respectively according to the second segmentation mask to obtain a first target moment and a second target moment;
[0062] Specifically, the zero-order moment and first-order moment of the irregular debris are calculated according to the screened second segmentation mask. The zero-order moment is obtained by M 00 =∑ I ∑ J V(i,j) is calculated, where V(i,j) is the gray value of point (i,j), and the first target moment is obtained. The first-order moment is obtained by M 10 =∑ I ∑ J i·V(i,j),M01 =∑ I ∑ J j·V(i,j) is calculated to obtain the second target moment, where i and j are the x and y coordinates of each pixel respectively. This definition is essentially the product of the x and y coordinates of all pixels and the pixel value, and then the sum is obtained.
[0063] Step S20222, substituting the first target moment and the second target moment into the second target formula, determining the center of gravity coordinates of the debris, and obtaining the second target coordinates;
[0064] Specifically, using the above second objective formula Calculate the centroid coordinates of the irregular object instance segmentation mask to obtain the second target coordinates.
[0065] Step S20223, determining a third detection frame according to the second target coordinates and the second detection frame, wherein the third detection frame is the second detection frame having the shortest distance from the second target coordinates;
[0066] Specifically, the third detection frame having the smallest distance from the second coordinates is determined according to the center of gravity coordinates of the irregular debris calculated by the second target formula, and is used to determine the information of other debris.
[0067] Step S20224, determining a third target coordinate and a fourth target coordinate based on the first target coordinate of the third detection frame, wherein the third target coordinate and the fourth target coordinate are the centroid coordinates of the two debris objects having the smallest distance from the first target coordinate;
[0068] Specifically, the centroid coordinates of the other two debris with the smallest distance are determined based on the third detection frame and the first target coordinates of the debris.
[0069] Step S20225 calculates the set distance based on the second target coordinates, the third target coordinates and the fourth target coordinates, and determines the minimum value as the second distance.
[0070] Specifically, take the distance to the debris x a The two debris x corresponding to the geometric center of the nearest detection frame m 、x n Next, the problem is transformed into the problem of finding the point (g i ,g j ) to the other two points, and find the corresponding distances respectively.
[0071] In order to determine the robot grasping mode, in an optional implementation, the above step S204 includes:
[0072] Step S2041, extracting through a backbone extraction algorithm according to the second segmentation mask to obtain a target backbone;
[0073] Specifically, take the debris x a The centroid (i, j) of the image is obtained by using the second segmentation mask, and then the boundary of the image is marked. Then, the redundant points of the marked boundary are removed, and the backbone extraction algorithm is used to extract the backbone f(x, y) of the segmentation mask to obtain the target backbone.
[0074] Step S2042, determining the point with the smallest distance between the target backbone and the second target coordinates as the second target point;
[0075] Specifically, calculate the target backbone f(x,y) up to the debris x a The point closest to the center of gravity (i, j) is denoted as (i nearest ,j nearest ), determined as the second target point.
[0076] Step S2043, draw a circle with the second target point as the center and the first distance as the diameter, and determine the intersection of the circle and the target backbone as the grasping point;
[0077] Specifically, taking the second target point (i nearest ,j nearest ) is the center of the circle, and the minimum safe working distance between the manipulators is L MIN Draw a circle with the diameter of f(x,y) and intersect with the target backbone f(x,y) at points p1 and p2 respectively, or only at one point p. Any point can be selected as the grasping point of a single manipulator.
[0078] Step S2044, connecting lines based on the above grasping points, and determining the direction perpendicular to the line as the grasping direction;
[0079] Specifically, connect the above intersection points p1 and p2 to obtain the line l p1,p2 , and determine the vertical direction of the connecting line as the grasping direction of the robot.
[0080] Step S2045: Control the robot arm to grasp based on the grasping point and the grasping direction.
[0081] Specifically, the robot is controlled to grasp in a single-hand grasping mode or a two-hand grasping mode according to the grasping point and the grasping direction.
[0082] In order to determine the robot grasping mode, in an optional implementation, the above step S205 includes:
[0083] Step S2051, when the first distance is greater than or equal to the second distance, a circle is drawn with the second target coordinate as the center and the first distance as the diameter, any intersection of the circle and the target backbone is determined as a grasping point, a line is drawn based on the grasping point, a direction perpendicular to the line is determined as the grasping direction, and the manipulator is controlled to grasp based on the grasping point and the grasping direction;
[0084] Specifically, determine the minimum safe working distance L between the manipulators MIN Is it greater than the minimum distance between debris? Minimum safe working distance L between manipulators MIN Greater than or equal to the minimum distance between debris In the case of the above debris center of gravity coordinates as the center of the circle, the minimum safe working distance between the manipulators is L MIN Draw a circle with the diameter, and select any point of the intersection with the target backbone f(x, y) as the grasping point of a single manipulator. Connect the above intersection points to get a line, and determine the vertical direction of the line as the grasping direction of the manipulator.
[0085] Step S2052, when the above-mentioned first distance is smaller than the above-mentioned second distance, a circle is made with the above-mentioned third target coordinates or the above-mentioned fourth target coordinates as the center and the above-mentioned first distance as the diameter, and any intersection point of the circle with the above-mentioned target backbone is determined as the grasping point, and a line is drawn based on the above-mentioned grasping point, and the direction perpendicular to the line is determined as the above-mentioned grasping direction, and the above-mentioned robot arm is controlled to grasp based on the above-mentioned grasping point and the above-mentioned grasping direction.
[0086] Specifically, the minimum safe working distance L between the manipulators MIN Less than the minimum distance between debris In the case of, with the third target coordinate or the fourth target coordinate as the center, the minimum safe working distance L between the manipulators MIN Draw a circle with the diameter, and select any point of the intersection with the target backbone f(x, y) as the grasping point of a single manipulator. Connect the above intersection points to get a line, and determine the vertical direction of the line as the grasping direction of the manipulator.
[0087] In order to determine the robot grasping mode, in an optional implementation, the above step S206 includes:
[0088] Step S2061, when the ratio of the first target data to the second target number is less than a third threshold, if the debris is hard debris, the manipulator is controlled to grasp it in the two-hand grasping mode according to the first distance; if the debris is not hard debris, the manipulator is controlled to grasp it in the one-hand grasping mode according to the first distance and the second distance;
[0089] Specifically, when the ratio of the number of debris to the total number of manipulators is less than the third threshold value, if the debris is hard debris, a dual-manipulator debris grasping strategy is adopted, and according to the minimum safe working distance between the manipulators, the manipulator is controlled to grasp in the two-hand grasping mode; if the debris is not hard debris, a single-manipulator debris grasping strategy is adopted, and according to the minimum safe working distance between the manipulators and the distance between the debris, the manipulator is controlled to grasp in the single-hand grasping mode.
[0090] Step S2062, when the ratio of the first target data to the second target quantity is greater than or equal to the third threshold, if the debris is the hard debris and the aspect ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in the two-hand grasping mode according to the first distance; if the debris is the hard debris and the aspect ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in the one-hand grasping mode according to the first distance and the second distance; if the debris is not the hard debris, no grasping is performed.
[0091] Specifically, when the ratio of the number of debris to the total number of manipulators is greater than or equal to the third threshold, if the debris is hard and the aspect ratio of the debris is greater than the second threshold F>F THRESHOLD , a dual-manipulator debris grasping strategy is adopted. According to the minimum safe working distance between the manipulators, the manipulators are controlled to grasp in the dual-manipulator grasping mode. If the debris is hard and the aspect ratio of the debris is less than or equal to the second threshold F≤F THRESHOLD , a single-manipulator debris grasping strategy is adopted. According to the minimum safe working distance between the manipulators and the distance between the debris, the manipulator is controlled to grasp in the above-mentioned single-hand grasping mode. If the above-mentioned debris is not the above-mentioned hard debris, it will not be grasped and will jump directly to the next debris.
[0092] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0093] The embodiment of the present application also provides a manipulator control device for picking out debris in coal. It should be noted that the manipulator control device for picking out debris in coal in the embodiment of the present application can be used to execute the manipulator control method for picking out debris in coal provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0094] The following is an introduction to the manipulator control device for picking debris in coal provided in an embodiment of the present application.
[0095] Figure 3 1 is a structural block diagram of a manipulator control device for selecting impurities in coal according to an embodiment of the present application. Figure 3 As shown, the device comprises:
[0096] The first acquisition unit 10 is used to acquire a first detection frame, a first segmentation mask, a target direction and a first distance, wherein the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between manipulators, and the target direction is a flow direction of the coal;
[0097] Specifically, the debris picking system inputs the target detection frame corresponding to the debris in the detection image, the instance segmentation mask corresponding to the debris, and the movement direction of the given coal flow. Minimum safety distance L between the robot and the robot MIN And other parameters.
[0098] A first calculation unit 20 is used to screen the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, and determine the first target quantity, stacking complexity, debris aspect ratio, and second distance based on at least the second detection frame and the second segmentation mask, wherein the second distance is the minimum distance between the debris, the first target quantity is the quantity of the debris, and the second detection frame and the second segmentation mask are the first detection frame and the first segmentation mask having the same direction as the target and overlapping with the image border;
[0099] Specifically, according to the flow direction of the coal, the target detection frame corresponding to the debris in the detection image and the instance segmentation mask corresponding to the debris are screened, and the instance segmentation mask and the corresponding detection frame corresponding to the debris that intersect the upper and lower edges of the debris detection image along the flow direction of the coal are removed. The removed result is returned to obtain the above-mentioned second detection frame and second segmentation mask. Each detection result is traversed to determine the number of debris N, that is, the first target number, the stacking complexity C, the debris aspect ratio F and the minimum distance l between debris. m , which is the second distance mentioned above.
[0100] A second calculation unit 30 is used to determine a first threshold according to the first distance, calculate the distance of each of the above-mentioned debris according to the first segmentation mask, and determine a second threshold according to the distance of each of the above-mentioned debris, wherein the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold;
[0101] Specifically, the backbone extraction algorithm is used to extract the backbone f(x, y) of the debris segmentation mask, combined with the sorting scene debugging results and the number of available manipulators N MANIPULATOR , the stacking complexity threshold C is determined based on the minimum safe working distance between manipulators THRESHOLD , calculate the distance between detected debris according to the instance segmentation mask corresponding to the debris, and determine the debris aspect ratio threshold F according to the distance between the debris THRESHOLD .
[0102] A second acquisition unit 40 is used to acquire a second target quantity. When the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in a two-hand grasping mode according to the first distance. The second target quantity is the total quantity of the manipulator.
[0103] Specifically, get the total number of current manipulators N MANIPULATOR , determine whether the number of detected debris N is greater than the total number of current manipulators N MANIPULATOR , when the number of debris N is greater than the number of available manipulators N MANIPULATOR , that is, N>N MANIPULATOR In the case of insufficient number of manipulators, the priority picking strategy is adopted to determine whether the length-to-width ratio F of the debris is greater than the second threshold F THRESHOLD , if F>F THRESHOLD , then adopt the "dual manipulator debris grasping strategy", control the above manipulators to grasp in a two-handed grasping mode according to the minimum safe working distance between the manipulators, and return the index number (index), grasping point (loc), and hand direction (drc) of the debris to the computer to complete the instruction registration.
[0104] A first control unit 50 is used to control the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold;
[0105] Specifically, when the number of debris N is greater than the number of available manipulators N MANIPULATOR , that is, N>N MANIPULATOR In the case of insufficient number of manipulators, the priority picking strategy is adopted to determine whether the length-to-width ratio F of the debris is greater than the second threshold F THRESHOLD , if F≤F THRESHOLD , then adopt the "single robot debris grasping strategy", control the above robot to grasp in two-handed grasping mode according to the minimum safe working distance between the robots and the minimum distance between the debris, and return the index number (index), grasping point (loc), and hand direction (drc) of the debris to the computer to complete the instruction registration.
[0106] A second control unit 60 is used for, when the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number, adjusting the first threshold until the third target number is greater than the second target number, and if the third target number is greater than the second target number, controlling the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to a ratio of the fourth target number to the second target number, the first distance and the second distance;
[0107] Specifically, when the number of debris N is less than or equal to the number of available manipulators N MANIPULATOR , that is, N>N MANIPULATOR , adopt the strategy of giving priority to picking when the number of manipulators is sufficient, and the stacking complexity is greater than the first threshold, that is, C>C THRESHOLD , in the case of high stacking scene debris sorting strategy, if the number of debris in the stacking area of the third target number N ′ Less than or equal to the total number of manipulators N MANIPULATOR , that is, N ′ ≤N MANIPULATOR , then increase the preset stacking complexity threshold C THRESHOLD Until the condition is met, the number of debris in the stacking area N ′ Greater than the number of manipulators N MANIPULATOR , and then re-debug the sorting robot. If the third target number of debris in the stacking area is N ′ Greater than the total number of manipulators N MANIPULATOR , that is, N ′ ≤N MANIPULATOR, then according to the re-judgment of the number of debris at this time, according to the ratio of the number of debris to the number of available manipulators, the minimum safe working distance between the manipulators, and the minimum distance between the debris, the "single manipulator debris grasping strategy" or "double manipulator debris grasping strategy" is adopted accordingly, and the manipulator is controlled to grasp in two-hand grasping mode or one-hand grasping mode, and the index number (index), grasping point (loc), and lower hand direction (drc) of the debris are returned to the computer to complete the instruction registration.
[0108] The third control unit 70 is used to control the manipulator to grasp in a two-handed grasping mode according to the first distance and the second distance when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, and if the aspect ratio of the debris is greater than the second threshold, and to control the manipulator to grasp in a single-handed grasping mode according to the first distance and the second distance when the aspect ratio of the debris is less than or equal to the second threshold.
[0109] Specifically, when the number of debris N is less than or equal to the number of available manipulators N MANIPULATOR , and the stacking complexity is less than or equal to the first threshold, that is, C≤C THRESHOLD , the low stacking scene debris sorting strategy is adopted. If the aspect ratio of the debris is greater than the second threshold, that is, F>F THRESHOLD In the case of , the dual-manipulator debris grasping strategy is executed, and the above-mentioned manipulators are controlled to grasp in a two-hand grasping mode according to the minimum safe working distance between the manipulators and the minimum distance between the debris. If the aspect ratio of the above-mentioned debris is less than or equal to the above-mentioned second threshold, that is, F≤F THRESHOLD In this case, a single-handed manipulator debris grasping strategy is executed, and the above-mentioned manipulator is controlled to grasp in a single-handed grasping mode according to the minimum safe working distance between the manipulators and the minimum distance between the debris.
[0110] Through this embodiment, the first acquisition unit acquires a first detection frame, a first segmentation mask, a target direction and a first distance, wherein the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between manipulators, and the target direction is a flow direction of coal; the first calculation unit screens the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, and determines the first target quantity, stacking complexity, debris aspect ratio and second distance at least based on the second detection frame and the second segmentation mask, wherein the second distance is the distance between the debris. the minimum distance between the first and second targets, the first target quantity is the quantity of the debris, the second detection frame and the second segmentation mask are the first detection frame and the first segmentation mask that are in the same direction as the target and overlap with the image frame; the second calculation unit determines a first threshold according to the first distance, calculates the distance of each of the debris according to the first segmentation mask, and determines a second threshold according to the distance of each of the debris, the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold; the second acquisition unit acquires the second target quantity, and when the first target quantity is greater than the second target quantity and the debris aspect ratio is greater than the second threshold, controls the manipulator according to the first distance The grasping is performed in a two-hand grasping mode, and the second target number is the total number of the manipulators; the first control unit controls the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold; the second control unit adjusts the first threshold until the third target number is greater than the second target number if the third target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is greater than the second target number. , at least according to the ratio of the fourth target number to the above-mentioned second target number, the above-mentioned first distance and the above-mentioned second distance, the above-mentioned manipulator controls the above-mentioned manipulator to grasp in the above-mentioned two-hand grasping mode or the above-mentioned one-hand grasping mode; the third control unit controls the above-mentioned manipulator to grasp in the above-mentioned two-hand grasping mode according to the above-mentioned first distance and the above-mentioned second distance when the above-mentioned first target number is less than or equal to the above-mentioned second target number and the above-mentioned stacking complexity is less than or equal to the above-mentioned first threshold, if the above-mentioned debris aspect ratio is greater than the above-mentioned second threshold, and controls the above-mentioned manipulator to grasp in the above-mentioned one-hand grasping mode according to the above-mentioned first distance and the above-mentioned second distance when the above-mentioned debris aspect ratio is less than or equal to the above-mentioned second threshold.The present application screens and removes redundant results based on the category and position detection results of the debris, and then determines the use of single-handed grasping strategy and two-handed grasping strategy based on the debris, the shape of the debris and the scene restrictions, such as the number of debris, the number of manipulators, and the scene stacking situation, to achieve automatic screening of debris that has a greater impact on subsequent work processes, solving the technical problem of low efficiency and poor precision of automatic sorting in the prior art.
[0111] The above-mentioned manipulator control device for selecting impurities in coal includes a processor and a memory, and the above-mentioned units are stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.
[0112] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the automatic sorting accuracy can be improved by adjusting the kernel parameters.
[0113] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0114] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the robot control method for picking debris in coal.
[0115] Specifically, the robot control method for selecting debris in coal includes:
[0116] Step S201, obtaining a first detection frame, a first segmentation mask, a target direction and a first distance, wherein the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between manipulators, and the target direction is a flow direction of the coal;
[0117] Step S202, screening the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, determining the first target quantity, stacking complexity, debris aspect ratio, and second distance based at least on the second detection frame and the second segmentation mask, the second distance being the minimum distance between the debris, the first target quantity being the quantity of the debris, and the second detection frame and the second segmentation mask being the first detection frame and the first segmentation mask having the same direction as the target and overlapping with the image border;
[0118] Step S203, determining a first threshold according to the first distance, calculating the distance of each of the debris according to the first segmentation mask, and determining a second threshold according to the distance of each of the debris, wherein the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold;
[0119] Step S204, obtaining a second target number, and when the first target number is greater than the second target number and the aspect ratio of the debris is greater than the second threshold, controlling the manipulator to grasp in a two-hand grasping mode according to the first distance, and the second target number is the total number of the manipulators;
[0120] Step S205, when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold, controlling the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance;
[0121] Step S206, when the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number, adjust the first threshold until the third target number is greater than the second target number, and if the third target number is greater than the second target number, control the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the fourth target number to the second target number, the first distance and the second distance;
[0122] Step S207, when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, if the aspect ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in a two-hand grasping mode according to the first distance and the second distance; if the aspect ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance.
[0123] An embodiment of the present invention provides a processor, which is used to run a program, wherein the robot control method for selecting debris in coal is executed when the program is run.
[0124] Specifically, the robot control method for selecting debris in coal includes:
[0125] Step S201, obtaining a first detection frame, a first segmentation mask, a target direction and a first distance, wherein the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between manipulators, and the target direction is a flow direction of the coal;
[0126] Step S202, screening the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, determining the first target quantity, stacking complexity, debris aspect ratio, and second distance based at least on the second detection frame and the second segmentation mask, the second distance being the minimum distance between the debris, the first target quantity being the quantity of the debris, and the second detection frame and the second segmentation mask being the first detection frame and the first segmentation mask having the same direction as the target and overlapping with the image border;
[0127] Step S203, determining a first threshold according to the first distance, calculating the distance of each of the debris according to the first segmentation mask, and determining a second threshold according to the distance of each of the debris, wherein the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold;
[0128] Step S204, obtaining a second target number, and when the first target number is greater than the second target number and the aspect ratio of the debris is greater than the second threshold, controlling the manipulator to grasp in a two-hand grasping mode according to the first distance, and the second target number is the total number of the manipulators;
[0129] Step S205, when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold, controlling the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance;
[0130] Step S206, when the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number, adjust the first threshold until the third target number is greater than the second target number, and if the third target number is greater than the second target number, control the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the fourth target number to the second target number, the first distance and the second distance;
[0131] Step S207, when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, if the aspect ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in a two-hand grasping mode according to the first distance and the second distance; if the aspect ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance.
[0132] An embodiment of the present invention provides a sundry item picking system, the sundry item picking system comprising a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:
[0133] Step S201, obtaining a first detection frame, a first segmentation mask, a target direction and a first distance, wherein the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between manipulators, and the target direction is a flow direction of the coal;
[0134] Step S202, screening the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, determining the first target quantity, stacking complexity, debris aspect ratio, and second distance based at least on the second detection frame and the second segmentation mask, the second distance being the minimum distance between the debris, the first target quantity being the quantity of the debris, and the second detection frame and the second segmentation mask being the first detection frame and the first segmentation mask having the same direction as the target and overlapping with the image border;
[0135] Step S203, determining a first threshold according to the first distance, calculating the distance of each of the debris according to the first segmentation mask, and determining a second threshold according to the distance of each of the debris, wherein the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold;
[0136] Step S204, obtaining a second target number, and when the first target number is greater than the second target number and the aspect ratio of the debris is greater than the second threshold, controlling the manipulator to grasp in a two-hand grasping mode according to the first distance, and the second target number is the total number of the manipulators;
[0137] Step S205, when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold, controlling the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance;
[0138] Step S206, when the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number, adjust the first threshold until the third target number is greater than the second target number, and if the third target number is greater than the second target number, control the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the fourth target number to the second target number, the first distance and the second distance;
[0139] Step S207, when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, if the aspect ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in a two-hand grasping mode according to the first distance and the second distance; if the aspect ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance.
[0140] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:
[0141] Step S201, obtaining a first detection frame, a first segmentation mask, a target direction and a first distance, wherein the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between manipulators, and the target direction is a flow direction of the coal;
[0142] Step S202, screening the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, determining the first target quantity, stacking complexity, debris aspect ratio, and second distance based at least on the second detection frame and the second segmentation mask, the second distance being the minimum distance between the debris, the first target quantity being the quantity of the debris, and the second detection frame and the second segmentation mask being the first detection frame and the first segmentation mask having the same direction as the target and overlapping with the image border;
[0143] Step S203, determining a first threshold according to the first distance, calculating the distance of each of the debris according to the first segmentation mask, and determining a second threshold according to the distance of each of the debris, wherein the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold;
[0144] Step S204, obtaining a second target number, and when the first target number is greater than the second target number and the aspect ratio of the debris is greater than the second threshold, controlling the manipulator to grasp in a two-hand grasping mode according to the first distance, and the second target number is the total number of the manipulators;
[0145] Step S205, when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold, controlling the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance;
[0146] Step S206, when the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number, adjust the first threshold until the third target number is greater than the second target number, and if the third target number is greater than the second target number, control the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the fourth target number to the second target number, the first distance and the second distance;
[0147] Step S207, when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, if the aspect ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in a two-hand grasping mode according to the first distance and the second distance; if the aspect ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance.
[0148] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0149] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0153] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0154] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0155] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0156] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0157] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:
[0158] 1) The robot control method for picking debris in coal of the present application first obtains a first detection frame, a first segmentation mask, a target direction and a first distance, wherein the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between the robots, and the target direction is a flow direction of the coal; then, the first detection frame and the first segmentation mask are screened based on the target direction to obtain a second detection frame and a second segmentation mask, and the first target quantity, stacking complexity, debris aspect ratio and second distance are determined at least based on the second detection frame and the second segmentation mask. The second distance is the minimum distance between the above-mentioned debris, the first target number is the number of debris, the second detection frame and the second segmentation mask are the first detection frame and the first segmentation mask that are in the same direction as the above-mentioned target and overlap with the image border; then, the first threshold is determined according to the first distance, the distance of each of the above-mentioned debris is calculated according to the first segmentation mask, and the second threshold is determined according to the distance of each of the above-mentioned debris, the first threshold is the stacking complexity threshold, and the second threshold is the debris aspect ratio threshold; then, the second target number is obtained, and when the first target number is greater than the second target number and the debris aspect ratio is greater than the second threshold, the first distance is used to calculate the first threshold. The manipulator is controlled to grasp in a two-hand grasping mode, and the second target number is the total number of the manipulators; then, when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in a single-hand grasping mode according to the first distance and the second distance; then, when the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number, adjust the first threshold until the third target number is greater than the second target number, if the third target number is greater than the second target number quantity, at least according to the ratio of the fourth target quantity to the second target quantity, the first distance and the second distance, the manipulator is controlled to grasp in the two-hand grasping mode or the one-hand grasping mode; finally, when the first target quantity is less than or equal to the second target quantity and the stacking complexity is less than or equal to the first threshold, if the aspect ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in the two-hand grasping mode according to the first distance and the second distance; if the aspect ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in the one-hand grasping mode according to the first distance and the second distance.The present application screens and removes redundant results based on the category and position detection results of the debris, and then determines the use of single-handed grasping strategy and two-handed grasping strategy based on the debris, the shape of the debris and the scene restrictions, such as the number of debris, the number of manipulators, and the scene stacking situation, to achieve automatic screening of debris that has a greater impact on subsequent work processes, solving the technical problem of low efficiency and poor precision of automatic sorting in the prior art.
[0159] 2) The robot control device for picking debris in coal of the present application, the first acquisition unit acquires a first detection frame, a first segmentation mask, a target direction and a first distance, the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between robots, and the target direction is a flow direction of coal; the first calculation unit screens the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, and determines the first target quantity, stacking complexity, debris aspect ratio and second distance at least based on the second detection frame and the second segmentation mask. , the second distance is the minimum distance between the above-mentioned debris, the first target number is the number of debris, the second detection frame and the second segmentation mask are the first detection frame and the first segmentation mask that are in the same direction as the above-mentioned target and overlap with the image frame; the second calculation unit determines a first threshold according to the first distance, calculates the distance of each of the above-mentioned debris according to the first segmentation mask and determines a second threshold according to the distance of each of the above-mentioned debris, the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold; the second acquisition unit acquires the second target number, and when the first target number is greater than the second target number and the debris aspect ratio is greater than the second threshold, the second detection frame and the second segmentation mask are determined according to the first distance. The first control unit controls the manipulator to grasp in a two-hand grasping mode, and the second target number is the total number of the manipulators; the first control unit controls the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold; the second control unit controls the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance when the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number, adjusts the first threshold until the third target number is greater than the second target number, and if the third target number is greater than the second target number the third control unit controls the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the fourth target number to the second target number, the first distance and the second distance; the third control unit controls the manipulator to grasp in the two-hand grasping mode according to the first distance and the second distance when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, if the aspect ratio of the debris is greater than the second threshold, and controls the manipulator to grasp in the one-hand grasping mode according to the first distance and the second distance when the aspect ratio of the debris is less than or equal to the second threshold.The present application screens and removes redundant results based on the category and position detection results of the debris, and then determines the use of single-handed grasping strategy and two-handed grasping strategy based on the debris, the shape of the debris and the scene restrictions, such as the number of debris, the number of manipulators, and the scene stacking situation, to achieve automatic screening of debris that has a greater impact on subsequent work processes, solving the technical problem of low efficiency and poor precision of automatic sorting in the prior art.
[0160] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A robot control method for selecting debris in coal, characterized in that: include: Acquire a first detection frame, a first segmentation mask, a target direction, and a first distance, wherein the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between manipulators, and the target direction is a flow direction of the coal; The first detection frame and the first segmentation mask are screened based on the target direction to obtain a second detection frame and a second segmentation mask, and the first target quantity, stacking complexity, debris aspect ratio, and second distance are determined based at least on the second detection frame and the second segmentation mask, where the second distance is the minimum distance between the debris, the first target quantity is the quantity of the debris, and the second detection frame and the second segmentation mask are the first detection frame and the first segmentation mask that have the same direction as the target and overlap with the image border; Determine a first threshold according to the first distance, calculate the distance of each of the debris according to the first segmentation mask, and determine a second threshold according to the distance of each of the debris, the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold; Acquire a second target quantity, and when the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, control the manipulator to grasp in a two-hand grasping mode according to the first distance, and the second target quantity is the total quantity of the manipulator; When the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold, controlling the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance; In the case where the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, if the third target number is less than or equal to the second target number, adjusting the first threshold until the third target number is greater than the second target number, and if the third target number is greater than the second target number, controlling the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to a ratio of the fourth target number to the second target number, the first distance, and the second distance; When the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold, if the length-to-width ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in a two-handed grasping mode according to the first distance and the second distance; if the length-to-width ratio of the debris is less than or equal to the second threshold, the manipulator is controlled to grasp in a single-handed grasping mode according to the first distance and the second distance.
2. The method according to claim 1, characterized in that Determining a stacking complexity based on the second detection box and the second segmentation mask includes: Determine the corresponding geometric center according to the second detection frame to obtain a first target point, and determine the corresponding coordinates according to the first target point to obtain first target coordinates; Determine an intersection-and-union ratio with other second detection frames according to any second detection frame to obtain a plurality of target intersection-and-union ratios, and determine a minimum distance according to the first target coordinates corresponding to the second detection frame and the first target coordinates corresponding to other second detection frames to obtain a third distance; Substituting the target intersection-and-union ratio and the third distance corresponding to each of the first target points into the first target formula respectively to obtain a plurality of candidate stacking complexities; A maximum value of a plurality of the candidate stacking complexities is determined as the stacking complexity.
3. The method according to claim 1, characterized in that Determining the aspect ratio of the debris based at least on the second detection frame and the second segmentation mask includes: Calculating the area of the debris according to the second detection frame to obtain a first area, and calculating the area of the debris according to the second segmentation mask to obtain a second area; Acquire a target length and a target width, wherein the target length is the length of the second detection frame, and the target width is the width of the second detection frame; Substituting the first area, the second area, the target length and the target width into a second target formula obtains the debris aspect ratio.
4. The method according to claim 2, characterized in that: Determining a second distance based at least on the second detection box and the second segmentation mask includes: Calculate the zero-order moment and the first-order moment of the corresponding debris according to the second segmentation mask to obtain a first target moment and a second target moment; Substituting the first target moment and the second target moment into the second target formula, determining the center of gravity coordinates of the debris, and obtaining the second target coordinates; Determine a third detection frame according to the second target coordinates and the second detection frame, wherein the third detection frame is the second detection frame having the shortest distance from the second target coordinates; Determine a third target coordinate and a fourth target coordinate based on the first target coordinate of the third detection frame, wherein the third target coordinate and the fourth target coordinate are the centroid coordinates of the two debris with the smallest distance from the first target coordinate; A set distance is calculated based on the second target coordinate, the third target coordinate and the fourth target coordinate, respectively, and a minimum value is determined as the second distance.
5. The method according to claim 4, characterized in that When the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, controlling the manipulator to grasp in a two-hand grasping mode according to the first distance includes: Extracting by a backbone extraction algorithm according to the second segmentation mask to obtain a target backbone; Determine the point with the smallest distance between the target backbone and the second target coordinates as the second target point; Draw a circle with the second target point as the center and the first distance as the diameter, and determine the intersection of the circle and the target backbone as the grasping point; Connecting lines based on the grasping points, and determining a direction perpendicular to the lines as a grasping direction; The manipulator is controlled to grasp based on the grasping point and the grasping direction.
6. The method according to claim 5, characterized in that Controlling the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance includes: When the first distance is greater than or equal to the second distance, draw a circle with the second target coordinate as the center and the first distance as the diameter, determine any intersection of the circle and the target backbone as a grasping point, draw a line based on the grasping point, determine the direction perpendicular to the line as the grasping direction, and control the manipulator to grasp based on the grasping point and the grasping direction; When the first distance is smaller than the second distance, a circle is drawn with the third target coordinate or the fourth target coordinate as the center and the first distance as the diameter, and any intersection of the circle and the target backbone is determined as a grasping point. A line is drawn based on the grasping point, and a direction perpendicular to the line is determined as the grasping direction. The manipulator is controlled to grasp based on the grasping point and the grasping direction.
7. The method according to claim 6, characterized in that If the third target quantity is greater than the second target quantity, controlling the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to the ratio of the fourth target quantity to the second target quantity, the first distance, and the second distance, includes: When the ratio of the first target data to the second target number is less than a third threshold, if the debris is hard debris, the manipulator is controlled to grasp it in the two-hand grasping mode according to the first distance; if the debris is not hard debris, the manipulator is controlled to grasp it in the one-hand grasping mode according to the first distance and the second distance; When the ratio of the first target data to the second target quantity is greater than or equal to the third threshold, if the debris is the hard debris and the debris aspect ratio is greater than the second threshold, the manipulator is controlled to grasp in the two-hand grasping mode according to the first distance; if the debris is the hard debris and the debris aspect ratio is less than or equal to the second threshold, the manipulator is controlled to grasp in the one-hand grasping mode according to the first distance and the second distance; if the debris is not the hard debris, no grasping is performed.
8. A manipulator control device for selecting debris in coal, characterized in that: The device comprises: a first acquisition unit, configured to acquire a first detection frame, a first segmentation mask, a target direction, and a first distance, wherein the first detection frame is a target detection frame corresponding to the debris in the detection image, the first segmentation mask is an instance segmentation mask corresponding to the debris, the first distance is a minimum safe working distance between manipulators, and the target direction is a flow direction of the coal; a first calculation unit, configured to screen the first detection frame and the first segmentation mask based on the target direction to obtain a second detection frame and a second segmentation mask, and determine a first target quantity, a stacking complexity, an object aspect ratio, and a second distance based at least on the second detection frame and the second segmentation mask, wherein the second distance is a minimum distance between the objects, the first target quantity is the quantity of the objects, and the second detection frame and the second segmentation mask are the first detection frame and the first segmentation mask having the same direction as the target and overlapping with the image border; a second calculation unit, configured to determine a first threshold according to the first distance, calculate the distance of each of the debris according to the first segmentation mask, and determine a second threshold according to the distance of each of the debris, wherein the first threshold is a stacking complexity threshold, and the second threshold is a debris aspect ratio threshold; A second acquisition unit is used to acquire a second target quantity, and when the first target quantity is greater than the second target quantity and the aspect ratio of the debris is greater than the second threshold, the manipulator is controlled to grasp in a two-hand grasping mode according to the first distance, and the second target quantity is the total quantity of the manipulator; a first control unit, configured to control the manipulator to grasp in a single-hand grasping mode according to the first distance and the second distance when the first target number is greater than the second target number and the aspect ratio of the debris is less than or equal to the second threshold; a second control unit, configured to, when the first target number is less than or equal to the second target number and the stacking complexity is greater than the first threshold, adjust the first threshold until the third target number is greater than the second target number if the third target number is less than or equal to the second target number, and, if the third target number is greater than the second target number, control the manipulator to grasp in the two-hand grasping mode or the one-hand grasping mode at least according to a ratio of a fourth target number to the second target number, the first distance, and the second distance; A third control unit is used to control the manipulator to grasp in a two-handed grasping mode according to the first distance and the second distance when the first target number is less than or equal to the second target number and the stacking complexity is less than or equal to the first threshold and if the length-to-width ratio of the debris is greater than the second threshold; and to control the manipulator to grasp in a single-handed grasping mode according to the first distance and the second distance when the length-to-width ratio of the debris is less than or equal to the second threshold.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.
10. A debris picking system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.
Citation Information
Patent Citations
Intelligent sorting system and method for sundries in coal based on deep learning
CN113289925A
Automatic coal gangue sorting method and system based on image recognition and medium
CN114904798A
Visual tracking-based on-sieve impurity real-time monitoring method
CN117314962A
Sundry sorting method and device, computer program product and sorting system
CN118417192A
Method, apparatus, and computer program product for automatically performing sorting operations
EP4011509A2
