Material sorting method, system, equipment and medium based on robotic arm

Abstract

The present application discloses a material sorting method, system, equipment and medium based on a robotic arm, and the technical field to which it belongs is robotic arm control technology. The material sorting method based on the robotic arm includes: obtaining a current scene image of the working area, and obtaining material description information of the target material; segmenting the current scene image according to the material description information to obtain a material area and a material segmentation mask; performing posture estimation according to the position of the material area and the material segmentation mask to obtain a grasping posture; obtaining a standard posture of the target material, and determining the grasping position of the end effector for the target material according to the grasping posture and the standard posture; controlling the robotic arm to grasp the grasping position of the target material with the grasping posture, adjusting the posture of the end effector so that the grasped target material is in the standard posture, and performing a sorting operation on the target material. The present application can enable the robotic arm to adapt to complex sorting scenarios and improve the working accuracy and efficiency of the robotic arm.

Description

Technical Field

[0001] The present application relates to the field of robotic arm control technology, and in particular to a material sorting method, system, equipment and medium based on a robotic arm. Background Art

[0002] A robotic arm is an automated device that can imitate the movements of a human arm. It achieves precise movement and operation through programmed control and is widely used in manufacturing, logistics and other fields.

[0003] In related technologies, robotic arms typically follow a fixed trajectory to repeatedly pick up and release materials. When materials are scattered in an open and flexible environment, the robotic arms cannot accurately detect and grasp the materials, resulting in low accuracy and efficiency. In response to this situation, the field typically relies on manual labor to complete sorting tasks, a process that is not only labor-intensive but also lacks a high degree of intelligence.

[0004] Therefore, how to make the robotic arm adapt to complex sorting scenarios and improve the working accuracy and efficiency of the robotic arm is a technical problem that technical personnel in this field currently need to solve. Summary of the Invention

[0005] The purpose of this application is to provide a material sorting method, system, electronic equipment and storage medium based on a robotic arm, which can enable the robotic arm to adapt to complex sorting scenarios and improve the working accuracy and efficiency of the robotic arm.

[0006] To solve the above technical problems, the present application provides a material sorting method based on a robotic arm, which includes:

[0007] Get the current scene image of the working area and obtain the material description information of the target material;

[0008] Segment the current scene image according to the material description information to obtain a material region where the target material is located and a material segmentation mask corresponding to the material region; wherein the material region is the region where a single target material is located in the current scene image;

[0009] Performing posture estimation based on the position of the material area and the material segmentation mask to obtain a grasping posture of the end effector of the robotic arm when grasping the target material;

[0010] Acquiring a standard posture of the target material, and determining a grasping position of the end effector on the target material according to the grasping posture and the standard posture;

[0011] The robot arm is controlled to grasp the grasping position of the target material in the grasping posture, the posture of the end effector is adjusted so that the grasped target material is in the standard posture, and a sorting operation is performed on the target material.

[0012] Optionally, before obtaining the current scene image of the working area, the following is also included:

[0013] If there is material stacking and / or material obstruction in the initial scene image of the working area, a motion path of the robotic arm is generated, and the robotic arm is controlled to move along the motion path to break up the material until there is no material stacking and / or material obstruction in the current scene image of the working area; wherein the motion path passes through an area where there is material stacking and / or material obstruction.

[0014] Optionally, the initial scene image includes a color image and a depth image;

[0015] Generating the motion path of the robotic arm includes:

[0016] Inputting the initial scene image into a path generation model so that the path generation model identifies areas in the initial scene image where materials are stacked and / or blocked by materials and outputs a plurality of action sequences; wherein the path generation model is an end-to-end trained network model, and the action sequences include a plurality of joint values;

[0017] The motion path is generated according to all the action sequences.

[0018] Optionally, the standard posture is determined in the following manner:

[0019] Acquiring a three-dimensional geometric model of the target material and structural information of the robotic arm;

[0020] The standard posture of the target material is determined according to the three-dimensional geometric model of the target material and the structural information of the robotic arm.

[0021] Optionally, performing posture estimation based on the position of the material area and the material segmentation mask to obtain a grasping posture of the end effector of the robot arm when grasping the target material includes:

[0022] Determining contour information of the target material according to the material segmentation mask;

[0023] Acquire a three-dimensional geometric model of the target material, and determine a current posture of the target material according to the contour information and the three-dimensional geometric model;

[0024] Determining the size parameters of the target material according to the position of the material area and the contour information;

[0025] The gripping posture of the end effector of the robot arm when gripping the target material is calculated according to the current posture and size parameters of the target material.

[0026] Optionally, obtaining a standard posture of the target material, and determining a grasping position of the end effector for the target material according to the grasping posture and the standard posture, includes:

[0027] Determining a three-dimensional geometric model of the target material, and determining a graspable area of ​​the target material based on the three-dimensional geometric model;

[0028] Selecting a first contact point set matching the grasping posture from the graspable area; wherein the first contact point set includes contact points between the end effector and the target material when the end effector grasps the target material in the grasping posture;

[0029] Selecting a second contact point set matching the standard posture from the graspable area; wherein the second contact point set includes contact points between the end effector and the target material when the target material is in the standard posture;

[0030] The gripping position of the end effector for the target material is determined according to the intersection of the first contact point set and the second contact point set.

[0031] Optionally, performing a sorting operation on the target material includes:

[0032] Get the slot description information of the target slot;

[0033] Segmenting the current scene image according to the slot description information to obtain a slot area where the target slot is located and a slot segmentation mask corresponding to the slot area;

[0034] Performing posture estimation based on the position of the slot area and the slot segmentation mask to obtain a placement posture of the end effector when placing the target material;

[0035] Based on the placement posture, the robotic arm is controlled to place the target material into the target slot.

[0036] Optionally, performing posture estimation based on the position of the slot area and the slot segmentation mask to obtain a placement posture of the end effector when placing the target material includes:

[0037] Determining contour information of the target slot according to the material segmentation mask;

[0038] Acquire a three-dimensional geometric model of the target slot, and determine a current posture of the target slot according to the contour information and the three-dimensional geometric model;

[0039] The placement posture of the end effector of the robotic arm when placing the target material is calculated according to the position of the slot area and the current posture of the target slot.

[0040] Optionally, obtaining material description information of the target material includes:

[0041] Determine the target frame added by the user in the current scene image;

[0042] Inputting the image in the target frame into an object recognition model to obtain a recognition result;

[0043] Generate material description information of the target material according to the recognition result.

[0044] This application also provides a material sorting system based on a robotic arm, comprising:

[0045] The information acquisition module is used to obtain the current scene image of the working area and obtain the material description information of the target material;

[0046] an image processing module, configured to segment the current scene image according to the material description information to obtain a material region where the target material is located and a material segmentation mask corresponding to the material region; wherein the material region is the region where a single target material is located in the current scene image;

[0047] a posture estimation module, configured to perform posture estimation based on the position of the material area and the material segmentation mask, and obtain a grasping posture of the end effector of the robotic arm when grasping the target material;

[0048] a position determination module, configured to obtain a standard posture of the target material and determine a grasping position of the end effector on the target material according to the grasping posture and the standard posture;

[0049] The robot arm control module is used to control the robot arm to grasp the grasping position of the target material in the grasping posture, adjust the posture of the end effector so that the grasped target material is in the standard posture, and perform a sorting operation on the target material.

[0050] The present application also provides a storage medium on which a computer program is stored. When the computer program is executed, the steps of the material sorting method based on the above-mentioned robotic arm are implemented.

[0051] The present application also provides an electronic device including a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the material sorting method based on the above-mentioned robotic arm are implemented.

[0052] This application provides a material sorting method implemented using a robotic arm. This method segments a current scene image of a work area based on the material description information of the target material, obtaining material regions and material segmentation masks to facilitate identification of the target material in a complex environment. Based on the location of the material region and the segmentation mask, posture estimation is performed to determine the robotic arm's posture during grasping, i.e., the grasping posture. This application also obtains the target material's standard posture and, in combination with the grasping posture, determines the final grasping position. During the sorting task, the robotic arm grasps the target material at the grasping position using the grasping posture, adjusts the posture of the end effector to maintain the grasped target material in the standard posture, and completes the sorting operation. This solution can grasp materials in any posture. Even if materials are scattered in an open and flexible work area, the robotic arm can adaptively adjust the grasping posture and grasping position to achieve reliable material grasping. Therefore, this application enables the robotic arm to adapt to complex sorting scenarios and improves the robotic arm's operating accuracy and efficiency. This application also provides a material sorting system, a storage medium, and an electronic device implemented using the robotic arm, all of which have the aforementioned beneficial effects and are not further described here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 A flowchart of a material sorting method based on a robotic arm provided in an embodiment of the present application;

[0055] Figure 2 A top view of a working scene of a robotic arm provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of the principle of a vision-based material sorting method provided in an embodiment of the present application;

[0057] Figure 4 A schematic structural diagram of a material sorting system based on a robotic arm provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0059] In traditional industrial assembly lines, robotic arms usually perform material sorting operations in a closed environment according to predefined trajectories. When materials are scattered in an open and flexible environment, the robotic arm cannot accurately detect and grab the materials. In order to enable the robotic arm to adapt to complex sorting scenarios, the embodiment of the present application provides a material sorting method based on a robotic arm. Through this method, the robotic arm can complete sorting tasks more flexibly and adapt to more complex production needs. The above-mentioned material sorting method based on the robotic arm may include Figure 1 Steps shown:

[0060] S101: Acquire the current scene image of the working area and obtain the material description information of the target material;

[0061] This embodiment can be applied to a control device for a robotic arm, which processes information about the working area to control the robotic arm to sort target materials. An imaging device can be provided within the working area of ​​the robotic arm, and this step can capture an image of the current scene in the working area captured by the imaging device.

[0062] This step can also obtain material description information of the target material. The above-mentioned material description information can be information describing the appearance, and / or function, and / or type of the target material, such as "round parts", "metal round parts", "white round material blocks", "bearings", "power supply devices", etc.

[0063] The material description information may be information manually input by the user.

[0064] This embodiment can also utilize an object recognition model to automatically acquire material description information. The process is as follows: determining a target frame added by the user to the current scene image; inputting the image within the target frame into the multimodal large model to obtain a recognition analysis result; and generating a material description of the target material based on the recognition analysis result. During this process, the user simply adds a target frame to be identified within the current scene image. Object recognition analysis is performed on the image within the target frame to obtain a recognition analysis result, and then material description information is generated based on the appearance characteristics of the recognition analysis result. For example, if the recognition result is a tennis ball, the appearance characteristics of the tennis ball can be analyzed to generate a material description, such as yellow-green, spherical, and 6-7 cm in diameter.

[0065] S102: Segmenting the current scene image according to the material description information to obtain a material region where the target material is located and a material segmentation mask corresponding to the material region;

[0066] In this embodiment, the material description information and the current scene image can be input into an image segmentation model to obtain the material region where the target material is located. A mask representing the material region can also be used as a material segmentation mask. This process can extract multiple material regions, each representing the region within the current scene image where a single target material is located. Specifically, this embodiment can process the current scene image using a multimodal detection segmentation model to obtain the material regions and their corresponding material segmentation masks.

[0067] This embodiment uses material description information to segment the current scene image. When new materials are introduced into the scene, there is no need to retrain the corresponding recognition model, which saves computing resources and improves the adaptability of this solution.

[0068] S103: performing posture estimation based on the position of the material area and the material segmentation mask to obtain a grasping posture of the end effector of the robot arm when grasping the target material;

[0069] Among them, the grasping posture is the posture of the robotic arm when the robotic arm grasps the target material. Specifically, this step can use the segmentation mask to locate the outline of the target material, and then combine the position information of the material area to analyze the posture of the target material through a computer vision algorithm or a deep learning model. By combining the posture of the target material with the kinematic model of the robotic arm, the posture of the robotic arm end effector that is most suitable for grasping the target material, that is, the grasping posture, can be calculated. Specifically, this embodiment can input the position of the material area and the material segmentation mask into a 6D (degree of freedom) object posture estimation model for posture estimation, and obtain the grasping posture of the robotic arm end effector when grasping the target material.

[0070] In an open and flexible work area, there is uncertainty in the size, contour, placement, and posture of materials. When the robot arm needs to grab multiple scattered materials from the work area, or when there are strict requirements on the orientation of the materials (such as the positive and negative poles of the battery) during the sorting process, the traditional predetermined grasping posture is often not applicable. In the above cases, fixed sorting strategies are difficult to cope with the random changes of materials, thereby limiting the operational flexibility and efficiency of the robot arm. This application performs posture estimation based on the position of the material area and the material segmentation mask to obtain the grasping posture of the robot arm during grasping, ensuring that the robot arm can complete tasks efficiently and accurately in complex and changing working environments.

[0071] S104: Acquire a standard posture of the target material, and determine a grasping position of the end effector for the target material according to the grasping posture and the standard posture;

[0072] The standard posture is the ideal posture that the target material is expected to achieve after being grasped. The standard posture of the target material can be stored in the knowledge base. In this step, the standard posture of the target material can be obtained from the knowledge base.

[0073] After obtaining the target material's standard posture, this standard posture can be compared with the grasping posture to calculate the contact point of the robot arm's end effector relative to the target material, thereby determining the grasping position. The grasping position is the point on the target material where the robot arm contacts the target material.

[0074] S105: Control the robotic arm to grasp the grasping position of the target material in the grasping posture, adjust the posture of the end effector so that the grasped target material is in the standard posture, and perform a sorting operation on the target material.

[0075] Based on the calculated grasping position and grasping posture, corresponding control instructions can be generated and sent to the robotic arm, so that the robotic arm grasps the target material according to the grasping position and grasping posture. After successfully grasping the material, the posture of the end effector can be adjusted so that the grasped target material is in the standard posture, and a sorting operation can be performed on the target material in the standard posture.

[0076] This embodiment segments the current scene image of the work area based on the target material's material description information, obtaining a material region and a material segmentation mask to facilitate identification of the target material within a complex environment. Based on the location of the material region and the segmentation mask, posture estimation is performed to determine the robotic arm's grasping posture, i.e., the grasping posture. This embodiment also obtains the target material's standard posture and, in combination with the grasping posture, determines the final grasping position. During the sorting process, the robotic arm grasps the target material at the grasping position using the grasping posture, adjusts the posture of the end effector to bring the grasped target material into the standard posture, and thus completes the sorting operation. This solution is capable of grasping materials in any posture. Even if materials are scattered in an open and flexible work area, the robotic arm can adaptively adjust its grasping posture and grasping position to achieve reliable material grasping. Therefore, this embodiment enables the robotic arm to adapt to complex sorting scenarios, improving its operational accuracy and efficiency.

[0077] As for Figure 1Further introduction of the corresponding embodiment: when the target materials are scattered in an open and flexible working area, there are materials stacked and / or materials blocking each other. The robot arm may collide with the surrounding materials during sorting, which limits its operating space and reduces its flexibility. In view of the above situation, this embodiment can first start the material scattering function of the robot arm, and then execute Figure 1 Corresponding embodiment. As can be seen, this embodiment not only adaptively adjusts the gripping posture according to the environment, but also effectively handles issues such as material stacking and occlusion. The robotic arm in this solution has greater flexibility and applicability for sorting tasks in industrial scenarios, achieving a high degree of automation.

[0078] Specifically, before obtaining the current scene image of the working area, an initial scene image of the working area can be obtained first. If there is material stacking and / or material obstruction in the initial scene image of the working area, a motion path of the robotic arm is generated, and the robotic arm is controlled to move according to the motion path to break up the material until there is no material stacking and / or material obstruction in the current scene image of the working area; wherein, the motion path passes through an area where material stacking and / or material obstruction exists.

[0079] By implementing the above solution, automatic separation of materials can be achieved. This process not only improves the sorting efficiency, but also greatly enhances the flexibility and adaptability of the robotic arm in complex and changing working environments, ensuring the stability and reliability of operations.

[0080] The above-mentioned initial scene image may include a color image and a depth image. In this embodiment, the color image and the depth image may be processed by a path generation model to obtain a motion path. Specifically, in this embodiment, the initial scene image may be input into a path generation model so that the path generation model identifies areas in the initial scene image where materials are stacked and / or blocked and outputs a plurality of action sequences; the motion path is generated according to all the action sequences. The above-mentioned path generation model is an end-to-end trained network model, and the action sequence includes a plurality of joint values. This path generation model enables the robotic arm to have the ability to separate parts, thereby reducing the interference of material stacking and / or material blocking on the sorting operation.

[0081] As for Figure 1Further describing the corresponding embodiment, the standard posture can be determined by a robotic arm or other device. The process of determining the standard posture includes the following operations: obtaining a three-dimensional geometric model of the target material and structural information of the robotic arm; and determining the standard posture of the target material based on the three-dimensional geometric model of the target material and the structural information of the robotic arm. The three-dimensional geometric model can be determined based on a CAD (Computer Aided Design) file for the material.

[0082] As for Figure 1 For further description of the corresponding embodiment, the process of determining the grasping posture includes the following steps:

[0083] Step A1: Determine the contour information of the target material according to the material segmentation mask.

[0084] Step A2: Acquire a three-dimensional geometric model of the target material, and determine the current posture of the target material according to the contour information and the three-dimensional geometric model.

[0085] Among them, the three-dimensional geometric model and posture of the target material jointly affect the contour information. This application matches the contour information with the three-dimensional geometric model to determine the current posture of the target material.

[0086] Step A3: Determine the size parameters of the target material according to the position of the material area and the contour information.

[0087] Based on the position and contour information of the material region in the image, this embodiment can calculate the size parameters of the target material. The size parameters may include but are not limited to the length, width, height, and center of gravity of the material.

[0088] Step A4: Calculating the gripping posture of the end effector of the robot arm when gripping the target material according to the current posture and size parameters of the target material.

[0089] This step calculates the grasping posture of the end effector of the robot arm based on the determined current posture and size parameters, so that the robot arm can efficiently complete the grasping task in a complex working environment.

[0090] As for Figure 1 For further description of the corresponding embodiment, the process of determining the grasping position includes the following steps:

[0091] Step B1: Determine a three-dimensional geometric model of the target material, and determine a graspable area of ​​the target material based on the three-dimensional geometric model.

[0092] Based on this three-dimensional model, this embodiment can simulate which parts are suitable for being contacted and grasped by the end effector of the robot arm, that is, the graspable area.

[0093] Step B2: Selecting a first contact point set matching the grasping posture from the graspable area.

[0094] The first contact point set includes contact points between the end effector and the target material when the end effector grasps the target material in the grasping posture.

[0095] Step B3: Selecting a second set of contact points matching the standard posture from the graspable area.

[0096] Among them, the second contact point set includes the contact points between the end effector and the target material when the target material is in the standard posture. The contact points in the above second contact point set are the contact points between the end effector and the target material when the robotic arm grasps the target material and the target material is in the standard posture.

[0097] Step B4: determining the gripping position of the end effector for the target material according to the intersection of the first contact point set and the second contact point set.

[0098] In this step, the first contact point set and the second contact point set are compared to find the intersection between the two. The intersection represents the contact points that can satisfy the current gripping posture and ensure that the material reaches the standard posture after being gripped.

[0099] See also Figure 2 , Figure 2 This is a top view of a working scene of a robotic arm provided by an embodiment of the present application. In the figure, multiple No. 5 batteries, such as 201, 202, 203, and 204, are scattered in the working area. Battery 201 is placed vertically, battery 202 is placed at an angle, and batteries 203 and 204 are placed horizontally. Through this solution, a corresponding grasping posture can be generated for each battery so that the robotic arm can effectively grasp the battery. Battery 203 and battery 204 have the same posture. The distance between battery 203 and the robotic arm is 50 cm, the distance between battery 204 and the robotic arm is 52 cm, the length of the battery is 5 cm, and the maximum grasping distance of the robotic arm is 55 cm. In order to ensure that the robotic arm grasps the battery and does not fall off the robotic arm when the battery is in a standard posture, the contact point between the robotic arm end effector and the battery is specified to be in the area of ​​40% to 60% of the battery length. Therefore, the robotic arm can use the point on battery 203 that is 52.5 cm away from the robotic arm as the grasping position D1 and the point on battery 204 that is 54.5 cm away from the robotic arm as the grasping position D2.

[0100] As for Figure 1Further introduction to the corresponding embodiment: some types of materials need to be placed in slots after sorting, such as batteries need to be placed in slots of appropriate size. Therefore, the process of controlling the robot arm to perform sorting operations on the target material includes: obtaining slot description information of the target slot; segmenting the current scene image according to the slot description information to obtain the slot area where the target slot is located and the slot segmentation mask corresponding to the slot area; performing posture estimation based on the position of the slot area and the slot segmentation mask to obtain the placement posture of the end effector when placing the target material; and controlling the robot arm to place the target material into the target slot based on the placement posture. The above placement posture is the posture of the robot arm when the robot arm places the target material. The above process realizes efficient control from grasping to placement, ensuring the safety and reliability of the operation.

[0101] As for Figure 1 For further description of the corresponding embodiment, the process of determining the placement posture includes the following steps:

[0102] Step C1: determining the contour information of the target slot according to the material segmentation mask.

[0103] Step C2: Acquire a three-dimensional geometric model of the target slot, and determine a current posture of the target slot according to the contour information and the three-dimensional geometric model.

[0104] Step C3: Calculating the placement posture of the end effector of the robot arm when placing the target material according to the position of the slot area and the current posture of the target slot.

[0105] The above operations ensure that the robotic arm can complete the placement task efficiently and accurately in a complex working environment, while avoiding placement failure or material damage caused by incorrect posture.

[0106] As a feasible implementation method, some types of materials can be directly put into a specified range after sorting. Therefore, the process of controlling the robotic arm to perform sorting operations on the target materials based on the grasping position and the grasping posture includes: controlling the robotic arm to grasp the target material based on the grasping position and the grasping posture; controlling the robotic arm to move above the target container, and controlling the end effector of the robotic arm to relax so that the target material falls into the target container.

[0107] See Figure 3 , Figure 3This is a schematic diagram illustrating the principles of a vision-based material sorting method provided in an embodiment of the present application. The input information includes the material's CAD file S1 (including the material's 3D geometric model), material description information T1, a color image P1 (e.g., an RGB image), a depth image P2, and a knowledge base K1. The models used include a multimodal detection and segmentation model M1, a multimodal 6D pose estimation model M2, and a separation skill operation model (i.e., a path generation model with separation skills) M3. The multimodal detection and segmentation model M1 is used to generate a target area P3 and a segmentation mask P4. The multimodal 6D pose estimation model M2 is used to generate a grasping posture Z1 and a placement posture Z2. The separation skill operation model M3 is used to generate a motion trajectory A1. Based on the grasping posture Z1 and the placement posture Z2, the robotic arm can perform sorting and loading and unloading. Based on the motion trajectory A1, the material can be dispersed. The target area P3 can be a material area or a slot area; the segmentation mask P4 can be a material segmentation mask or a slot segmentation mask.

[0108] The vision-based material sorting method in this embodiment includes the following steps:

[0109] Step 1: Capture a scene through a sensing element to obtain a color image P1 and a depth image P2, where the color image P1 and the depth image P2 have been registered.

[0110] Step 2: Express the material that needs to be operated manually to obtain the material description information T1 of the material in the operation scenario.

[0111] Step 3: Obtain the material's CAD file S1 from the material's processing side.

[0112] Step 4: Design knowledge about the material. Refer to the structure and posture representation of the end effector of the robot arm and define the orientation of the material itself as information in the knowledge base K1. This orientation satisfies the execution of downstream tasks after the robot arm grasps it.

[0113] Step 5: Feed the color image P1 and depth image P2 into the separation skill operation model M3.

[0114] The separation skill operation model M3 is an end-to-end training network model that can directly output the motion trajectory of the robot arm, which is recorded as , where n is the predicted n-th action position of the robot arm; the action action is represented by constitute, denoted as the i-th joint value of the robotic arm. By collecting data on material separation actions, the separation skill model M3 is trained to predict the motion trajectory and guide the robotic arm to perform the material separation operation. This operation enables preliminary material separation, avoiding collisions and interference during subsequent grasping.

[0115] Step 6: Send the material description information T1 and the color image P1 into the multimodal detection segmentation network model M1; wherein M1 can be a multimodal detection segmentation network model that meets the requirements of text description and image input. This embodiment does not limit the type of multimodal detection segmentation model used. Through M1, the target area P3 of the material and slot in the material description information can be obtained, and the mask Mask of the target area can also be obtained as the segmentation mask P4. Through the above scheme, in an open environment, even if a new material is replaced, as long as the corresponding material description is provided, the target area P3 and the segmentation mask P4 can be generated, so that preliminary positioning can be performed, so that it can quickly adapt to the new material and the new environment.

[0116] Step 7: Input the previously obtained target area P3, segmentation mask P4 and predefined into the multimodal 6D pose estimation model M2. M2 is also a freely replaceable module. It can be a self-designed network model that performs 6D pose estimation by training material information of industrial scenes, or it can be an open source method with strong generalization. Since the identified target area P3 and segmentation mask P4 contain the parts to be sorted and loaded and the placement slot area in the material, the output of M2 contains a set of material parts grasping posture Z1 and placement slot placement posture Z2, Z1 and Z2 are represented by Indicates that Indicates the location of materials and slots. The quadruple representing the posture of the material and slot is combined with the material part template knowledge base K1 to obtain the actual grasping position of the robot arm for the sorted material. Based on the posture of the required loading and unloading slot, the robot arm inserts the material into the required slot, thus completing the loading and unloading operation. This process achieves automated material sorting and loading and unloading operations without human intervention.

[0117] This embodiment can use a multimodal model to identify, locate and estimate the posture of materials and slots. It is not restricted by the type of materials and can adapt quickly even with new materials, which greatly reduces the cost of iteration. This embodiment can be combined with the upstream design of a multimodal large model, and the skills of grabbing and pushing can be designed simultaneously. Whether it is conventional placement, arbitrary placement or stacked placement, it can also guide the robotic arm to complete the sorting and loading and unloading operations. This embodiment has the operation of breaking up stacked parts, which can cope with complex and intensive sorting scenarios. This embodiment adopts a multimodal method, which can effectively weaken the material identification of specific labels, and has operation tasks in open scenarios, which greatly reduces the iteration cost. The modules involved in this embodiment are all flexible modules, which can be quickly replaced and upgraded.

[0118] See Figure 4 , Figure 4This is a schematic diagram of the structure of a material sorting system based on a robotic arm provided in an embodiment of the present application. The system may include:

[0119] The information acquisition module 401 is used to acquire the current scene image of the working area and obtain the material description information of the target material;

[0120] An image processing module 402 is configured to segment the current scene image according to the material description information to obtain a material region where the target material is located and a material segmentation mask corresponding to the material region; wherein the material region is the region where a single target material is located in the current scene image;

[0121] a posture estimation module 403 for performing posture estimation based on the position of the material region and the material segmentation mask to obtain a grasping posture of the end effector of the robotic arm when grasping the target material;

[0122] a position determination module 404 for acquiring a standard posture of the target material and determining a grasping position of the end effector on the target material according to the grasping posture and the standard posture;

[0123] The robot arm control module 405 is used to control the robot arm to grasp the grasping position of the target material in the grasping posture, adjust the posture of the end effector so that the grasped target material is in the standard posture, and perform a sorting operation on the target material.

[0124] Furthermore, it also includes:

[0125] The material separation module is used to generate a motion path for the robotic arm and control the robotic arm to move along the motion path to break up the materials until there is no material stacking and / or material occlusion in the current scene image of the working area before obtaining the current scene image of the working area; wherein the motion path passes through an area where there is material stacking and / or material occlusion.

[0126] Furthermore, the initial scene image includes a color image and a depth image;

[0127] The process of the material separation module generating the motion path of the robotic arm includes: inputting the initial scene image into the path generation model, so that the path generation model identifies areas in the initial scene image where materials are stacked and / or blocked by materials and outputs multiple action sequences; wherein the path generation model is an end-to-end trained network model, and the action sequence includes multiple joint values; and generating the motion path based on all the action sequences.

[0128] Furthermore, it also includes:

[0129] The standard posture determination module is used to obtain the three-dimensional geometric model of the target material and the structural information of the robotic arm; and is also used to determine the standard posture of the target material based on the three-dimensional geometric model of the target material and the structural information of the robotic arm.

[0130] Furthermore, the posture estimation module performs posture estimation based on the position of the material area and the material segmentation mask, and obtains the grasping posture of the end effector of the robot arm when grasping the target material. The process includes: determining the contour information of the target material based on the material segmentation mask; obtaining the three-dimensional geometric model of the target material, and determining the current posture of the target material based on the contour information and the three-dimensional geometric model; determining the size parameters of the target material based on the position of the material area and the contour information; and calculating the grasping posture of the end effector of the robot arm when grasping the target material based on the current posture and size parameters of the target material.

[0131] Furthermore, the position determination module obtains the standard posture of the target material, and determines the grasping position of the end effector for the target material based on the grasping posture and the standard posture. The process includes: determining the three-dimensional geometric model of the target material, and determining the graspable area of ​​the target material based on the three-dimensional geometric model; selecting a first contact point set matching the grasping posture from the graspable area; wherein the first contact point set includes the contact points between the end effector and the target material when the end effector grasps the target material in the grasping posture; selecting a second contact point set matching the standard posture from the graspable area; wherein the second contact point set includes the contact points between the end effector and the target material when the target material is in the standard posture; and determining the grasping position of the end effector for the target material based on the intersection of the first contact point set and the second contact point set.

[0132] Furthermore, the process of the robotic arm control module performing a sorting operation on the target material includes: obtaining slot description information of the target slot; segmenting the current scene image according to the slot description information to obtain the slot area where the target slot is located and the slot segmentation mask corresponding to the slot area; performing posture estimation according to the position of the slot area and the slot segmentation mask to obtain the placement posture of the end effector when placing the target material; and controlling the robotic arm based on the placement posture to place the target material into the target slot.

[0133] Furthermore, the robotic arm control module performs posture estimation based on the position of the slot area and the slot segmentation mask, and the process of obtaining the placement posture of the end effector when placing the target material includes: determining the contour information of the target slot based on the material segmentation mask; obtaining the three-dimensional geometric model of the target slot, and determining the current posture of the target slot based on the contour information and the three-dimensional geometric model; and calculating the placement posture of the end effector of the robotic arm when placing the target material based on the position of the slot area and the current posture of the target slot.

[0134] Furthermore, the process of the information acquisition module acquiring the material description information of the target material includes: determining the target frame added by the user in the current scene image; inputting the image in the target frame into the object recognition model to obtain a recognition result; and generating the material description information of the target material based on the recognition result.

[0135] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, and will not be repeated here.

[0136] This application also provides a storage medium having a computer program stored thereon. When executed, the computer program can implement the steps provided in the above embodiments. The storage medium may include: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium capable of storing program code.

[0137] The present application also provides an electronic device that may include a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the steps provided in the above embodiment can be implemented. Of course, the electronic device may also include various network interfaces, a power supply, and other components.

[0138] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of this application.

[0139] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A material sorting method based on a robotic arm, characterized in that: include: Acquire a current scene image of the work area and obtain material description information of a target material; wherein the material description information is information used to describe the appearance, and / or function, and / or type of the target material; Segmenting the current scene image according to the material description information to obtain a material region where the target material is located and a material segmentation mask corresponding to the material region; wherein each material region is a region where a single target material is located in the current scene image; Performing posture estimation based on the position of the material area and the material segmentation mask to obtain a grasping posture of the end effector of the robotic arm when grasping the target material; Obtaining a standard posture of the target material, and determining a gripping position of the end effector on the target material based on the gripping posture and the standard posture; wherein the standard posture is an ideal posture that the target material is expected to achieve after being gripped, and the standard posture is determined based on a three-dimensional geometric model of the target material and structural information of the robotic arm; Controlling the robotic arm to grasp the grasping position of the target material in the grasping posture, adjusting the posture of the end effector so that the grasped target material is in the standard posture, and performing a sorting operation on the target material; Wherein, obtaining the standard posture of the target material and determining the gripping position of the end effector on the target material according to the gripping posture and the standard posture includes: Determining a three-dimensional geometric model of the target material, and determining a graspable area of ​​the target material based on the three-dimensional geometric model; Selecting a first contact point set matching the grasping posture from the graspable area; wherein the first contact point set includes contact points between the end effector and the target material when the end effector grasps the target material in the grasping posture; Selecting a second contact point set matching the standard posture from the graspable area; wherein the second contact point set includes contact points between the end effector and the target material when the target material is in the standard posture; The gripping position of the end effector for the target material is determined according to the intersection of the first contact point set and the second contact point set.

2. The material sorting method based on a robotic arm according to claim 1, characterized in that: Before getting the current scene image of the working area, it also includes: If there is material stacking and / or material obstruction in the initial scene image of the working area, a motion path of the robotic arm is generated, and the robotic arm is controlled to move along the motion path to break up the material until there is no material stacking and / or material obstruction in the current scene image of the working area; wherein the motion path passes through an area where there is material stacking and / or material obstruction.

3. The material sorting method based on a robotic arm according to claim 1, characterized in that: Performing posture estimation based on the position of the material area and the material segmentation mask to obtain a grasping posture of the end effector of the robot arm when grasping the target material includes: Determining contour information of the target material according to the material segmentation mask; Acquire a three-dimensional geometric model of the target material, and determine a current posture of the target material according to the contour information and the three-dimensional geometric model; Determining the size parameters of the target material according to the position of the material area and the contour information; The gripping posture of the end effector of the robot arm when gripping the target material is calculated according to the current posture and size parameters of the target material.

4. The material sorting method based on a robotic arm according to claim 1, characterized in that: Performing a sorting operation on the target material, including: Get the slot description information of the target slot; Segmenting the current scene image according to the slot description information to obtain a slot area where the target slot is located and a slot segmentation mask corresponding to the slot area; Performing posture estimation based on the position of the slot area and the slot segmentation mask to obtain a placement posture of the end effector when placing the target material; Based on the placement posture, the robotic arm is controlled to place the target material into the target slot.

5. The material sorting method based on a robotic arm according to claim 4, characterized in that: Performing posture estimation based on the position of the slot area and the slot segmentation mask to obtain a placement posture of the end effector when placing the target material includes: Determining contour information of the target slot according to the material segmentation mask; Acquire a three-dimensional geometric model of the target slot, and determine a current posture of the target slot according to the contour information and the three-dimensional geometric model; The placement posture of the end effector of the robotic arm when placing the target material is calculated according to the position of the slot area and the current posture of the target slot.

6. A material sorting system based on a robotic arm, characterized in that: include: An information acquisition module, configured to acquire a current scene image of a working area and acquire material description information of a target material; wherein the material description information is information used to describe the appearance, and / or function, and / or type of the target material; an image processing module, configured to segment the current scene image according to the material description information to obtain a material region where the target material is located and a material segmentation mask corresponding to the material region; wherein each material region is a region where a single target material is located in the current scene image; a posture estimation module, configured to perform posture estimation based on the position of the material area and the material segmentation mask, and obtain a grasping posture of the end effector of the robotic arm when grasping the target material; a position determination module, configured to obtain a standard posture of the target material and determine a grasping position of the end effector on the target material based on the grasping posture and the standard posture; wherein the standard posture is an ideal posture that the target material is expected to achieve after being grasped, and the standard posture is determined based on a three-dimensional geometric model of the target material and structural information of the robotic arm; a robotic arm control module, configured to control the robotic arm to grasp the target material at the grasping position in the grasping posture, adjust the posture of the end effector so that the grasped target material is in the standard posture, and perform a sorting operation on the target material; The process in which the position determination module obtains the standard posture of the target material and determines the gripping position of the end effector for the target material according to the gripping posture and the standard posture includes: Determine a three-dimensional geometric model of the target material, and determine a graspable area of ​​the target material based on the three-dimensional geometric model; select a first contact point set that matches the grasping posture from the graspable area; wherein the first contact point set includes the contact points between the end effector and the target material when the end effector grasps the target material in the grasping posture; select a second contact point set that matches the standard posture from the graspable area; wherein the second contact point set includes the contact points between the end effector and the target material when the target material is in the standard posture; determine the grasping position of the end effector for the target material based on the intersection of the first contact point set and the second contact point set.

7. An electronic device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the material sorting method based on the robotic arm as described in any one of claims 1 to 5 are implemented.

8. A storage medium, characterized in that: The storage medium stores computer-executable instructions, which, when loaded and executed by the processor, implement the steps of the material sorting method based on the robotic arm as described in any one of claims 1 to 5.

Images