Manipulator control method and device, manipulator, robotic arm, robot, and chip

Abstract

The application provides a robot control method and device, a robot, a robot arm, and a chip, and relates to the technical field of robots. The method comprises the following steps: in response to a grasping task of a robot, performing instance segmentation on a multi-channel image of an environment in which the robot is located to obtain a plurality of object masks; the multi-channel image carries depth information of each pixel; determining a candidate object from the plurality of objects according to size information of the plurality of masks and depth information of the objects; determining a graspable space of the candidate object based on an estimated pose of the candidate object and the mask; determining a target object and a corresponding target grasping pose from the candidate object according to the graspable space of the candidate object; and controlling the robot to grasp the target object based on the target grasping pose. Therefore, an object that is fully exposed, has a reasonable size, and has good reachability can be effectively identified and preferentially selected for grasping, and the grasping success rate of the robot in a complex stacked environment can be significantly improved.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a manipulator control method, device, manipulator, robotic arm, robot, and chip. Background Technology

[0002] Random Bin Picking based on 3D vision is a key technology in the fields of intelligent manufacturing and robotics. Its core objective is to guide robots to perform accurate and reliable picking operations in scenarios where multiple objects are stacked randomly, using 3D vision recognition and positioning technology.

[0003] The related technology uses a disordered grasping scheme, randomly determining the object to be grasped, resulting in a low success rate for the robot to grasp objects. Summary of the Invention

[0004] This application proposes a robotic arm control method, device, robotic arm, robot, and chip to at least partially solve one of the technical problems in the related art.

[0005] One embodiment of this application proposes a robotic arm control method, including:

[0006] In response to the grasping task of the robotic arm, the multi-channel image of the environment in which the robotic arm is located is segmented to obtain masks of multiple objects; wherein, the multi-channel image carries depth information of each pixel; Based on the size information of the multiple masks and the depth information of the objects to which they belong, candidate objects that can be grabbed are determined from the multiple objects. Based on the estimated pose and mask of the candidate object, the graspable space of the candidate object is determined. Based on the graspable space of the candidate objects, the target object to be grasped and the corresponding target grasping pose are determined from the candidate objects, so as to control the robotic arm to grasp the target object based on the target grasping pose.

[0007] Another embodiment of this application proposes a robotic arm control device, including: The segmentation module is used to perform instance segmentation on the multi-channel image of the environment in which the robot is located in response to the grasping task of the robot, and obtain masks of multiple objects; wherein, the multi-channel image carries depth information of each pixel; The first determining module is used to determine a candidate object that can be grabbed from the plurality of objects based on the size information of the plurality of masks and the depth information of the objects to which they belong. The second determining module is used to determine the graspable space of the candidate object based on the estimated pose and mask of the candidate object. The third determining module is used to determine the target object to be grasped and the corresponding target grasping pose from the candidate objects based on the graspable space of the candidate objects. The control module is used to control the robotic arm to grasp the target object based on the target grasping pose.

[0008] Another aspect of this application provides a robotic arm for implementing the robotic arm control method described in the foregoing aspect.

[0009] In another aspect, this application provides a robotic arm, wherein the robotic arm integrates a robotic hand, and the robotic arm is used to implement the robotic hand control method as described in the foregoing aspect.

[0010] In another aspect of this application, a robot is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the robotic arm control method as described in the preceding aspect.

[0011] Another aspect of this application provides a chip including an interface circuit and a processing circuit coupled to each other, the interface circuit being used to input or output signals, and the processing circuit being configured to perform the robotic arm control method as described in the foregoing aspect.

[0012] In another aspect of this application, a non-transitory computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the robotic arm control method as described in the foregoing aspect.

[0013] The robotic arm control method proposed in this application introduces instance segmentation based on multi-channel images (including depth information) into the grasping task. This, combined with mask size and depth information, filters out reliable and graspable candidate objects, effectively eliminating unreliable targets caused by occlusion, truncation, or perceptual uncertainty. Based on the candidate object's mask and estimated pose, its graspable space is determined, and the target object and its grasping pose are optimized accordingly. This enables high-success-rate grasping of small, densely stacked objects. This application is not limited to specific gripper types; it can effectively identify and prioritize objects that are sufficiently exposed, reasonably sized, and have good orientational reachability for grasping. This significantly improves the success rate, robustness, and adaptability of the robotic arm in complex stacked environments, providing a feasible path for platforms such as humanoid robots that require highly flexible operational capabilities.

[0014] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0015] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a robotic arm control method provided for an exemplary embodiment of this application; Figure 2 A schematic diagram of an application scenario provided for an exemplary embodiment of this application; Figure 3 A flowchart illustrating another robotic arm control method provided for an exemplary embodiment of this application; Figure 4 A schematic diagram showing the grasping position and grasping direction corresponding to objects of different placement categories, provided for an exemplary embodiment of this application; Figure 5 A schematic diagram of a robotic arm grasping a vertically placed object, provided for an exemplary embodiment of this application; Figure 6 A flowchart illustrating another robotic arm control method provided for an exemplary embodiment of this application; Figure 7 A schematic diagram illustrating a method for determining checkpoints corresponding to a vertically placed object, provided for an exemplary embodiment of this application; Figure 8 A schematic diagram illustrating a method for determining checkpoints corresponding to an inverted object, provided for an exemplary embodiment of this application; Figure 9 A flowchart illustrating yet another robotic arm control method provided for an exemplary embodiment of this application; Figure 10 A schematic diagram illustrating the principle of depth remapping provided for an exemplary embodiment of this application; Figure 11 A schematic diagram illustrating the implementation principle of an exemplary embodiment of this application; Figure 12 A schematic diagram of the structure of a robotic arm control device provided for an exemplary embodiment of this application; Figure 13 A schematic diagram of the structure of a robotic hand, robotic arm, or robot provided for exemplary embodiments of this application; Figure 14 This is a schematic diagram of the structure of a chip proposed as an exemplary embodiment of this application. Detailed Implementation

[0016] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0017] Disordered grasping schemes in related technologies typically employ structured light sensors or 3D cameras to acquire scene point clouds, combined with pose estimation algorithms, to achieve the identification and grasping control of randomly piled target objects. However, such systems are often highly integrated with low modularity, and generally rely on high-performance large graphics processing units (GPUs) for real-time processing of complex point cloud data. They also require expensive industrial-grade 3D cameras, resulting in high deployment costs and poor flexibility. Furthermore, these methods usually rely on specially configured end effectors, such as two-finger grippers or suction cups, suitable for grasping regular geometries with large contact surfaces (such as cubes and brick-shaped objects), but difficult to effectively grasp small objects with narrow surfaces (such as bolts, gears, and connectors). They also cannot be adapted to robotic hands with high degrees of freedom (such as five-finger dexterous hands), making them difficult to deploy on platforms such as humanoid robots that require high operational flexibility.

[0018] On the other hand, research on disordered grasping with multi-fingered dexterous hands mainly employs reinforcement learning or sampling optimization methods: the former trains grasping strategies in a simulation environment, while the latter generates grasping postures without shading and combines analysis of force closure and energy criteria to select effective grasping configurations. Although these methods have achieved some success in everyday scenarios with larger, low-density objects such as bottles, cans, and cardboard boxes, their model generalization ability is limited, making it difficult to transfer to scenarios with small, high-density stacked industrial parts in deep boxes. Especially under conditions of severe occlusion, small graspable areas, and high requirements for contact stability, these technologies generally suffer from insufficient perception accuracy, low grasping success rates, and limited practicality.

[0019] In view of at least one of the problems existing in the above-mentioned related technologies, this application proposes a robotic arm control method, device, robotic arm, robot and chip.

[0020] The robotic arm control method, device, robotic arm, robot, and chip of this application are described below with reference to the accompanying drawings. Before specifically describing the embodiments of this application, for ease of understanding, commonly used technical terms are first introduced: A robotic arm is a device capable of grasping, manipulating, or performing fine motor skills, typically used as an end-effector. Its core function is to interact directly with the environment (such as grasping objects or assembling parts).

[0021] A robotic arm is an arm-shaped structure with multiple degrees of freedom of motion, primarily used for spatial positioning and large-scale movement. Its flexible form allows it to adapt to various scenarios, such as industrial robotic arms, medical assistive arms, and bionic prostheses. The robotic hand, for example, can be a gripping or manipulating device mounted at the end of the robotic arm, used for directly contacting and manipulating objects.

[0022] A robot is an intelligent machine capable of semi-autonomous or fully autonomous operation. It performs tasks through programming and automatic control, possesses core capabilities such as perception, decision-making, and execution, and is widely used in industries, medical fields, and service industries.

[0023] Figure 1 This is a flowchart illustrating a robotic arm control method provided for an exemplary embodiment of this application.

[0024] It should be noted that the robotic arm control method of this application embodiment can be applied to a robotic arm control device. In some possible embodiments, the robotic arm control device can be configured in a robotic arm, robotic arm, robot, or chip, so that the robotic arm, robotic arm, robot, or chip can perform the control function of the robotic arm. In addition, in some possible embodiments, the robotic arm control device can also be software in a robotic arm, robotic arm, or robot.

[0025] In any embodiment of this application, the chip can be integrated into a robotic hand, robotic arm, or robot. The chip includes a Central Processing Unit (CPU), Image Signal Processing (ISP), Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), System-on-Chip (SOC), Reduced Instruction Set Computer (RISC), etc., which will not be listed here.

[0026] For ease of explanation, the following embodiments will use the application of a robotic arm control device to a robotic arm as an example.

[0027] like Figure 1 As shown, the robotic arm control method may include the following steps S101 to S104: Step S101: In response to the grasping task of the robotic arm, the multi-channel image of the environment in which the robotic arm is located is segmented to obtain the masks of multiple objects; wherein, the multi-channel image carries the depth information of each pixel.

[0028] The grasping task can be an independent task performed by the robotic arm, or it can be a subtask within a complex interactive task. This application embodiment does not impose any limitations on this. For example, taking an interactive task as an object classification task, the object classification task may include: an object grasping subtask and a classification and placement subtask; taking an interactive task as a screw-tightening task, the screw-tightening task may include a bolt grasping subtask and a bolt tightening subtask.

[0029] A multi-channel image includes multiple color channels (such as red (R), green (G), and blue (B) color channels) and a depth channel (D). The color channels indicate the color information of each pixel, and the depth channel indicates the depth information of each pixel. For example, a multi-channel image can be an RGBD image directly output by the acquisition device, or an image obtained by processing an RGBD image.

[0030] The depth information includes, but is not limited to: the real depth obtained by the acquisition device (used to indicate the distance from the spatial point corresponding to each pixel in the image to the acquisition device) and the virtual mapped depth obtained by remapping the real depth. This application embodiment does not limit this.

[0031] The data collection device can be deployed in the working environment of the robotic arm. For example, the data collection device can be deployed above multiple objects to be grasped, or it can be deployed on the robotic arm or robot to which the robotic arm is integrated. This application embodiment does not limit this.

[0032] The acquisition device can be an acquisition camera (such as a depth camera) or a multi-sensor fusion system that includes radar and an acquisition camera.

[0033] The objects to be grasped by the robotic arm include, but are not limited to, multiple small objects, multiple disorderly stacked objects, etc.

[0034] In this embodiment of the application, before the robotic arm receives the grasping task but does not start to execute the grasping task, or during the process of the robotic arm executing the grasping task, a multi-channel image of the environment in which the robotic arm is located can be acquired, and an instance segmentation algorithm, an object segmentation algorithm or a semantic segmentation algorithm can be used to segment the multi-channel image to obtain masks of multiple objects; wherein, the mask of each object is used to indicate the position of the object in the multi-channel image.

[0035] Step S102: Based on the size information of multiple masks and the depth information of the objects to which they belong, determine the candidate objects that can be grabbed from multiple objects.

[0036] The depth information for each object includes the depth information of each pixel belonging to that object.

[0037] The size information of each mask includes, but is not limited to: the pixel area of ​​the mask, the diagonal length of the bounding box (such as the minimum bounding rectangle) of the mask, etc.

[0038] In this embodiment of the application, the size information of multiple masks can be determined, and multiple objects can be filtered based on the size information of multiple masks and the depth information of the objects to which the multiple masks belong, to obtain candidate objects that can be grabbed.

[0039] As an example, based on the size information of multiple masks and the depth information of the objects to which the multiple masks belong, the object stacked on the upper layer with a relatively large mask size can be identified from multiple objects and used as a candidate object.

[0040] Step S103: Based on the estimated pose and mask of the candidate object, determine the grabbable space of the candidate object.

[0041] In the embodiments of this application, the estimated pose of each candidate object can be determined based on the multi-channel image, and the graspable space of each candidate object can be determined based on the estimated pose and mask of each candidate object.

[0042] Step S104: Based on the graspable space of the candidate objects, determine the target object to be grasped and the corresponding target grasping pose from the candidate objects, so as to control the robot arm to grasp the target object based on the target grasping pose.

[0043] In the embodiments of this application, the target object to be grasped can be determined from each candidate object according to the graspable space of each candidate object. For example, the candidate object with the largest graspable space can be taken as the target object, and the target grasping pose can be determined according to the graspable space of the target object. Thus, the robot arm can be controlled to grasp the target object based on the target grasping pose.

[0044] As an application scenario, taking multiple objects as randomly stacked objects in an object bin, such as bolts, gears, brackets, castings, connectors, etc., and a robotic arm as a five-fingered dexterous hand as an example, the technical solution provided in this application can be applied to, for example... Figure 2 In the industrial scenario shown, RGBD images can be acquired using a depth camera, and based on these RGBD images, the target object to be grasped and the target grasping pose of the target object can be identified from multiple objects that are randomly stacked. Based on the target grasping pose, the five-fingered dexterous hand can be controlled to grasp the target object and perform continuous operations.

[0045] It should be understood that the above is only an example of an industrial scenario, but this application is not limited thereto. The technical solution provided in this application can also be applied to other fields, such as the scenario of grabbing disorderly stacked packages in the logistics industry, the scenario of sorting and grabbing waste metal or electronic components on the recycling production line, and the scenario of disorderly sorting of multiple categories of goods in warehouse automation.

[0046] The robotic arm control method of this application introduces instance segmentation based on multi-channel images (including depth information) into the grasping task. It combines mask size and depth information to filter reliable and graspable candidate objects, effectively eliminating unreliable targets caused by occlusion, truncation, or perceptual uncertainty. Based on the mask and estimated pose of the candidate object, its graspable space is determined, and the target object and its grasping pose are optimized accordingly. This enables high-success-rate grasping of small, densely stacked objects. This application is not limited to specific gripper types and can effectively identify and prioritize objects that are sufficiently exposed, reasonably sized, and have good orientational reachability for grasping. This significantly improves the success rate, robustness, and adaptability of the robotic arm in complex stacked environments, providing a feasible path for platforms such as humanoid robots that require highly flexible operational capabilities.

[0047] As one possible implementation method, Figure 3 A flowchart illustrating another robotic arm control method provided for an exemplary embodiment of this application.

[0048] It should be noted that the robotic arm control method can be executed alone, or it can be executed together with any embodiment or possible implementation in the embodiment of this application, or it can be executed together with any technical solution in the related technology. The embodiments of this application do not limit this.

[0049] like Figure 3 As shown, the robotic arm control method may include the following steps S301 to S306: Step S301: In response to the grasping task of the robotic arm, perform instance segmentation on the multi-channel image of the environment in which the robotic arm is located to obtain masks of multiple objects.

[0050] In this multi-channel image, the depth information of each pixel is carried.

[0051] Step S302: Based on the size information of multiple masks and the depth information of the objects to which they belong, determine the candidate objects that can be grabbed from multiple objects.

[0052] It should be noted that the explanations of steps S301 to S302 can be found in the relevant descriptions in any embodiment of this application, and will not be repeated here.

[0053] Step S303: Based on the multi-channel image, perform pose estimation on the candidate object to obtain the estimated pose of the candidate object.

[0054] In the embodiments of this application, a pose estimation algorithm can be used to estimate the pose of each candidate object based on multi-channel images.

[0055] Step S304: Based on the estimated pose of the candidate objects, classify the placement methods of the candidate objects to obtain the placement categories of the candidate objects.

[0056] The placement categories include, but are not limited to: upside down (or sideways) and upright.

[0057] In this embodiment of the application, the placement method of each candidate object can be classified based on the estimated pose of each candidate object to obtain the placement category to which the candidate object belongs.

[0058] For example, the angle between the axis of the candidate object and its placement plane (referred to as the second plane in this application, such as a desktop) can be determined based on the estimated pose of the candidate object. If the angle is greater than a set angle, the placement category of the candidate object is determined to be upright, and if the angle is less than or equal to the set angle, the placement category of the candidate object is determined to be upside down.

[0059] The set angle is a pre-set angle threshold with a relatively large value, such as 60° or 70°.

[0060] Step S305: Determine the grabbable space of the candidate object based on the placement category and mask of the candidate object.

[0061] In this embodiment of the application, the grabbable space of each candidate object can be calculated specifically based on the placement category and mask of each candidate object.

[0062] Step S306: Based on the graspable space of the candidate objects, determine the target object to be grasped and the corresponding target grasping pose from the candidate objects, so as to control the robot arm to grasp the target object based on the target grasping pose.

[0063] It should be noted that the explanation of step S306 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0064] In any embodiment of this application, the candidate object with the largest graspable space can be selected as the target object, and the target grasping pose can be determined based on the center position and orientation of the graspable space of the target object. For example, the center position of the largest graspable space of the target object can be selected as the grasping position in the target grasping pose, and the grasping direction in the target grasping pose can be determined based on the orientation of the largest graspable space, so as to control the robot arm to grasp the target object based on the target grasping pose.

[0065] For example, taking the placement category of the target object as upright, the graspable space (i.e., the graspable range) of the target object can be as follows: Figure 4 (a) shows the gray area; taking the placement category of the target object as an example of inverted placement, the grabbable space (i.e., grabbable range) of the target object can be as follows: Figure 4 (b) shows the gray area. Regardless of whether the target object is upside down or upright, its gripping position can be the middle position of the maximum gripping space, marked as the gripping position ( Figure 4 (a) and 4(b) are denoted as the grab center, which is the pixel position (u,v) in the image coordinate system. The grab direction is represented by the deflection angle (θ) relative to the vertical axis of the image (the line containing the grab direction is parallel to the second plane where the target object is placed). Figure 4 (a) and 4(b) are denoted as the optimal grasping direction. At this point, the target grasping pose can be transformed into a 6D pose in the robot coordinate system based on (u,v) and (θ). .

[0066] in, The three-dimensional spatial coordinates representing the center of the capture can be determined by (u,v) and the intrinsic and extrinsic parameters of the acquisition device; It's an Euler angle. This indicates Roll (roll angle). Indicates pitch (angle of tilt), and and All are 0. Yaw (yaw angle) can be determined by θ and the external parameters of the data acquisition device.

[0067] It should be noted that, Figure 4 The ungraspable area in (a) and 4(b) refers to the area outside the graspable area of ​​the target object in the plane.

[0068] Taking a robotic arm as an example of a five-fingered dexterous hand, when controlling the robotic arm to grasp a target object, a fixed pre-grasping gesture can be set. , where p iLet n be the joint angles of the robotic hand, with n joints, such that the closing direction of the index finger and thumb is approximately horizontal. Let the Euler angle be the direction of the line connecting the centers of the index finger and thumb. The center point between the index finger and the thumb is For example, a schematic diagram of a robotic arm grasping a vertically placed object can be shown as follows: Figure 5 As shown.

[0069] in, Figure 5 The closing direction of the fingers in the text refers to the closing direction of the index finger and thumb mentioned above.

[0070] It should be noted that after the robotic arm has finished grasping the target object, it can perform a return operation and repeat the above steps to complete the grasping of all objects.

[0071] In summary, it can achieve continuous grasping of disordered stacked objects, significantly reducing system hardware costs and deployment complexity. At the same time, it is compatible with high-degree-of-freedom multi-finger manipulators, making it easy to promote and apply on general platforms such as humanoid robots. It effectively solves the problem of automated grasping of small, densely stacked industrial parts in deep box environments.

[0072] The robotic arm control method of this application combines pose estimation and placement category recognition to explicitly model the geometric orientation and spatial layout of candidate objects, and further integrates the visible area reflected by its segmentation mask to accurately define the graspable space applicable to the current robotic arm. Thus, under conditions of severe occlusion or partial exposure, a safe and feasible graspable area can be dynamically generated based on the actual visible part and posture semantics, which significantly improves the reliability and executability of grasping planning for multiple objects to be grasped in complex scenarios such as deep boxes and high-density stacking.

[0073] As one possible implementation method, Figure 6 A schematic flowchart illustrating another robotic arm control method provided for an exemplary embodiment of this application.

[0074] It should be noted that the robotic arm control method can be executed alone, or it can be executed together with any embodiment or possible implementation in the embodiment of this application, or it can be executed together with any technical solution in the related technology. The embodiments of this application do not limit this.

[0075] like Figure 6 As shown, the robotic arm control method may include the following steps S601 to S609: Step S601: In response to the gripping task of the robotic arm, perform instance segmentation on the multi-channel image of the environment in which the robotic arm is located to obtain masks of multiple objects.

[0076] In this multi-channel image, the depth information of each pixel is carried.

[0077] Step S602: Based on the size information of multiple masks and the depth information of the objects to which they belong, determine the candidate objects that can be grabbed from multiple objects.

[0078] Step S603: Based on the multi-channel image, perform pose estimation on the candidate object to obtain the estimated pose of the candidate object.

[0079] Step S604: Based on the estimated pose of the candidate objects, classify the placement methods of the candidate objects to obtain the placement categories of the candidate objects.

[0080] It should be noted that the explanations of steps S601 to S604 can be found in the relevant descriptions in any embodiment of this application, and will not be repeated here.

[0081] Step S605: Project each pixel belonging to the candidate object in the multi-channel image from the pixel space to the world space to obtain the spatial position of each pixel belonging to the candidate object.

[0082] In this embodiment of the application, the pixels belonging to the candidate object in the multi-channel image can be projected from the pixel space to the world space based on the mapping relationship between the pixel space and the world space (i.e., the mapping relationship between the pixel coordinate system and the world coordinate system), so as to obtain the spatial position of each pixel belonging to the candidate object in the world space.

[0083] Step S606: Determine the axis of the candidate object based on the spatial position of each pixel belonging to the candidate object.

[0084] In this embodiment of the application, the axis of the candidate object can be determined based on the spatial position of each pixel belonging to the candidate object.

[0085] Step S607: Based on the candidate object's axis, placement category, and mask, sample multiple target checkpoints from the surrounding area of ​​the candidate object.

[0086] In this embodiment of the application, multiple target checkpoints can be obtained by selectively sampling around each candidate object based on its axis, placement category, and mask.

[0087] As one possible implementation, the sampling method for target checkpoints is as follows: When the candidate object is placed vertically, firstly, a first plane perpendicular to the axis can be determined based on the highest point of the candidate object's axis. Then, multiple first checkpoints can be sampled on the first plane. For example, multiple circles with different radii can be drawn on the first plane with the highest point as the center, and multiple first checkpoints can be sampled on the boundaries of the multiple circles using a set sampling method. Then, the multiple first checkpoints can be projected from world space to pixel space to obtain the pixel positions of the multiple first checkpoints. Finally, the first checkpoints whose pixel positions are located in the image area corresponding to the mask of the candidate object can be deleted to obtain the retained target checkpoints.

[0088] The sampling method can be set, including but not limited to: uniform sampling and random sampling.

[0089] As an example, taking candidate objects as Figure 7 (a) Taking the vertically placed object as an example, with uniform sampling as the sampling method, a first plane perpendicular to the axis can be drawn near the highest point of the axis of the vertically placed object in the world coordinate system. Circles of different radii can be drawn on this first plane with the axis as the center, and uniform sampling can be performed on these circles. Figure 7 (a) shows the first checkpoint (or 3D checkpoint, depth checkpoint). The first checkpoint can then be projected into pixel space, and checkpoints within the mask area of ​​the standing object can be filtered out to obtain the retained target checkpoints. For example, the target checkpoints can be as follows: Figure 7 (b) shown or Figure 7 As shown in (c), where, for Figure 7 (b) The first checkpoint, which should have circled the rear of the upright object, was deleted because the lower right portion was within its own masking range; Regarding Figure 7 (c) The first checkpoint covered by the head mask of the upright object is also deleted.

[0090] As another possible implementation, the sampling method for the target checkpoint is as follows: when the placement category of the candidate object is reversed, firstly, the inspection area can be determined based on the axial direction of the candidate object, and multiple second checkpoints can be sampled in the inspection area using a set sampling method. Then, the second checkpoints can be projected from world space to pixel space to obtain the pixel positions of multiple second checkpoints. Finally, the second checkpoints whose pixel positions are located in the image area corresponding to the mask of the candidate object can be deleted to obtain the retained target checkpoints.

[0091] The shape of the inspection area includes, but is not limited to, rectangles, circles, trapezoids, etc. Taking a rectangular inspection area as an example, the length of the inspection area is parallel to the second plane, and the width of the inspection area is parallel to the axis of the candidate object. The second plane is the plane on which multiple objects are placed.

[0092] For example, taking candidate objects as Figure 8 The inverted object shown has a rectangular inspection area. Taking uniform sampling as an example, a rectangular inspection area can be drawn along the axis of the inverted object in the world coordinate system, and uniform sampling can be performed within this rectangular inspection area. Figure 8 The first checkpoint shown is called the 3D checkpoint or depth checkpoint. Then, the first checkpoint in world coordinates can be projected into pixel space, and the first checkpoints within the mask range of the inverted object can be filtered out to obtain the retained target checkpoints.

[0093] Figure 8 In the middle, the length of the rectangular inspection area is parallel to the second plane, and the width of the rectangular inspection area is parallel to the axis of the inverted object.

[0094] Step S608: Determine the grabbable space of the candidate object based on the difference between the depth information of the target checkpoint and the depth information of the axis.

[0095] In this embodiment of the application, the grabbable space of a candidate object can be determined based on the difference between the depth information of the target checkpoint and the depth information of the axis of the candidate object.

[0096] For example, for a candidate object placed upright, if the difference between the depth information of the target checkpoint and the axis of the candidate object is less than a set threshold, the radial line where the target checkpoint is located is determined to be an ungraspable direction, and a graspable direction or graspable space is determined based on the other directions in the first plane besides the ungraspable direction. For a candidate object placed upside down, if the difference between the depth information of the target checkpoint and the axis of the candidate object is less than a set threshold, the line perpendicular to the axis is determined to be an ungraspable direction, and a graspable direction or graspable space is determined based on the other directions in the inspection area besides the ungraspable direction.

[0097] Step S609: Based on the graspable space of the candidate objects, determine the target object to be grasped and the corresponding target grasping pose from the candidate objects, so as to control the robot arm to grasp the target object based on the target grasping pose.

[0098] It should be noted that the explanation of step S609 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0099] The robotic arm control method of this application reconstructs the three-dimensional geometry of an object by projecting each pixel in the masked area onto world space, extracting the axis and sampling the target checkpoints accordingly, and then combining depth difference analysis to determine local accessibility. This achieves refined, physically-aware modeling of the graspable space, effectively identifying ungraspable areas caused by occlusion, interference from nearby objects, or surface incompleteness, ensuring that the selected grasping pose has sufficient operating clearance and contact stability in the real environment, thereby significantly reducing the grasping failure rate. It is especially suitable for grasping high-density stacked objects in deep boxes.

[0100] As one possible implementation method, Figure 9 This is a flowchart illustrating yet another robotic arm control method provided for an exemplary embodiment of this application.

[0101] It should be noted that the robotic arm control method can be executed alone, or it can be executed together with any embodiment or possible implementation in the embodiment of this application, or it can be executed together with any technical solution in the related technology. The embodiments of this application do not limit this.

[0102] like Figure 9 As shown, the robotic arm control method may include the following steps S901 to S907: Step S901: In response to the gripping task of the robotic arm, perform instance segmentation on the multi-channel image of the environment in which the robotic arm is located to obtain masks of multiple objects.

[0103] The multi-channel image carries depth information for each pixel. It should be noted that the explanation of step S901 can be found in the relevant descriptions of any embodiment of this application, and will not be repeated here.

[0104] In any embodiment of this application, the depth information may include a virtual mapped depth. The method of obtaining the mapped depth is, for example, to obtain the real depth of each pixel acquired by the acquisition device for acquiring multi-channel images, and to remap the real depth of each pixel according to the external parameters of the acquisition device to obtain the mapped depth of each pixel.

[0105] The true depth of each pixel is used to indicate the distance from the spatial point corresponding to that pixel to the acquisition device.

[0106] The mapping depth of each pixel indicates the distance between the spatial point corresponding to that pixel and the third plane where the acquisition device is located. For example, the third plane satisfies at least one of the following conditions: the height of the third plane matches that of the acquisition device; the third plane is parallel to the second plane on which multiple objects are placed.

[0107] For example, taking the acquisition device as a depth camera and the multiple objects to be captured as randomly stacked objects in an object bin, it is possible to... Figure 10 A depth camera is deployed above the object bin, allowing it to observe the stacked objects within. Based on the depth camera's extrinsic parameters, the true depth of each pixel in the RGBD image captured by the depth camera is remapped so that the mapped depth of each pixel is the vertical distance from the object surface to a plane above it (referred to as the third plane in this application). This third plane is parallel to the second plane (such as a tabletop) where the object bin is placed, and its height is the same as the height of the depth camera. For example, Figure 10 The distance between a spatial point on the surface of an object and the actual camera plane of the depth camera is d1 (i.e., the actual depth in the field of view of the tilted depth camera), and the remapped virtual depth is d. 1r The distance between a spatial point on the surface of another object and the real camera plane of the depth camera is d2, and the virtual mapped depth after remapping is d. 2r .

[0108] Understandably, since robotic arms typically employ a vertically downward approach strategy in disordered grasping scenarios to avoid interference between the wrist or fingers and the sidewalls of the object bin, this mapping depth directly reflects the vertical height of each point on the object's surface from the ideal grasping reference plane (such as the third plane), thus more accurately depicting the stacking hierarchy and visibility of objects. Based on this mapping depth, object instance segmentation, size evaluation, and graspable space modeling can effectively improve the recognition priority of exposed objects on the upper layer, suppress misselection of deeply buried or severely occluded objects, significantly enhance the rationality and success rate of grasping decisions, and improve the overall system operating efficiency and robustness.

[0109] Step S902: Determine the sparsity score of each mask based on the size information of each mask; wherein, the sparsity score is used to characterize the distance between the object to which the corresponding mask belongs and the objects to which the other masks belong.

[0110] The sparsity score indicates the degree of openness of the space surrounding the object to which the mask belongs. The sparsity score is positively correlated with distance; that is, the smaller the distance, the lower the sparsity score, and vice versa.

[0111] In this embodiment of the application, the sparsity fraction of each mask can be determined based on the size information of the mask of each object. The sparsity fraction of each mask is used to characterize the distance between the object to which the mask belongs and the objects to which the other masks belong.

[0112] As one possible implementation, the sparsity score of any first mask among multiple masks can be determined as follows: when the size information includes the diagonal length of the bounding box of the mask, the average diagonal length of the bounding box of each mask can be determined, and a distance threshold can be determined based on the product of the average value and a set first coefficient. Thus, in this application, a second mask located within a circle with the center point of the bounding box of the first mask as the center and the distance threshold as the radius can be determined, and the sparsity score of the first mask can be determined based on the pixel distance from the center point of the bounding box of the second mask to the center of the circle.

[0113] The first coefficient is a pre-set constant; for example, the value of the first coefficient is greater than 1.

[0114] The center point of the bounding box of the second mask falls inside the circle.

[0115] The sparsity score of the first mask is positively correlated with the pixel distance from the center point of the bounding box of the second mask to the center of the circle.

[0116] For example, if the distance threshold is r, then within a circle with the center point of the bounding box of the first mask as the center and r as the radius, the pixel distance from the center point of the bounding box of other masks located within this circle to the center of the circle is denoted as d, and the sparsity fraction of the object's mask is defined as s. crd Then we have: (1) in, d is a configurable parameter. i Let be the pixel distance from the center point of the bounding box of the i-th mask to the center of the circle. The average pixel area of ​​all masks, or... s is the average pixel area of ​​the effective mask; crd The higher the value, the higher the priority for capturing the object to which the first mask belongs.

[0117] The effective mask can be a mask with a confidence score higher than a set confidence threshold, and / or a mask with a pixel area greater than or equal to an area threshold; wherein the confidence score is used to indicate the segmentation confidence of the corresponding mask.

[0118] In summary, by dynamically calculating the distance threshold based on the diagonal length of the bounding box of the mask, and using this distance threshold as the radius, the distribution density of other objects in the neighborhood is statistically analyzed. Then, based on the pixel distance from the center point of the neighboring mask to the center point of the current mask, a sparsity score is calculated. This adaptively quantifies the local crowding around an object: the greater the pixel distance and the fewer the neighboring objects, the higher the sparsity score, indicating a more open operating space around the object and a lower risk of collision or interference during grasping. Therefore, spatially isolated, easily accessible, high-priority grasping targets can be accurately identified, significantly improving the safety and success rate of grasping planning in disordered stacking scenarios.

[0119] Step S903: Determine the depth score of each mask based on the mapping depth of the object to which each mask belongs; wherein, the depth score is used to characterize the minimum distance between the object to which the corresponding mask belongs and the third plane.

[0120] The third plane is the plane where the data acquisition device is located.

[0121] In this embodiment of the application, the depth score of each mask can be determined according to the mapping depth of the object to which each mask belongs. The depth score of each mask is used to characterize the minimum distance between the object to which the mask belongs and the third plane. The minimum distance can indirectly indicate the exposure height of the object to which the mask belongs in the stack.

[0122] For example, the minimum mapping depth (i.e., minimum distance) of the i-th mask is denoted as The depth score of the i-th mask is Then we have: (2) in, It is the maximum value among the minimum mapping depths of all masks (i.e., the minimum distance between the bottommost object and the third plane). A higher depth score indicates that the object to which the i-th mask belongs is more likely to be on the upper layer of the stack, and thus has a higher capture priority. In other words, in this application, the depth score of each mask is negatively correlated with the minimum distance between the object to which the mask belongs and the third plane.

[0123] As one possible implementation, the depth score of the i-th mask can also be normalized using the following formula: ;in, It refers to the maximum value among all the depth scores of the masks. It refers to the minimum value among all the depth scores of the masks.

[0124] Step S904: Determine the grasping score of the object to which each mask belongs based on the sparse score and / or depth score of each mask; wherein, the grasping score is used to indicate the success rate of the robot arm in grasping the object.

[0125] In the embodiments of this application, for any mask, the capture score of the object to which the mask belongs can be calculated based on the sparse score and / or depth score of the mask, wherein the capture score is positively correlated with the sparse score and the capture score is positively correlated with the depth score.

[0126] As an example, the capture score s of the object to which the j-th mask belongs can be calculated using the following formula. j : (3) in, It is the sparse fraction of the j-th mask. It is the minimum of the sparse fractions of all masks. It is the maximum value among all the sparse fractions of the masks. It is the depth score of the j-th mask. It is the minimum value of the depth score among all masks. It is the maximum value among all the depth scores of the masks.

[0127] As another example, the capture score s of the object to which the j-th mask belongs can also be calculated using the following formula. j : (4) in, and For pre-defined (or configurable) weighting coefficients.

[0128] Step S905: Based on the grabbing scores of multiple objects, determine the candidate objects that can be grabbed from the multiple objects.

[0129] In this embodiment of the application, a candidate object with a relatively large capture score can be determined from multiple objects based on the capture scores of multiple objects.

[0130] For example, the N objects with the highest scores can be selected as candidate objects. Here, N is a pre-set threshold number.

[0131] Step S906: Based on the estimated pose and mask of the candidate object, determine the grabbable space of the candidate object.

[0132] Step S907: Based on the graspable space of the candidate objects, determine the target object to be grasped and the corresponding target grasping pose from the candidate objects, so as to control the robot arm to grasp the target object based on the target grasping pose.

[0133] It should be noted that the explanation of steps S906 to S907 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0134] In any embodiment of this application, the size information of each mask may further include the pixel area of ​​the mask. In this application, before step S902, multiple masks may be filtered based on the pixel area. For example, firstly, a third mask may be determined from multiple masks based on the confidence scores of multiple masks. Then, the average pixel area of ​​each third mask may be determined, and an area threshold may be determined based on the product of the average and a set second coefficient. Thus, in this application, third masks with pixel areas smaller than the area threshold may be filtered out.

[0135] The second coefficient is a pre-defined positive number that is less than or equal to 1.

[0136] Among them, the confidence score of the third mask is higher than the set score threshold. The confidence score is used to indicate the segmentation confidence of the corresponding mask.

[0137] For example, masks with confidence scores below a set confidence threshold can be filtered out, as can masks with pixel area A less than an area threshold.

[0138] In summary, by filtering out masks with relatively low confidence scores and masks with pixel areas smaller than the area threshold, unreliable detection results caused by occlusion, truncation, or perceptual noise can be effectively eliminated. Masks with relatively low confidence scores typically correspond to areas of uncertain object recognition, while small-area masks often represent objects that are incompletely exposed or deeply occluded, making it difficult to provide stable and effective grasping points. This dual filtering mechanism significantly improves the reliability and operability of object grasping, preventing the robotic arm from attempting to grasp unreachable or easily failed objects, thereby improving the overall success rate and operational efficiency of the disordered grasping system.

[0139] The robotic arm control method of this application combines the sparse score and / or depth score of the mask to comprehensively calculate the grasping score of each object, effectively quantifying the feasibility and success rate of grasping. A high sparse score indicates minimal occlusion around the object and sufficient operating space, while a high depth score indicates the object is located on a higher level and is easily accessible. Based on this multi-dimensional scoring mechanism, candidate objects are selected preferentially, prioritizing those that are fully exposed, free from surrounding interference, and reachable in pose. This significantly improves the intelligence and robustness of grasping decisions in disordered stacking scenarios, avoiding ineffective attempts on deeply buried, crowded, or easily collided objects, thereby improving overall grasping efficiency and system stability.

[0140] In any embodiment of this application, taking a multi-channel image as an example of an RGBD image acquired by a depth camera, the implementation principle of the technical solution provided in this application can be as follows: Figure 11 It mainly includes the following steps: in, Figure 11 (a) is a schematic diagram of the training process of the object segmentation model. Figure 11 (b) is a schematic diagram of the object grasping principle.

[0141] 1. Capture RGBD images using a depth camera, wherein the RGBD image may include an RGB image and a depth image; wherein the depth camera can observe multiple objects stacked in a disordered manner from the image. 2. Depth remapping based on the extrinsic parameters of the depth camera: Based on the extrinsic parameters of the depth camera, the true depth of each pixel in the depth map is remapped so that the mapped depth of each pixel is the vertical distance from the object surface to a plane above it (referred to as the third plane in this application). The third plane is parallel to the second plane on which the object is placed, and the height of the third plane is the same as the height of the depth camera.

[0142] It should be noted that this step is performed because robotic arms typically employ a vertically downward approach strategy in disordered grasping scenarios to avoid interference between the wrist or fingers and the side wall of the container holding the object. Figure 11 The depths mentioned in the process diagram are all vertically downward mapping depths.

[0143] 3. Use an object segmentation model to segment the masks of each object: Use an object segmentation model to segment the RGB image into instances to obtain the mask of a single object. In a stacked scene, the object segmentation model will output the masks of multiple objects.

[0144] The training method for this object segmentation model can be as follows: Figure 11 As shown in (a), the process includes: collecting images of stacked objects (RGB images) in simulated and real-world scenarios; annotating the stacked object images to obtain annotated stacked object images; wherein the annotated stacked object images contain bounding boxes for each object; and training an object segmentation model based on the annotated stacked object images using a supervised training mechanism.

[0145] The object segmentation model also outputs a confidence score for each mask as an indicator of segmentation confidence: the higher the confidence score, the more accurately the mask is likely to segment a single object.

[0146] 4. Candidate object selection based on depth and mask: The object segmentation model outputs a mask set S. The purpose of this step is to select the masks of the N candidate objects in S that are most suitable for capture. The specific method is as follows: Mask selection is performed based on the confidence scores, pixel area, minimum mapping depth (i.e., the minimum distance between the object to which the mask belongs and the third plane), and sparsity score (used to characterize the openness or clustering of the space surrounding the object to which the mask belongs) of multiple masks to obtain the N most suitable candidate object masks for capture. The main steps include the following: 4.1 Filter out low-score masks with confidence scores below the set threshold; 4.2 Filter out pixel areas The mask, where A is the pixel area of ​​the mask. As the second coefficient, This is the average pixel area of ​​all masks in the RGB image. The reason for this step is that small-area masks are likely to be incompletely displayed or covered by objects in the underlying layers, making them unsuitable for robotic arm grasping.

[0147] 4.3 Define the radius r as the average of the diagonal lengths of the bounding boxes of all masks multiplied by a first coefficient. Within a circle with the center point of each mask's bounding box as the center and r as the radius, denote the pixel distance d from the center point of the bounding box of the other unfiltered masks to the center of the circle. i Define the sparse fraction of a single object. .

[0148] Wherein, the sparse fraction s crd The higher the value, the higher the priority for grasping the object. The sparsity score is positively correlated with the openness of the space surrounding the object, or negatively correlated with the density of the space surrounding the object.

[0149] 4.4. Define the depth fraction of the i-th object as: ,in, Let be the minimum mapping depth of the mask for the i-th object. The higher the value, the more likely the i-th object is to be on the upper layer of the stack, and the higher the grab priority.

[0150] 4.5 Output the captured score .

[0151] 4.6 Filter to retain the N masks with the highest capture scores as candidate objects for capture. For example, N=5.

[0152] In summary, it can effectively filter out objects that are difficult to grasp, thereby improving the success rate of single-grab operations by the robotic arm.

[0153] 5. Pose Estimation: Outputs the estimated pose of the candidate object.

[0154] 6. Depth Checkpoint Sampling and Collision Detection: This step aims to check whether the robotic arm will interfere with its surroundings when grasping candidate objects from top to bottom. Candidate object orientations are categorized into two types: upright and upside down. Two different grasping strategies are employed for depth checkpoint sampling and detection for these two orientations. The specific methods are as follows: 6.1 If the angle between the axis of the candidate object and the table exceeds a set angle, it is judged to be placed upright (both the tail and the head are considered to be in this category); in the world coordinate system, a plane perpendicular to the axis direction is drawn near the highest point of the axis of the candidate object (referred to as the first plane in this application). Circles of different radii are drawn on the first plane with the axis as the center. Checkpoints are uniformly sampled on these circles, and the checkpoints in the world coordinate system are projected into the pixel space. Finally, checkpoints within the mask range of the candidate object are filtered out.

[0155] 6.2 If the angle between the axis of the candidate object and the table does not exceed the set angle, it is judged as upside down; in the world coordinate system, a rectangular inspection area is made along the axis of the candidate object, and check points are uniformly sampled in the rectangular inspection area. The check points in the world coordinate system are projected into the pixel space, and finally the check points within the mask range of the candidate object are filtered out.

[0156] 6.3. Retrieve the mapping depth of each of these checkpoints. If the difference between the mapping depth and the axial depth of the candidate object is less than a certain threshold, then determine that the radial straight line where the checkpoint is located (if it is a vertically placed line as shown in 6.1) or the straight line perpendicular to the axis (if it is an upside-down line as shown in 6.2) is an ungrabable direction.

[0157] 6.4 Select the target object with the largest graspable space from all candidate objects, and use the center position and direction of the largest graspable space as the target grasping pose (e.g., ...). Figure 4 (As shown); if the grabbable space of a candidate object is too small, it is considered ungrabable.

[0158] Figure 4 The grasping center is the pixel position (u, v) in the image coordinate system. The optimal grasping direction is represented by a deflection angle (θ) relative to the vertical axis of the image (the line containing the grasping direction is parallel to the second plane on which the object is placed (such as a desktop)). This pose is then transformed into a 6D pose in the robot coordinate system. ,in It can be determined by (u,v) and the camera's intrinsic and extrinsic parameters. and All are 0. It can be determined by θ and camera extrinsic parameters.

[0159] It should be understood that the sampling range of the checkpoints can be adjusted according to actual application needs in order to effectively avoid interference and collision between the end effector and the material pile, and further improve the success rate of the robot's gripping.

[0160] 7. Perform grabbing and repositioning: Set a fixed pre-grabbing gesture as... , where p iLet n be the joint angles of the robotic arm, with n joints, such that the closing direction of the index finger and thumb is approximately horizontal. Let the direction of the line connecting the centers of the index finger and thumb be the Euler angle output in step 6. The center point is the position output in step 6. ,like Figure 5 This diagram illustrates the grasping method for objects placed vertically. After grasping, the object is returned to its original position, and the above steps are repeated to create a loop.

[0161] In summary, the object selection rules and collision detection rules proposed in this application can complete continuous unordered grasping tasks in a clear, accurate and efficient manner without training models or consuming additional computing power. They do not rely on highly specialized syndactyl grippers or end-effector suction cup structures, do not require large GPUs, and adopt a 3D pose estimation method, which has a higher grasping success rate for single scenes compared to reinforcement learning.

[0162] To achieve the above embodiments, this application also proposes a robotic arm control device.

[0163] Figure 12 This is a schematic diagram of the structure of a robotic arm control device provided for an exemplary embodiment of this application.

[0164] like Figure 12 As shown, the robotic arm control device 1200 may include: a segmentation module 1210, a first determination module 1220, a second determination module 1230, a third determination module 1240, and a control module 1250.

[0165] The segmentation module 1210 is used to perform instance segmentation on the multi-channel image of the environment in which the robot is located in response to the grasping task of the robot arm, and obtain masks of multiple objects; wherein, the multi-channel image carries the depth information of each pixel; The first determining module 1220 is used to determine the candidate objects that can be grabbed from multiple objects based on the size information of multiple masks and the depth information of the objects to which they belong. The second determining module 1230 is used to determine the graspable space of the candidate object based on the estimated pose and mask of the candidate object. The third determining module 1240 is used to determine the target object to be grasped and the corresponding target grasping pose from the candidate objects based on the graspable space of the candidate objects. The control module 1250 is used to control the robotic arm to grasp the target object based on the target grasping pose.

[0166] In one implementation of this application, the second determining module 1230 is used to: perform pose estimation on candidate objects based on multi-channel images to obtain estimated poses of candidate objects; classify candidate objects by placement method based on estimated poses of candidate objects to obtain placement categories of candidate objects; and determine the grabbable space of candidate objects based on placement categories and masks of candidate objects.

[0167] In one implementation of this application, the second determining module 1230 is configured to: project each pixel belonging to a candidate object in a multi-channel image from pixel space to world space to obtain the spatial position of each pixel belonging to the candidate object; determine the axis of the candidate object based on the spatial position of each pixel belonging to the candidate object; sample multiple target checkpoints from the surrounding area of ​​the candidate object based on the axis of the candidate object, the placement category, and the mask; and determine the grabbable space of the candidate object based on the difference between the depth information of the target checkpoints and the depth information of the axis.

[0168] In one implementation of this application, the second determining module 1230 is configured to: in response to the placement category being vertical, determine a first plane perpendicular to the axis based on the highest point of the axis; sample multiple first checkpoints on the first plane and project the multiple first checkpoints from world space to pixel space to obtain the pixel positions of the multiple first checkpoints; delete the first checkpoints whose pixel positions are located within the image area corresponding to the mask of the candidate object to obtain the retained target checkpoints.

[0169] In one implementation of this application, the second determining module 1230 is configured to: in response to the placement category being reversed, determine an inspection area based on the direction of the axis, and sample multiple second inspection points in the inspection area; project the second inspection points from world space to pixel space to obtain the pixel positions of the multiple second inspection points; and delete the second inspection points whose pixel positions are located within the image area corresponding to the mask of the candidate object to obtain the retained target inspection points.

[0170] In one implementation of this application, the inspection area is rectangular in shape and satisfies the following conditions: the length of the inspection area is parallel to the second plane; wherein, the second plane is a plane on which multiple objects are placed; and the width of the inspection area is parallel to the axis.

[0171] In one implementation of this application, the third determining module 1240 is used to: select the candidate object with the largest graspable space as the target object; and determine the target grasping pose based on the center position and direction of the graspable space of the target object.

[0172] In one implementation of this application, the depth information includes a mapping depth, which is obtained using the following module: The acquisition module is used to acquire the true depth of each pixel acquired by the acquisition device for acquiring multi-channel images; wherein, the true depth is used to indicate the distance from the spatial point corresponding to the pixel to the acquisition device; The remapping module is used to remap the true depth of each pixel based on the extrinsic parameters of the acquisition device to obtain the mapped depth of each pixel; wherein, the mapped depth is used to indicate the distance between the spatial point and the third plane where the acquisition device is located.

[0173] In one implementation of this application, the third plane satisfies at least one of the following conditions: the height of the third plane matches that of the acquisition device; the third plane is parallel to the second plane on which multiple objects are placed.

[0174] In one implementation of this application, the first determining module 1220 is configured to: determine the sparsity score of each mask based on the size information of each mask; wherein the sparsity score is used to characterize the distance between the object to which the corresponding mask belongs and the objects to which other masks belong; determine the depth score of each mask based on the mapping depth of the object to which each mask belongs; wherein the depth score is used to characterize the minimum distance between the object to which the corresponding mask belongs and the third plane; determine the grasping score of the object to which each mask belongs based on the sparsity score and / or the depth score of each mask; wherein the grasping score is used to indicate the success rate of the robot arm in grasping the object; and determine the candidate objects that can be grasped from the multiple objects based on the grasping scores of multiple objects.

[0175] In one implementation of this application, the size information includes the diagonal length of the bounding box of the mask; a first determining module 1220 is configured to: determine the mean of the diagonal lengths of the bounding boxes of each mask; determine a distance threshold based on the product of the mean and a set first coefficient; determine a second mask located within a circle with the center point of the bounding box of the first mask as the center and the distance threshold as the radius; wherein the first mask is any one of a plurality of masks; and determine the sparsity fraction of the first mask based on the pixel distance from the center point of the bounding box of the second mask to the center of the circle.

[0176] In one implementation of this application, the size information further includes the pixel area of ​​the mask. The first determining module 1220 is further configured to: determine a third mask from multiple masks based on the confidence scores of multiple masks; wherein the confidence score of the third mask is higher than a set score threshold, and the confidence score is used to indicate the segmentation confidence of the corresponding mask; determine the mean of the pixel area of ​​each third mask, and determine the area threshold according to the product of the mean and a set second coefficient; and filter out third masks whose pixel area is less than the area threshold.

[0177] It should be noted that the foregoing explanation of the robot control method embodiment also applies to the robot control device of this embodiment, and will not be repeated here.

[0178] In the robotic arm control device of this application embodiment, instance segmentation based on multi-channel images (including depth information) is introduced into the grasping task. Reliable and graspable candidate objects are selected by combining mask size and depth information, effectively eliminating unreliable targets caused by occlusion, truncation, or perceptual uncertainty. Based on the mask and estimated pose of the candidate object, its graspable space is determined, and the target object and its grasping pose are optimized accordingly. This enables high-success-rate grasping of small, densely stacked objects. This application is not limited to specific gripper types and can effectively identify and prioritize objects that are sufficiently exposed, reasonably sized, and have good orientational reachability for grasping. This significantly improves the success rate, robustness, and adaptability of the robotic arm in complex stacked environments, providing a feasible path for platforms such as humanoid robots that require highly flexible operational capabilities.

[0179] To implement the above embodiments, this application also proposes a robotic arm, wherein the robotic arm is used to implement the robotic arm control method as described in any of the foregoing embodiments.

[0180] To implement the above embodiments, this application also proposes a robotic arm, wherein the robotic arm integrates a robotic hand for implementing the robotic hand control method as described in any of the foregoing embodiments.

[0181] To implement the above embodiments, this application also proposes a robot, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the robotic arm control method as described in any of the foregoing embodiments.

[0182] To implement the above embodiments, this application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the robotic arm control method as described in any of the foregoing embodiments.

[0183] Figure 13 This is a schematic diagram of the structure of a robotic hand, robotic arm, or robot provided for exemplary embodiments of this application. (Refer to...) Figure 13 The robotic arm, robotic hand, or robot 1300 may include one or more of the following components: a processing component 1302, a memory 1304, a power component 1306, a multimedia component 1308, an audio component 1310, an input / output (I / O) interface 1312, a sensor component 1314, and a communication component 1316.

[0184] Processing component 1302 typically controls the overall operation of a robotic arm, robot 1300, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 1302 may include one or more processors 1320 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 1302 may include one or more modules to facilitate interaction between processing component 1302 and other components. For example, processing component 1302 may include a multimedia module to facilitate interaction between multimedia component 1308 and processing component 1302.

[0185] Memory 1304 is configured to store various types of data to support the operation of the robotic hand, robotic arm, or robot 1300. Examples of this data include instructions for any application or method used to operate on the robotic hand, robotic arm, or robot 1300, contact data, phone book data, messages, pictures, videos, etc. Memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0186] The power assembly 1306 provides power to various components of the robotic hand, robotic arm, or robot 1300. The power assembly 1306 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the robotic hand, robotic arm, or robot 1300.

[0187] The multimedia component 1308 includes a screen that provides an output interface between the robotic hand, robotic arm, or robot 1300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 1308 includes a front-facing camera and / or a rear-facing camera. When the robotic hand, robotic arm, or robot 1300 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0188] Audio component 1310 is configured to output and / or input audio signals. For example, audio component 1310 includes a microphone (MIC) configured to receive external audio signals when the robotic arm, robotic hand, or robot 1300 is in an operating mode, such as a call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory 1304 or transmitted via communication component 1316. In some embodiments, audio component 1310 also includes a speaker for outputting audio signals.

[0189] I / O interface 1312 provides an interface between processing component 1302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0190] Sensor assembly 1314 includes one or more sensors for providing status assessments of various aspects of the robotic hand, robotic arm, or robot 1300. For example, sensor assembly 1314 can detect the open / closed state of the robotic hand, robotic arm, or robot 1300; the relative positioning of components, such as the display and keypad of the robotic hand, robotic arm, or robot 1300; changes in position of the robotic hand, robotic arm, or robot 1300 (or one component of the robotic hand, robotic arm, or robot 1300); the presence or absence of user contact with the robotic hand, robotic arm, or robot 1300; the orientation or acceleration / deceleration of the robotic hand, robotic arm, or robot 1300; and temperature changes of the robotic hand, robotic arm, or robot 1300. Sensor assembly 1314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. In some embodiments, sensor assembly 1314 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0191] Communication component 1316 is configured to facilitate wired or wireless communication between the robotic hand, robotic arm, or robot 1300 and other devices. The robotic hand, robotic arm, or robot 1300 can access wireless networks based on communication standards, such as WiFi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 1316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1316 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), Bluetooth, and other technologies.

[0192] In an exemplary embodiment, the robotic hand, robotic arm, or robot 1300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0193] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1304 including instructions, which can be executed by a robotic hand, robotic arm, or processor 1320 of robot 1300 to complete the above-described method. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0194] To implement the above embodiments, this application also proposes a chip, wherein the chip includes an interface circuit and a processing circuit coupled to each other. The interface circuit is used to input or output signals, and the processing circuit is configured to execute the robot control method provided in any of the foregoing embodiments.

[0195] Figure 14 This is a schematic diagram of a chip structure proposed as an exemplary embodiment of this application. See also... Figure 14 The diagram shown is a schematic representation of the structure of chip 1400, but it is not limited to this.

[0196] Chip 1400 includes processing circuit 1401, which is configured to execute any of the above-described robotic arm control methods.

[0197] In some embodiments, chip 1400 further includes one or more interface circuits 1402. As one possible implementation, the interface circuit 1402 is connected to memory 1403, and can be used to receive signals from memory 1403 or other devices, and to send signals to memory 1403 or other devices. For example, the interface circuit 1402 can read instructions stored in memory 1403 and send those instructions to processing circuit 1401.

[0198] In some embodiments, the interface circuit 1402 performs at least one of the communication steps such as sending and / or receiving in the above method, while the processing circuit 1401 performs other steps.

[0199] In some embodiments, the terms interface circuit, interface, transceiver pin, transceiver, etc., can be used interchangeably.

[0200] In some embodiments, chip 1400 further includes one or more memories 1403 for storing instructions. As one possible implementation, all or part of the memories 1403 may be located outside of chip 1400.

[0201] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the robotic arm control method as described in any of the foregoing method embodiments.

[0202] To implement the above embodiments, this application also proposes a computer program product having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the robotic arm control method as described in any of the foregoing method embodiments.

[0203] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0204] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0205] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0206] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and compact disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0207] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0208] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0209] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0210] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for controlling a robotic arm, characterized in that, include: In response to the grasping task of the robotic arm, the multi-channel image of the environment in which the robotic arm is located is segmented to obtain masks of multiple objects; wherein, the multi-channel image carries depth information of each pixel; Based on the size information of the multiple masks and the depth information of the objects to which they belong, candidate objects that can be grabbed are determined from the multiple objects. Based on the estimated pose and mask of the candidate object, the graspable space of the candidate object is determined. Based on the graspable space of the candidate objects, the target object to be grasped and the corresponding target grasping pose are determined from the candidate objects, so as to control the robotic arm to grasp the target object based on the target grasping pose.

2. The method according to claim 1, characterized in that, The step of determining the graspable space of the candidate object based on the estimated pose and mask of the candidate object includes: Based on the multi-channel image, the candidate object is pose estimated to obtain the estimated pose of the candidate object; Based on the estimated pose of the candidate objects, the placement methods of the candidate objects are classified to obtain the placement category of the candidate objects; Based on the placement category and mask of the candidate object, the grabbable space of the candidate object is determined.

3. The method according to claim 2, characterized in that, The step of determining the grabbable space of the candidate object based on its placement category and mask includes: Project each pixel belonging to the candidate object in the multi-channel image from pixel space to world space to obtain the spatial position of each pixel belonging to the candidate object; The axis of the candidate object is determined based on the spatial position of each pixel belonging to the candidate object; Based on the candidate object's axis, placement category, and mask, multiple target checkpoints are sampled from the surrounding area of ​​the candidate object. The grabbable space of the candidate object is determined based on the difference between the depth information of the target checkpoint and the depth information of the axis.

4. The method according to claim 3, characterized in that, The step of sampling multiple target checkpoints from the vicinity of the candidate object based on its axis, placement category, and mask includes: In response to the placement category being upright, a first plane perpendicular to the axis is determined based on the highest point of the axis; Multiple first checkpoints are sampled on the first plane, and the multiple first checkpoints are projected from the world space to the pixel space to obtain the pixel positions of the multiple first checkpoints; The first checkpoint located within the image region corresponding to the mask of the candidate object is deleted to obtain the retained target checkpoint.

5. The method according to claim 3, characterized in that, The step of sampling multiple target checkpoints from the vicinity of the candidate object based on its axis, placement category, and mask includes: In response to the placement category being inverted, an inspection area is determined based on the direction of the axis, and multiple second inspection points are sampled in the inspection area; The second checkpoint is projected from the world space to the pixel space to obtain the pixel positions of the plurality of second checkpoints; The second checkpoint located within the image region corresponding to the mask of the candidate object is deleted to obtain the retained target checkpoint.

6. The method according to claim 5, characterized in that, The inspection area is rectangular in shape and satisfies the following conditions: The length of the inspection area is parallel to the second plane; wherein, the second plane is the plane on which the plurality of objects are placed; The width of the inspection area is parallel to the axis.

7. The method according to claim 1, characterized in that, The step of determining the target object to be grasped and its corresponding grasping pose from the candidate objects based on the graspable space of the candidate objects includes: The candidate object with the largest graspable space is selected as the target object. The target grasping pose is determined based on the center position and orientation of the graspable space of the target object.

8. The method according to any one of claims 1-7, characterized in that, The depth information includes a mapping depth, and the method for obtaining the mapping depth includes: The true depth of each pixel acquired by the acquisition device for acquiring the multi-channel image is obtained; wherein the true depth is used to indicate the distance from the spatial point corresponding to the pixel to the acquisition device; Based on the extrinsic parameters of the acquisition device, the true depth of each pixel is remapped to obtain the mapped depth of each pixel; wherein, the mapped depth is used to indicate the distance between the spatial point and the third plane where the acquisition device is located.

9. The method according to claim 8, characterized in that, The third plane satisfies at least one of the following conditions: The third plane is matched with the height of the acquisition device; The third plane is parallel to the second plane on which the plurality of objects are placed.

10. The method according to claim 8, characterized in that, The step of determining grabbable candidate objects from the plurality of objects based on the size information of the plurality of masks and the depth information of the objects to which they belong includes: Based on the size information of each mask, the sparsity fraction of each mask is determined; wherein, the sparsity fraction is used to characterize the distance between the object to which the corresponding mask belongs and the objects to which the other masks belong; Based on the mapping depth of the object to which each mask belongs, a depth score is determined for each mask; wherein, the depth score is used to characterize the minimum distance between the object to which the corresponding mask belongs and the third plane; Based on the sparse score and / or depth score of each mask, the grasping score of the object to which each mask belongs is determined; wherein the grasping score is used to indicate the success rate of the robotic arm in grasping the object; Based on the capture scores of the multiple objects, candidate objects that can be captured are determined from the multiple objects.

11. The method according to claim 10, characterized in that, The size information includes the diagonal length of the bounding box of the mask; The step of determining the sparsity fraction of each mask based on the size information of each mask includes: Determine the average length of the diagonal of the bounding box of each of the aforementioned masks; The distance threshold is determined by multiplying the mean value by a set first coefficient. Within a circle centered at the center of the bounding box of the first mask and with the distance threshold as its radius, a second mask is determined within the circle; wherein the first mask is any one of the plurality of masks. The sparsity fraction of the first mask is determined based on the pixel distance from the center point of the bounding box of the second mask to the center of the circle.

12. The method according to claim 10, characterized in that, The size information also includes the pixel area of ​​the mask. Before determining the sparsity fraction of each mask based on the size information of each mask, the method further includes: A third mask is determined from the multiple masks based on their confidence scores; wherein the confidence score of the third mask is higher than a set score threshold, and the confidence score is used to indicate the segmentation confidence of the corresponding mask. Determine the average pixel area of ​​each of the third masks, and determine the area threshold based on the product of the average and a set second coefficient; A third mask is used to filter out pixels whose area is less than the area threshold.

13. A robotic arm control device, characterized in that, include: The segmentation module is used to perform instance segmentation on the multi-channel image of the environment in which the robot is located in response to the grasping task of the robot, and obtain masks of multiple objects; wherein, the multi-channel image carries depth information of each pixel; The first determining module is used to determine a candidate object that can be grabbed from the plurality of objects based on the size information of the plurality of masks and the depth information of the objects to which they belong. The second determining module is used to determine the graspable space of the candidate object based on the estimated pose and mask of the candidate object. The third determining module is used to determine the target object to be grasped and the corresponding target grasping pose from the candidate objects based on the graspable space of the candidate objects. The control module is used to control the robotic arm to grasp the target object based on the target grasping pose.

14. The apparatus according to claim 13, characterized in that, The second determining module is used for: Based on the multi-channel image, the candidate object is pose estimated to obtain the estimated pose of the candidate object; Based on the estimated pose of the candidate objects, the placement methods of the candidate objects are classified to obtain the placement category of the candidate objects; Based on the placement category and mask of the candidate object, the grabbable space of the candidate object is determined.

15. The apparatus according to claim 14, characterized in that, The second determining module is used for: Project each pixel belonging to the candidate object in the multi-channel image from pixel space to world space to obtain the spatial position of each pixel belonging to the candidate object; The axis of the candidate object is determined based on the spatial position of each pixel belonging to the candidate object; Based on the candidate object's axis, placement category, and mask, multiple target checkpoints are sampled from the surrounding area of ​​the candidate object. The grabbable space of the candidate object is determined based on the difference between the depth information of the target checkpoint and the depth information of the axis.

16. A robotic arm, characterized in that, The robotic arm is used to implement the method according to any one of claims 1 to 12.

17. A robotic arm, characterized in that, The robotic arm integrates a robotic hand for implementing the method of any one of claims 1 to 12.

18. A robot, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 12.

19. A chip, characterized in that, The chip includes an interface circuit and a processing circuit that are coupled to each other. The interface circuit is used to input or output signals, and the processing circuit is used to implement the method of any one of claims 1 to 12.

20. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method described in any one of claims 1 to 12.

Images