Robot grasping control method, apparatus, device, storage medium and program product

Abstract

The application relates to a robot grasping control method, device, equipment, storage medium and program product, comprising: processing sensor data by using a preset target detection model to obtain instance information of a plurality of target objects; wherein the sensor data is data of a region to be grasped collected by a sensor component of a robot; determining a target to be grasped among the target objects according to a preset grasping strategy and the instance information; wherein the preset grasping strategy takes a target object with minimum grasping cost as the target to be grasped; and controlling a gripper component of the robot to grasp the target to be grasped according to a target grasping posture. The application can reduce the grasping difficulty and collision probability from the source based on the minimum grasping cost selection of the target to be grasped in the control of the robot grasping process, the overall process is lightweight, complex physical simulation is not needed, the real-time performance of the grasping control is significantly improved, and the application is suitable for dynamic multi-target scenes.

Description

Technical Field

[0001] This application relates to the field of robot grasping technology, and in particular to a robot grasping control method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the rapid development of industrial automation and intelligent manufacturing technologies, robotic grasping technology has been widely used in manufacturing, logistics and warehousing, and other fields. The core of robotic grasping tasks is to accurately identify target objects in complex environments and select the optimal grasping posture to achieve efficient and stable grasping operations.

[0003] Existing grasping technologies lack real-time and efficient grasping action selection and collision detection methods in multi-target environments. Especially in dynamic environments, they cannot respond to environmental changes in real time, resulting in high grasping failure rates, high computational resource consumption, and low efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a robot grasping control method, device, equipment, storage medium, and program product that can efficiently and accurately select the best grasping target and posture to complete high-precision grasping and operation when performing grasping tasks in complex dynamic environments, in order to address the above-mentioned technical problems.

[0005] In a first aspect, this application provides a robot grasping control method, including:

[0006] The sensor data is processed using a preset target detection model to obtain instance information of multiple target objects; wherein, the sensor data is the data of the area to be grasped collected by the robot's sensor components;

[0007] The target to be grasped is determined from the target object according to the preset grasping strategy and the instance information; wherein, the preset grasping strategy uses the target object with the minimum grasping cost as the target to be grasped;

[0008] The robot's gripper assembly is controlled according to the target grasping posture to grasp the target to be grasped.

[0009] In one embodiment, determining the target to be grasped in the target object according to the preset grasping strategy and the instance information includes:

[0010] The grab substitution component is calculated based on the instance information; wherein the grab substitution component includes the neighbor density component, the sweep collision component, the depth ranking component, and the axial misalignment component;

[0011] The target capture price is obtained by weighted fusion based on the capture price component and the dynamic influence weight corresponding to each capture price component;

[0012] Select the target object with the smallest target grab cost as the target to be grabbed.

[0013] In one embodiment, the preset target detection model is the YOLO target detection model; the instance information includes a mask and rotated bounding box parameters;

[0014] The step of calculating the capture valorem component based on the instance information includes:

[0015] The grasping-related parameters of the target object are calculated based on the mask and the parameters of the rotating rectangle. The grasping-related parameters include the centroid pixel coordinates of the mask, the mask depth statistics, and the grasping radius.

[0016] The capture valorem component is calculated based on the capture-related parameters and the instance information.

[0017] In one embodiment, the neighbor density component represents the number of instances within a grab radius around the target object;

[0018] The sweep collision component represents the proportion of the ratio of collisions between the gripper assembly and neighboring obstacle targets of the target object in the sweep area where the target object is being grasped;

[0019] The depth ranking component represents the ranking of the depth information of the target object;

[0020] The axial misalignment component represents the degree of alignment between the rotation angle of the target object and the gripping posture.

[0021] In one embodiment, the method further includes:

[0022] A quality score is calculated based on the detection confidence, depth validity, and occlusion degree corresponding to the instance information, and a quality threshold is determined based on the current environmental parameters; wherein, the current environmental parameters include image brightness, sharpness, and depth loss rate;

[0023] The quality score and the quality threshold are compared. If the quality score is greater than the quality threshold, the instance information is determined to be valid instance information.

[0024] The target to be crawled is determined based on the preset crawling strategy and the valid instance information.

[0025] In one embodiment, before controlling the gripper assembly of the robot to grasp the target according to the target grasping posture, the method further includes:

[0026] Obtain refined pose information based on the target to be captured;

[0027] The translation and rotation components of the current pose information of the gripper assembly are adjusted according to the refined pose information and the adjustment factor to obtain the target grasping posture.

[0028] Secondly, this application also provides a robot grasping control device, comprising:

[0029] The target detection module is used to process sensor data using a preset target detection model to obtain instance information of multiple target objects; wherein, the sensor data is data of the area to be grasped collected by the robot's sensor components;

[0030] The target determination module is used to determine the target to be grasped among the target objects according to the preset grasping strategy and the instance information; wherein, the preset grasping strategy uses the target object with the minimum grasping cost as the target to be grasped;

[0031] The grasping control module is used to control the gripper assembly of the robot to grasp the target to be grasped according to the target grasping posture.

[0032] Thirdly, this application also provides a robot device, including: a sensor assembly, a gripper assembly, and a control assembly; the control assembly is respectively connected to the sensor assembly and the gripper assembly;

[0033] The control component includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the robot grasping control method described in the first aspect.

[0034] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the robot grasping control method described in the first aspect.

[0035] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the robot grasping control method described in the first aspect.

[0036] In summary, this application proposes a robot grasping control method, apparatus, device, storage medium, and program product, comprising: processing sensor data using a preset target detection model to obtain instance information of multiple target objects; wherein the sensor data is data of the grasping area collected by the robot's sensor components; determining the grasping target among the target objects according to a preset grasping strategy and the instance information; wherein the preset grasping strategy uses the target object with the minimum grasping cost as the grasping target; and controlling the robot's gripper assembly to grasp the grasping target according to the target grasping posture. This application, in controlling the robot's grasping process, selects the grasping target based on the minimum grasping cost, which can reduce the grasping difficulty and collision probability from the source, and the overall process is lightweight, requiring no complex physical simulation, significantly improving the real-time performance of grasping control, and adapting to dynamic multi-target scenarios. Attached Figure Description

[0037] Figure 1 This is an application environment diagram of the robot grasping control method in one embodiment;

[0038] Figure 2 This is a flowchart illustrating a robot grasping control method in one embodiment;

[0039] Figure 3 This is a flowchart illustrating the steps for determining the target to be captured in one embodiment;

[0040] Figure 4 This is a flowchart illustrating the steps for determining the target to be captured in another embodiment;

[0041] Figure 5 This is a flowchart illustrating the robot grasping control method in another embodiment;

[0042] Figure 6 This is a structural block diagram of a robot grasping control device in one embodiment;

[0043] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] In related technologies, robotic grasping technology is widely used in manufacturing, logistics and warehousing, and other fields. The core of robotic grasping tasks lies in accurately identifying target objects in complex environments and selecting the optimal grasping posture to achieve efficient and stable grasping.

[0046] Existing grasping technologies have significant shortcomings in multi-target, dynamic environments. On the one hand, deep learning-based grasping methods consume large computational resources, have poor adaptability to environmental changes such as lighting and occlusion, and are prone to detection errors and grasping failures. On the other hand, collision detection methods based on physical simulation have high computational complexity and insufficient real-time performance, and cannot quickly respond to dynamic environmental changes, resulting in inefficient target selection, high collision risk, and persistently high failure rates in multi-target scenarios.

[0047] This application provides a robot grasping control scheme that can select the optimal grasping target and posture in real time and efficiently in complex dynamic multi-target environments, thereby reducing computational consumption and improving the grasping success rate.

[0048] The robot grasping control method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be various robotic devices, including but not limited to logistics robots, handling robots, and warehouse robots. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0049] In one embodiment, such as Figure 2 As shown, a robot grasping control method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0050] Step 201: Process the sensor data using a preset target detection model to obtain instance information of multiple target objects.

[0051] Among them, sensor data refers to the data collected by the robot's sensor components from the area to be grasped.

[0052] In this embodiment, sensor data is acquired by sensor components configured on the robot device. Specifically, the sensor components can be RGB cameras or depth cameras, and the sensor data can be image data captured by the RGB camera or point cloud data generated by the depth camera. The sensor data in this embodiment is used to construct the robot's visual understanding of the area to be grasped. It should be noted that the process of acquiring sensor data can be determined according to the needs of the actual application scenario.

[0053] In this embodiment, a pre-trained object detection model is used to process sensor data to identify instance information of multiple objects in the area to be grasped. Specifically, the instance information includes bounding boxes, masks, and category information. The bounding boxes indicate the position of the object in the image. The masks provide pixel-level contours, displaying the shape information of the object. Category information indicates the type of object, such as whether the object is a tool, a tin can, or a container.

[0054] In one embodiment, the target object can be a target-type object in the area to be grasped or a graspable object in the area to be grasped. The target object can be determined according to the needs of the actual application scenario. In a practical application scenario, a preset target detection model can filter objects of target type and determine the target object that meets the requirements by identifying the object's category information.

[0055] It should be noted that the preset target detection model in this embodiment can be configured according to the needs of the actual application scenario to adapt to the robot grasping scenario. In one embodiment, the preset target detection model in this embodiment can adopt the YOLO target detection model, which utilizes the characteristics of YOLO model such as high speed and high accuracy to adapt to robot grasping scenarios that require real-time response.

[0056] Step 202: Determine the target to be grasped among the target objects based on the preset grasping strategy and instance information. The preset grasping strategy uses the target object with the minimum grasp cost as the target to be grasped.

[0057] In this embodiment, the preset grasping strategy can also be called a congestion safety scoring mechanism. To ensure the stability and safety of the grasping action, this embodiment introduces a congestion safety scoring mechanism, which comprehensively considers the relative positions and collision risks between target objects to obtain the grasping cost of the target object. Furthermore, in the process of obtaining instance information, this embodiment can obtain instance information of multiple target objects. By comparing the grasping costs of each target object, the target object with the minimum cost is obtained and determined as the target to be grasped.

[0058] Specifically, the expression for the preset crawling strategy in this embodiment is:

[0059]

[0060]

[0061] in, Indicates the first The capture price corresponding to each target object Indicates the target to be captured. Indicates the first The neighbor density components corresponding to each target object Indicates the first The sweep collision components corresponding to each target object. Indicates the first The depth ranking component corresponding to each target object Indicates the first The axial misalignment component corresponding to each target object , , and These are the weighting coefficients. It is important to know that... It refers to the target object with the lowest capture cost among all target objects.

[0062] In practical applications, this can be achieved by setting... , , and The actual values ​​are used to adjust the degree of influence of each component on the total substitution valence. For example, if It is 0.4. , and All are 0.2, indicating that among all grabbing valence components, the collision risk corresponding to the sweeping collision component has the greatest impact on robot grabbing, and the target object with the lowest collision risk is given priority as the object to be grabbed.

[0063] Step 203: Control the robot's gripper assembly to grasp the target according to the target grasping posture.

[0064] In this embodiment, the target grasping posture of the gripper approaching and grasping the target can be calculated based on the 3D information of the target object. The target grasping posture includes grasping position information, gripper rotation angle, and gripper opening degree, etc., and the 3D information includes point cloud information and mask information.

[0065] In this embodiment, the planned target grasping posture is converted into specific motion commands for the robotic arm and gripper, driving the robot's gripper assembly to perform grasping actions in order to grasp the target to be grasped.

[0066] In practical applications, after the grasping action in step 203 is completed, if there are still target objects to be grasped in the grasping area of ​​the robot device, steps 202 and 203 can be run again to continue grasping target objects according to the robot grasping control method provided in this embodiment, until all target objects to be grasped have been grasped. In one embodiment, if a new target object appears in the grasping area, steps 201 to 203 can be run again to identify the instance information of the new target object, redetermine the target to be grasped according to the preset grasping strategy, and execute the grasping action.

[0067] In summary, this embodiment provides a robot grasping control method that selects the target to be grasped based on the minimum grasping cost, replacing the traditional random / sequential selection. This reduces the grasping difficulty and collision probability from the source, and the overall process is lightweight, requiring no complex physical simulation, significantly improving real-time performance, and can effectively adapt to dynamic multi-target scenarios.

[0068] In one embodiment, such as Figure 3 As shown, the target to be grabbed is determined from the target object according to the preset grabbing strategy and instance information, including:

[0069] Step 301: Calculate the grab substitution component based on the instance information. The grab substitution component includes the neighbor density component, the sweep collision component, the depth ranking component, and the axial misalignment component.

[0070] Step 302: A weighted fusion is performed based on the grab replacement cost component and the dynamic influence weight corresponding to each grab replacement cost component to obtain the target grab replacement cost corresponding to the target object. Specifically, the dynamic influence weight is as described above. In the expression , , and In a specific implementation, the dynamic influence weights are... , , and The design can be adaptively adjusted according to the degree of influence of each grabber substitution component in the actual application scenario, so that the calculation of the target grabber substitution value can be more adaptive through dynamic changes and can be adapted to a variety of application environments.

[0071] Step 303: Select the target object with the smallest target grab cost as the target to be grabbed.

[0072] In this embodiment, the preset target detection model is the YOLO target detection model. The instance information in this embodiment includes mask and rotated bounding box parameters. When calculating the grabbing valence component, this embodiment can calculate the grabbing-related parameters of the corresponding target object based on the mask and rotated bounding box parameters. These grabbing-related parameters include the mask centroid pixel coordinates, mask depth statistics, and grabbing radius. Then, the grabbing valence component is calculated based on the grabbing-related parameters and instance information.

[0073] Specifically, in this embodiment, the neighbor density component represents the number of instances within the grasping radius around the target object. The sweep collision component represents the proportion of the gripper assembly being collided with neighboring obstacle objects of the target object in the sweeping area of ​​the grasping target object. The depth ranking component represents the depth information ranking of the target object. The axial misalignment component represents the degree of alignment between the rotation angle of the target object and the grasping posture.

[0074] Specifically, after the YOLO object detection model detects a target object, the output instance information includes a mask. With rotating rectangle parameters , mask and rotating rectangle parameters Used to represent the position, shape, and pixel range of a target object in an image. Where:

[0075]

[0076] in, Centered on the rectangle , The width of the rectangle. The height of the rectangle. For rotation angle. Rotation rectangle parameters. Including the center of the rectangle Width of the rectangle ,high and rotation angle .

[0077] Specifically, the formula for calculating the centroid pixel coordinates of the mask is:

[0078]

[0079] It should be noted that in engineering implementation, the centroid pixel coordinates of the mask can be approximated using the center of the rectangular frame, i.e. .

[0080] The formula for calculating the mask depth statistics is:

[0081]

[0082] in, This is a depth map.

[0083] The formula for calculating the grab radius is:

[0084]

[0085] in, It is a proportionality coefficient and .

[0086] The formula for calculating the neighbor density component is:

[0087]

[0088] in, This is an indicator function that takes the value 1 when the masks overlap, and 0 otherwise. Indicates the current candidate crawling target , Indicates the current candidate crawl target Instances of other target objects compared one by one , This indicates the current candidate crawling target. Other instances within the surrounding radius Perform a traversal query to determine the instance. With examples Whether the masks overlap, in the instance With examples When the masks overlap, increment the neighbor count by one until all instances have been traversed. , obtain instance The total number of neighbors within the surrounding capture radius. Representation of instances The number of neighbors, i.e., the number of instances corresponding to the target object. The number of instances within the surrounding crawl radius.

[0089] The formula for calculating the swept-body collision component is:

[0090]

[0091] Among them, the sweeping collision component This indicates the proportion of the gripper assembly that is collided with neighboring obstacle objects in the sweep area of ​​the target object. For the looting zone, For safe collision obstacle zone, This represents the number of neighbor instances that could potentially collide. Specifically, the sweep area. It refers to Centered on, facing Rotational rectangular region, sweep area The expression is:

[0092]

[0093]

[0094]

[0095] in, The pixel width is calculated based on the geometry of the gripper. The pixel length is calculated based on the geometry of the gripper. and The intrinsic focal length component is composed of... For the safe width of the gripper opening, This refers to the gripper feed sweep length. and The unit can be set to meter (m). It is a two-dimensional rotation matrix.

[0096] Specifically, usage examples Neighbor mask constructs safe collision barrier zone For any neighbor instance Masking expansion is performed to provide a lateral safety clearance for the grippers. Safety collision obstacle zone. The expression is:

[0097]

[0098]

[0099]

[0100] in, Indicates morphological expansion. For radius structural elements, This represents the radius of expansion in pixels, calculated from depth. For safety clearance.

[0101]

[0102] in, This indicates the number of neighbors with similar depths who are likely to collide. This is a preset depth difference threshold. The preset depth difference threshold can be set according to the needs of the actual application scenario.

[0103] Specifically, assume that the input to the YOLO object detection model includes a depth map. Furthermore, the depth map has been aligned with the RGB image, and the YOLO object detection model outputs the first... Mask of an instance The formula for calculating depth information is:

[0104]

[0105] After obtaining all instances Then, sort them by depth from smallest to largest, with smaller depths being closer together, to obtain the depth ranking component. :

[0106]

[0107] in, That is, depth ranking component .

[0108] The formula for calculating the axial misalignment component is:

[0109]

[0110] in, Rotate by the target angle. For the preset grasping direction, the axial misalignment component is used to measure the alignment between the target's rotation angle and the grasping posture. In this embodiment, the target rotation angle can be calculated. The difference between the gripping direction and the preset gripping direction (such as the vertical direction) yields the axial misalignment component. It should be noted that the preset gripping direction can be configured according to the needs of the actual application scenario. When the preset gripping direction is vertical, .

[0111] Based on the above steps, this embodiment achieves high-precision grasping by comprehensively considering multiple factors such as the target object's neighbor density, collision risk, depth ranking, and axial misalignment. It efficiently scores the grasping cost of the target and selects the target with the lowest cost. By dynamically adjusting the target selection criteria and congestion safety score, the system can quickly adapt to changes in dynamic environments, ensuring the stability and high precision of the grasping action. Furthermore, when determining the target to be grasped, this embodiment does not rely solely on YOLO detection or IoU filtering. Instead, it uses cost minimization as the criterion, jointly quantifying and optimally selecting factors such as the number of overlapping mask neighbors, the mask region depth ranking, and the alignment error of the target angle relative to the preset grasping direction.

[0112] In one embodiment, such as Figure 4 As shown, the robot grasping control method also includes:

[0113] Step 401: Calculate the quality score based on the detection confidence, depth validity, and occlusion degree corresponding to the instance information, and determine the quality threshold based on the current environmental parameters. These current environmental parameters include image brightness, sharpness, and depth loss rate.

[0114] Step 402: Compare the quality score and the quality threshold. If the quality score is greater than the quality threshold, the instance information is determined to be valid instance information.

[0115] Step 403: Determine the target to be crawled based on the preset crawling strategy and valid instance information.

[0116] In this embodiment, to adapt to dynamic environments, a dynamic quality threshold adjustment mechanism is provided. This mechanism performs quality gating on candidate targets after target detection, preventing low-reliability targets from being included in the grasping decision under conditions such as changes in illumination, enhanced occlusion, and depth distortion. Specifically, for each detected target instance... The quality score is calculated based on the output information of the YOLO object detection model and sensor data. and with quality threshold Comparison, only when Determine instance information in time Based on the valid instance information, proceed to the subsequent capture price evaluation.

[0117] Specifically, a quality score is calculated based on the detection confidence, depth validity, and occlusion level corresponding to the instance information. The specific formula for calculating the quality score is as follows:

[0118]

[0119]

[0120]

[0121] in, Instance information output by the YOLO object detection model The detection confidence level, For instance information The effective depth ratio within the mask area, The degree of occlusion / overlap can be characterized by the percentage of overlap between the mask and the masks of neighboring target objects. , , These are the weighting coefficients. .

[0122] Specifically, quality threshold Adaptive adjustment based on environmental metrics, such as image brightness (Mean grayscale), sharpness (Laplace variance) and deep missing rate Given the input, we can obtain:

[0123]

[0124] In this embodiment, when the lighting dims or the image noise / blur increases, it typically manifests as an increase in image brightness. Reduce, clarity As the number of cells increases, the effective proportion of deep missing cells decreases (deep missing rate). (rise), at this time increase This allows for stricter filtering of valid instance information, retaining only instance information of target objects that are more reliably detected, have more complete depth, and less occlusion, thereby automatically adjusting the grabbing rules to prioritize grabbing high-quality visible targets and reduce false grabbing and collisions.

[0125] When the illumination is stable and the image quality is good, the system reduces This allows more instance information of target objects to enter the effective instance information set, improving the crawling throughput and maintaining the effect of subsequent cost optimization selection.

[0126] In one embodiment, such as Figure 5 As shown, before the robot's gripper assembly grasps the target according to the target grasping posture, the robot grasping control method further includes:

[0127] Step 501: Obtain refined pose information based on the target to be captured.

[0128] Step 502: Adjust the translation and rotation components of the current pose information of the gripper assembly according to the refined pose information and adjustment factor to obtain the target grasping posture.

[0129] In this embodiment, the pose information of the target in the camera coordinate system is calculated based on the target center point and angle of the target to be captured. :

[0130]

[0131] in, Let be a rotation matrix, representing the direction of the target. Let be the translation vector, representing the spatial position of the target. In this embodiment, based on depth information and camera intrinsic parameters, a depth alignment method is used to fine-tune the grasping posture, which can effectively ensure that the pose of the target to be grasped in the camera coordinate system is more accurate. The specific process of optimizing the pose information is as follows:

[0132] First, determine the initial pose. To obtain refined pose .

[0133] Second, perform a translation operation:

[0134]

[0135] Third, the rotation operation is performed using the spherical linear interpolation function SLERP:

[0136]

[0137] in, This is an adjustment factor used to balance the initial pose with the refined pose.

[0138] Based on the above steps, the system incorporates real-time depth information for target pose refinement and optimization, significantly improving the accuracy of target grasping. By combining depth map and camera intrinsic parameter refinement methods, the system can rapidly adjust the grasping pose in changing environments, thereby achieving precise grasping position and posture. The system can also dynamically adjust based on target depth information, ensuring the adaptability of the grasping action under different environmental conditions.

[0139] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0140] Based on the same inventive concept, this application also provides a robot grasping control device for implementing the robot grasping control method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more robot grasping control device embodiments provided below can be found in the limitations of the robot grasping control method described above, and will not be repeated here.

[0141] In one embodiment, such as Figure 6 As shown, a robot grasping control device 600 is provided, including: a target detection module 610, a target determination module 620, and a grasping control module 630, wherein:

[0142] The target detection module 610 is used to process sensor data using a preset target detection model to obtain instance information of multiple target objects; wherein, the sensor data is the data of the area to be grasped collected by the robot's sensor components;

[0143] The target determination module 620 is used to determine the target to be grasped among the target objects according to the preset grasping strategy and instance information; wherein, the preset grasping strategy uses the target object with the minimum grasping cost as the target to be grasped.

[0144] The grasping control module 630 is used to control the gripper assembly of the robot to grasp the target to be grasped according to the target grasping posture.

[0145] In one embodiment, the target determination module 620 is further configured to calculate the capture cost component based on instance information; wherein the capture cost component includes a neighbor density component, a sweep collision component, a depth ranking component, and an axial misalignment component; weighted fusion is performed based on the capture cost component and the dynamic influence weights corresponding to each capture cost component to obtain the target capture cost corresponding to the target object; and the target object with the minimum target capture cost is selected as the target to be captured.

[0146] In one embodiment, the target determination module 620 is further configured to calculate the grasping-related parameters of the corresponding target object based on the mask and the rotation rectangle parameters, wherein the grasping-related parameters include the centroid pixel coordinates of the mask, the mask depth statistics, and the grasping radius; and calculate the grasping valence component based on the grasping-related parameters and instance information.

[0147] In one embodiment, the target determination module 620 is further configured to calculate a quality score based on the detection confidence, depth validity, and occlusion degree corresponding to the instance information, and determine a quality threshold based on the current environmental parameters; wherein, the current environmental parameters include image brightness, sharpness, and depth loss rate; compare the quality score and the quality threshold, and determine the instance information as valid instance information if the quality score is greater than the quality threshold; and determine the target to be captured based on the preset capture strategy and the valid instance information.

[0148] In one embodiment, the grasping control module 630 is further configured to acquire refined pose information based on the target to be grasped; and adjust the translation and rotation components of the current pose information of the gripper assembly based on the refined pose information and the adjustment factor to obtain the target grasping posture.

[0149] In summary, this embodiment provides a robot grasping control device that significantly improves the efficiency and accuracy of grasping tasks by combining target detection technology, congestion safety scoring algorithms, and real-time grasping posture optimization mechanisms. The robot grasping control device provided in this embodiment can evaluate the grasping cost of targets in real time and dynamically select the optimal grasping target and posture based on environmental changes, thereby ensuring efficient execution of grasping tasks. Furthermore, the robot grasping control device provided in this embodiment can not only handle multi-target grasping tasks in dynamic environments in real time, but also improves task execution efficiency and optimizes the overall system response time through parallel processing and fault-tolerant mechanisms. By reducing the reliance on computational resources from deep learning models and physical simulation methods, it effectively reduces the computational burden, improves production efficiency, and promotes the application of intelligent manufacturing technology. It provides strong support for achieving efficient and accurate automatic grasping task execution and offers a reliable automation solution for the manufacturing industry, further promoting the widespread application of robot automation in production.

[0150] Each module in the aforementioned robot grasping control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0151] In one embodiment, a robotic device is provided, including a sensor assembly, a grasping assembly, and a control assembly. The control assembly is connected to both the sensor assembly and the control assembly.

[0152] Specifically, the sensor components include vision sensors, laser sensors, etc., used to collect image data, point cloud data, etc. It should be noted that the actual type of sensor component in this embodiment can be configured according to the needs of the actual application scenario, and the sensor data collected by the sensor component can be determined according to the actual type of sensor component in the actual application scenario.

[0153] The gripping assembly includes an extension mechanism and a gripper mechanism. The extension mechanism is mechanically connected to the gripper mechanism and is used to drive the gripper mechanism to perform rotational, extension, and other movement operations. It should be noted that the actual structure of the gripping assembly can be designed according to the needs of the specific application scenario.

[0154] Specifically, the control component is used to execute the robot grasping control method in the above embodiments.

[0155] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a robot grasping control method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0156] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0157] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0158] A pre-defined target detection model is used to process sensor data, obtaining instance information of multiple target objects. The sensor data consists of data collected by the robot's sensor components from the area to be grasped. Based on a pre-defined grasping strategy and the instance information, the target to be grasped is determined among the target objects. The pre-defined grasping strategy uses the target object with the minimum grasping cost as the target to be grasped. The robot's gripper assembly is controlled to grasp the target according to the target grasping posture.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0160] A pre-defined target detection model is used to process sensor data, obtaining instance information of multiple target objects. The sensor data consists of data collected by the robot's sensor components from the area to be grasped. Based on a pre-defined grasping strategy and the instance information, the target to be grasped is determined among the target objects. The pre-defined grasping strategy uses the target object with the minimum grasping cost as the target to be grasped. The robot's gripper assembly is controlled to grasp the target according to the target grasping posture.

[0161] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0162] A pre-defined target detection model is used to process sensor data, obtaining instance information of multiple target objects. The sensor data consists of data collected by the robot's sensor components from the area to be grasped. Based on a pre-defined grasping strategy and the instance information, the target to be grasped is determined among the target objects. The pre-defined grasping strategy uses the target object with the minimum grasping cost as the target to be grasped. The robot's gripper assembly is controlled to grasp the target according to the target grasping posture.

[0163] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0164] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robot grasping control method, characterized in that, include: The sensor data is processed using a preset target detection model to obtain instance information of multiple target objects; wherein, the sensor data is the data of the area to be grasped collected by the robot's sensor components; The target to be grasped is determined from the target object according to the preset grasping strategy and the instance information; wherein, the preset grasping strategy uses the target object with the minimum grasping cost as the target to be grasped; The robot's gripper assembly is controlled according to the target grasping posture to grasp the target to be grasped.

2. The method according to claim 1, characterized in that, The step of determining the target to be grasped in the target object according to the preset grasping strategy and the instance information includes: The grab substitution component is calculated based on the instance information; wherein the grab substitution component includes the neighbor density component, the sweep collision component, the depth ranking component, and the axial misalignment component; The target capture price is obtained by weighted fusion based on the capture price component and the dynamic influence weight corresponding to each capture price component; Select the target object with the smallest target grab cost as the target to be grabbed.

3. The method according to claim 2, characterized in that, The preset target detection model is the YOLO target detection model; the instance information includes mask and rotated rectangle parameters; The step of calculating the capture valorem component based on the instance information includes: The grasping-related parameters of the target object are calculated based on the mask and the parameters of the rotating rectangle. The grasping-related parameters include the centroid pixel coordinates of the mask, the mask depth statistics, and the grasping radius. The capture valorem component is calculated based on the capture-related parameters and the instance information.

4. The method according to claim 2, characterized in that, The neighbor density component represents the number of instances within the grab radius around the target object; The sweep collision component represents the proportion of the ratio of collisions between the gripper assembly and neighboring obstacle targets of the target object in the sweep area where the target object is being grasped; The depth ranking component represents the ranking of the depth information of the target object; The axial misalignment component represents the degree of alignment between the rotation angle of the target object and the gripping posture.

5. The method according to claim 1, characterized in that, The method further includes: A quality score is calculated based on the detection confidence, depth validity, and occlusion degree corresponding to the instance information, and a quality threshold is determined based on the current environmental parameters; wherein, the current environmental parameters include image brightness, sharpness, and depth loss rate; The quality score and the quality threshold are compared. If the quality score is greater than the quality threshold, the instance information is determined to be valid instance information. The target to be crawled is determined based on the preset crawling strategy and the valid instance information.

6. The method according to claim 1, characterized in that, Before the gripper assembly of the robot grasps the target according to the target grasping posture, the method further includes: Obtain refined pose information based on the target to be captured; The translation and rotation components of the current pose information of the gripper assembly are adjusted according to the refined pose information and the adjustment factor to obtain the target grasping posture.

7. A robot grasping control device, characterized in that, The device includes: The target detection module is used to process sensor data using a preset target detection model to obtain instance information of multiple target objects; wherein, the sensor data is data of the area to be grasped collected by the robot's sensor components; The target determination module is used to determine the target to be grasped among the target objects according to the preset grasping strategy and the instance information; wherein, the preset grasping strategy uses the target object with the minimum grasping cost as the target to be grasped; The grasping control module is used to control the gripper assembly of the robot to grasp the target to be grasped according to the target grasping posture.

8. A robotic device, characterized in that, include: A sensor assembly, a gripper assembly, and a control assembly; the control assembly is connected to both the sensor assembly and the gripper assembly. The control component includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the robot grasping control method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the robot grasping control method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the robot grasping control method according to any one of claims 1 to 6.

Images