Robot disordered grabbing method and system

Abstract

The invention relates to the technical field of robot grabbing, and discloses a robot disordered grabbing method and system, and the method comprises the steps: collecting scene three-dimensional data, recognizing a target object, and generating a grabbing pose and an evacuation direction; the target object is grabbed according to the grabbing pose, verification lifting is executed, verification data are collected, and state judgment data are obtained according to the verification data; acquiring historical data, and establishing and training a state judgment model; a state judgment model is called, state judgment is conducted according to verification data, and when a state judgment result is in an unstable grabbing state, countermeasures are generated, and the grabbing posture and the evacuation direction are updated; and when the state judgment result is in the stable grabbing state, the target object is taken out according to the current evacuation direction. According to the method, the grabbing stability, the taking-out success rate and the continuous operation reliability of the robot in the disordered stacking scene can be improved, the probability that the target object slips, collides and is jammed in the taking-out process is reduced, and the overall grabbing efficiency can be improved.

Description

Technical Field

[0001] This invention relates to the field of robot grasping technology, specifically to a method and system for unordered robot grasping. Background Technology

[0002] In robotic grasping scenarios, target objects are often distributed in a disordered stacking manner within bins, turnover boxes, or workbench areas. The robot needs to first identify the target objects, then determine the grasping position and extraction direction, and subsequently complete the gripping and extraction. In disordered stacking scenarios, target objects typically obstruct, press against, overlap, and interfere with each other. The exposed surfaces, accessible areas, and extraction channels of the target objects change with the stacking state, making the grasping process more prone to instability.

[0003] Most existing unordered grasping methods for robots focus on pre-grasping processing, such as identifying the target object through 3D vision, generating the grasping pose, and determining the extraction direction, before controlling the robot to perform grasping and extraction. While these methods can complete target recognition and path planning before grasping, they lack specific judgment on the actual grasping state after the robot completes the gripping action. In other words, whether the target object has formed a stable holding relationship and whether it can be smoothly extracted along the current direction after the robot completes the gripping action is often only reflected during the actual extraction process. As a result, if the gripping position shifts, the grasping relationship becomes unstable, or the path ahead is obstructed, slippage, collision, jamming, or extraction failure can easily occur during the subsequent extraction process.

[0004] Furthermore, existing methods for handling grasping failures typically involve re-grabbing or replanning, but they lack differentiation between different sources of failure. In reality, gripping position misalignment, grasping position failure, and obstructed retrieval channels represent different types of problems, each requiring different adjustments: some problems require correcting the grasping position, others require reselecting the contact area, and still others simply require changing the retrieval direction. Using the same backtracking or retrying approach for all these different situations not only increases the number of ineffective adjustments but may also cause the robot to repeatedly fail at the same position, impacting the efficiency and stability of disordered grasping. Summary of the Invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] To solve the above technical problems, the present invention provides the following technical solution: a robot disordered grasping method, comprising: step S1: collecting three-dimensional scene data of the work area and identifying the target object, generating grasping pose and withdrawal direction; Step S2: Control the robot to grasp the target object according to the grasping posture, perform verification lifting and collect verification data, and obtain state judgment data based on the verification data; Step S3: Obtain historical data, and build and train a state determination model based on the historical data; Step S4: Call the state determination model, determine the state based on the verification data, and when the state determination result is in an unstable grasping state, generate countermeasures and update the grasping pose and withdrawal direction. Step S5: When the status determination result is in a stable grasping state, retrieve the target object according to the current evacuation direction.

[0007] As a preferred embodiment of the robot unordered grasping method described in this invention, step S1 specifically comprises: Perform background separation on the 3D scene data, remove fixed scene areas, and retain entity data located in the grab space; The retained entity data is segmented to obtain the spatial contours of each target object; For each target object, extract the surface area exposed to the grasping space, calculate the center position and surface orientation of the surface area, and generate the grasping pose of the target object; Using the opposite direction of the surface orientation as the initial evacuation direction, the occupancy status in the scene's 3D data is detected along the initial evacuation direction; If there is interference in the initial evacuation direction, adjust the direction and re-detect until an interference-free evacuation direction is obtained; Output the grab pose and evacuation direction for each target object.

[0008] As a preferred embodiment of the robot unordered grasping method described in this invention, step S2 specifically comprises: Set the commanded lifting displacement. After the robot finishes gripping the target object, it will perform a short-range lifting along the withdrawal direction. The stroke of the short-range lifting is the commanded lifting displacement. During the verification lifting process, verification data is collected, including end effector feedback data, robot motion response data, and local verification 3D data. The end effector feedback data includes the gripper opening / closing amount, gripping force, and drive current. The robot motion response data includes commanded lifting displacement, actual lifting displacement, and joint torque. The local verification 3D data refers to the local point cloud data collected in the grasping area during the verification lifting process; Based on the local verification of 3D data, determine the changes in the target object's lifting height and spatial occupancy in the evacuation direction; The end effector feedback data, robot motion response data, target object lifting height and space occupancy change data in the withdrawal direction are used as state determination data.

[0009] As a preferred embodiment of the robot unordered grasping method described in this invention, the step of acquiring historical data and establishing and training a state determination model based on the historical data includes: The historical data includes historical state determination data and corresponding annotation data, with the annotation data indicating the state determination result to which the historical state determination data belongs. The state determination results include stable grasping state and unstable grasping state. The unstable grasping state includes offset clamping state, target failure state and channel failure state. A state determination model is established, which includes a first temporal convolutional branch, a second temporal convolutional branch, a third temporal convolutional branch, a feature fusion layer, a first-level determination layer, and a second-level determination layer. The state determination model is trained using a hierarchical supervised training method. The first-level determination layer uses stable and unstable grasping states in the labeled data as first-level supervision labels. The secondary decision layer is trained under supervision only on historical state decision data labeled as unstable grasping states, and uses the offset clamping state, target failure state, and channel failure state in the labeled data as secondary supervision labels.

[0010] Supervised training is performed using binary focus loss for the first-level decision layer and multi-class focus loss for the second-level decision layer. The weighted sum of the binary focus loss and the multi-class focus loss is used as the joint training loss. Input historical state determination data into the state determination model, and update the model parameters using backpropagation and adaptive moment estimation optimization algorithms based on the joint training loss. When the joint training loss converges or the preset number of training rounds is reached, training is stopped, and the state judgment model of the completed training is obtained.

[0011] As a preferred embodiment of the robot unordered grasping method described in this invention, the state determination model includes: The first temporal convolutional branch is used to extract temporal features from historical end effector feedback data; the second temporal convolutional branch is used to extract temporal features from historical robot motion response data; and the third temporal convolutional branch is used to extract temporal features from historical target object lifting height and spatial occupancy change data in the withdrawal direction. The temporal convolution branches all employ multi-scale temporal convolution blocks composed of one-dimensional causal dilated convolution and residual connections; The temporal features output from the first temporal convolution branch and the temporal features output from the second temporal convolution branch are fused together to obtain the offset determination features. The temporal features output from the first temporal convolutional branch and the temporal features output from the third temporal convolutional branch are fused together to obtain the target failure determination features. The temporal features output from the second temporal convolution branch are fused with the temporal features output from the third temporal convolution branch to obtain the channel failure determination features. Then, the temporal features output from all temporal convolution branches are integrated with the offset judgment features, target failure judgment features, and channel failure judgment features to obtain the global judgment features. The global judgment features are input into the first-level judgment layer, and the first-level judgment result is output, indicating whether the current grasping result belongs to a stable grasping state or an unstable grasping state. Input the offset determination features, target failure determination features, and channel failure determination features into the secondary determination layer, and output the secondary determination result of whether the current capture result belongs to the offset clamping state, target failure state, or channel failure state.

[0012] As a preferred embodiment of the robot's disordered grasping method described in this invention, the countermeasures include: When the state determination result is a misalignment clamping state, misalignment countermeasures are generated; Based on the feedback data from the end effector and the robot's motion response data, the offset direction and amount of the current grasping pose are determined. The robot is then controlled to release the target object, and the scene's 3D data is re-acquired. The target object is identified based on the re-acquired scene 3D data, and the grasping pose generation process is corrected by combining the offset direction and amount. The grasping pose and withdrawal direction are regenerated, resulting in an updated grasping pose and withdrawal direction. The robot is then controlled to grasp the target object again according to the updated grasping pose and withdrawal direction and perform the verification lifting again. When the status determination result is that the target has failed, generate the target failure response measures; The robot is controlled to release the target object and re-acquire the scene's 3D data. The target object is identified based on the re-acquired scene 3D data. The avoidance area is determined based on the contact area on the target object's surface corresponding to the current grasping pose. The grasping pose and withdrawal direction are regenerated to obtain the updated grasping pose and withdrawal direction. The robot is then controlled to re-grasp the target object according to the updated grasping pose and withdrawal direction and perform the verification lifting again. When the status determination result is a channel failure status, generate channel failure response measures; Retain the current grasping pose, adjust the current withdrawal direction based on the spatial occupancy change data in the withdrawal direction to obtain the updated withdrawal direction, control the robot to maintain the gripping of the target object, and complete the removal of the target object along the updated withdrawal direction.

[0013] A robot unordered grasping system employing any of the methods described in this invention, wherein: The acquisition module collects 3D scene data of the work area and identifies target objects, generating the grab pose and withdrawal direction; The verification module controls the robot to grasp the target object according to the grasping posture, performs verification lifting and collects verification data, and obtains state judgment data based on the verification data. The modeling module acquires historical data and builds and trains a state determination model based on the historical data. The judgment correction module calls the state judgment model, performs state judgment based on the verification data, and generates countermeasures and updates the grasp pose and withdrawal direction when the state judgment result is in an unstable grasping state. The execution module, when the status determination result is in a stable grasping state, completes the extraction of the target object according to the current evacuation direction.

[0014] The beneficial effects of this invention are as follows: The method of this invention sets up a verification lifting and state determination process before the formal retrieval, which can dynamically verify the current grasping result and make corresponding adjustments for different abnormal situations, thereby reducing invalid retries and retrieval failures; at the same time, it can improve the robot's grasping stability, retrieval success rate and continuous operation reliability in disordered stacking scenarios, reduce the probability of the target object slipping, rubbing and getting stuck during the retrieval process, and help improve the overall grasping efficiency. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is an overall flowchart of a robot unordered grasping method provided in Embodiment 1 of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0018] Example 1, referring to Figure 1 As an embodiment of the present invention, a robot unordered grasping method is provided, comprising: Step S1: Collect 3D scene data of the work area and identify target objects to generate grab pose and evacuation direction.

[0019] The robot performs a 3D scan of the work area to obtain 3D scene data. This 3D scene data is point cloud data corresponding to the work area, used to characterize the actual occupancy status of the target object inside the bin, the bin's inner walls, and the surrounding space. Each sampling point in the point cloud data corresponds to a spatial location, thereby reconstructing the 3D geometry of the current grasping scene. This provides the initial data foundation for subsequent target object identification, grasping pose generation, and determination of the withdrawal direction.

[0020] After obtaining the 3D scene data, background subtraction is first performed to remove data corresponding to the edges of the hopper, supporting components, and other fixed areas, retaining only the entity data within the grasping space. This process ensures that the effective portion of the 3D scene data primarily corresponds to the target object currently being grasped within the hopper, reducing interference from fixed scenes in the subsequent recognition process.

[0021] After background subtraction, clustering and segmentation are performed on the retained entity data. Spatially separated or clearly defined regions are divided into data blocks corresponding to different objects, thereby identifying each target object in the current scene. Target objects are those located within the current grasping scene that need to be grasped and retrieved by the robot.

[0022] After identifying the target object, the exposed surface facing outwards from the grasping space is extracted for each object, and normal estimation is performed on this exposed surface. The result of normal estimation is the outward normal direction of the surface, representing its orientation in three-dimensional space. The center position of the exposed surface is then used as the grasping position, and a grasping reference coordinate system is constructed using the normal direction of the exposed surface. The grasping pose corresponding to the target object is generated through point cloud grasping pose detection. The grasping pose represents the position and orientation of the robot's end effector when grasping the target object. The grasping pose directly corresponds to the currently exposed contact area of ​​the target object, enabling the robot to complete the grasp along the most accessible surface of the target object.

[0023] After obtaining the grasping pose, the approach direction of the end effector when grasping the target object is obtained based on the grasping pose. The reverse of the approach direction of the end effector is used as the initial withdrawal direction. Collision detection and path feasibility verification are performed along the expected motion path of the target object along the initial withdrawal direction to check whether the target object will have spatial interference with surrounding target objects or the material box structure during the removal process. If there is interference with the expected motion path corresponding to the current direction, the withdrawal direction is adjusted, and collision detection and path feasibility verification are performed again. The withdrawal direction is determined as the withdrawal direction corresponding to the target object when the adjusted direction meets the condition of non-interference removal.

[0024] Step S2: Control the robot to grasp the target object according to the grasping posture, perform verification lifting and collect verification data, and obtain state judgment data based on the verification data.

[0025] The robot first drives the end effector to approach the target object according to the grasping posture, and completes the gripping after reaching the grasping posture.

[0026] After gripping, the robot does not directly remove the target object as a whole along the evacuation direction. Instead, it first performs a verification lift. The verification lift refers to a short lifting process along the evacuation direction before the formal removal, with its stroke limited to a pre-set commanded lifting displacement. The commanded lifting displacement is less than the stroke required for the complete removal of the target object, ensuring that the target object remains inside the bin at the end of the verification lift. This allows for dynamic verification of the current gripping result without entering the formal removal stage.

[0027] It should be noted that the commanded lifting displacement is set based on the exposed height of the target object in the current gripping pose and the remaining retrieval stroke along the evacuation direction. First, the complete retrieval stroke required for the target object to reach the opening of the hopper or break free from the surrounding stacking constraints from the current gripping position is determined based on the scene's 3D data. Then, a portion of this complete retrieval stroke is taken as the commanded lifting displacement. This commanded lifting displacement must be greater than the minimum displacement required for observable attitude and load changes after the target object is gripped, and it must also ensure that the target object is still inside the hopper and has not broken free from the surrounding constraints when the lifting is completed.

[0028] During the lifting verification process, the robot simultaneously collects verification data. The verification data consists of end effector feedback data, robot motion response data, and local verification 3D data. The end effector feedback data includes the gripper opening and closing amount, gripping force, and drive current; the robot motion response data includes the commanded lifting displacement, the actual lifting displacement, and the joint torque.

[0029] During the verification lifting process, local 3D data was acquired from the grasped area to obtain local verification 3D data. The local verification 3D data consists of point cloud data re-acquired around the target object and its adjacent areas during the verification lifting process, and is used to reflect the real-time spatial state of the target object after lifting.

[0030] Based on the locally verified 3D data, point cloud registration is performed between the local point cloud acquired at the start of the verification lift and the local point cloud acquired at the current sampling time. The point cloud region corresponding to the target object is extracted from the registration result. Then, based on the change in the center of gravity position of the point cloud region corresponding to the target object, the displacement of the center of gravity along the withdrawal direction is calculated, and this displacement is determined as the target object lift height. The target object lift height represents the actual lifting amplitude of the target object relative to the initial grasping position along the withdrawal direction during the verification lift process.

[0031] Simultaneously, using the evacuation direction as the analysis axis, grid occupancy statistics are performed on the local point cloud within a preset channel range in front of the target object. The grid occupancy result at the current sampling time is compared with the grid occupancy result at the start of the verification lift to obtain spatial occupancy change data in the evacuation direction. This spatial occupancy change data in the evacuation direction represents the degree of change in the point cloud occupancy state within the channel in front of the target object relative to the start of the verification lift, and is used to characterize the channel obstruction situation when the target object continues to be retrieved along the evacuation direction. When new point cloud aggregations appear in front, the existing gaps decrease, or nearby objects simultaneously enter the evacuation path, the spatial occupancy change data in the evacuation direction increases, indicating that the current retrieval result is at risk of obstruction during the subsequent formal retrieval process.

[0032] The end effector feedback data, robot motion response data, target object lifting height and space occupancy change data in the withdrawal direction are used as state determination data.

[0033] It should be noted that the commanded lifting displacement represents the planned lifting amount of the robot, the actual lifting displacement represents the actual lifting amount of the robot, and the target object lifting height represents the actual lifting height of the target object relative to the initial grasping position.

[0034] The verification lifting process ends when the actual lifting displacement reaches the commanded lifting displacement. In other words, the robot stops lifting when the actual lifting displacement reaches the commanded lifting displacement, thus ending the verification lifting process.

[0035] If an abnormal response occurs prematurely during the verification lifting process, the verification lifting process can be terminated before the actual lifting displacement reaches the commanded lifting displacement. An abnormal response is any one of the following: an abnormal increase in clamping force or driving current, an abnormal increase in joint torque, or a rapid increase in space occupancy data in the withdrawal direction. When an abnormal response occurs, it indicates that continuing lifting along the current withdrawal direction carries the risk of grasping instability or channel obstruction. At this point, the verification lifting process is stopped, and the collected state determination data is input into the state determination model for state determination. In this way, the verification lifting process can complete fixed-stroke verification under normal conditions and stop promptly under abnormal conditions, thus balancing determination accuracy and grasping safety.

[0036] Step S3: Obtain historical data, and build and train a state determination model based on the historical data.

[0037] First, historical data is acquired, consisting of historical state determination data and corresponding labeled data. Historical state determination data comprises historical end-effector feedback data, historical robot motion response data, historical target object lifting height, and historical space occupancy changes in the withdrawal direction. Specifically, the historical end-effector feedback data reflects changes in the gripping end during historical grasping, the historical robot motion response data reflects changes in the robot's motion during verification lifting, and the historical target object lifting height and historical space occupancy changes in the withdrawal direction reflect the actual spatial response of the target object during the verification lifting process.

[0038] The labeled data provides the actual state determination result for each set of historical state determination data. The state determination results are divided into stable grasping states and unstable grasping states. Unstable grasping states are further divided into offset clamping states, target failure states, and channel failure states. With this setup, the goal of model training is not merely to determine whether the grasping was successful, but to further distinguish which specific abnormal state an unstable grasping state belongs to, thus providing a direct basis for different subsequent response measures.

[0039] Secondly, a state determination model is established, which consists of a first time convolution branch, a second time convolution branch, a third time convolution branch, a feature fusion layer, a first-level determination layer, and a second-level determination layer.

[0040] The end effector feedback data, robot motion response data, and spatial change data correspond to the clamping state, motion state, and channel state, respectively. The three types of data have different effects when forming abnormal states. If they are directly mixed at the input end, it is easy to weaken the distinguishability between different abnormal states. Therefore, the historical state judgment data is first processed separately according to the data source and physical meaning, and then targeted fusion is carried out in the subsequent stage.

[0041] The first-time convolutional branch receives historical end effector feedback data to extract timing features during the gripping process. The gripper opening / closing amount, gripping force, and drive current continuously change over time during the lifting process. These changes reflect whether the gripping is stable, whether it is tilted, and whether the end effector load is abnormal. The first-time convolutional branch extracts common features of the gripping state in both short-term and continuous changes by performing layer-by-layer convolution operations on the historical end effector feedback data.

[0042] The second temporal convolutional branch receives historical robot motion response data to extract temporal features during the robot's verification lifting process. The commanded lifting displacement reflects the theoretical amount of motion the robot should complete at the current sampling moment, the actual lifting displacement reflects the actual amount of motion achieved at the current sampling moment, and the joint torque reflects the dynamic load changes experienced by the robot during the lifting process. By analyzing the historical robot motion response data, it can be determined whether the robot exhibits insufficient motion tracking, sudden load increases, or mechanical obstruction during the verification lifting process. The features extracted by the second temporal convolutional branch are primarily used to characterize the temporal variation patterns of the motion execution state during the verification lifting process.

[0043] The third temporal convolutional branch receives historical target object lifting height and historical spatial occupancy change data in the withdrawal direction, used to extract temporal features of the target object's spatial response. The historical target object lifting height characterizes the actual lifting magnitude of the target object relative to the initial grasping position during the verification lifting process; the historical spatial occupancy change data in the withdrawal direction characterizes the change in the spatial occupancy state within the forward channel as the target object moves along the withdrawal direction. If the target object moves upward synchronously with the robot's lifting, and the spatial occupancy state in the withdrawal direction remains unobstructed, it is closer to a stable grasping state; if the target object is insufficiently lifted or the spatial occupancy in the withdrawal direction increases rapidly, it is more likely to correspond to a target failure state or a channel failure state. The features extracted by the third temporal convolutional branch are used to characterize the interaction and change patterns between the target object and the surrounding space during the verification lifting process.

[0044] All three temporal convolutional branches employ multi-scale temporal convolutional blocks composed of one-dimensional causal dilated convolutions and residual connections. The one-dimensional causal dilated convolution, while preserving the temporal order, ensures that feature extraction at the current moment relies solely on the sampling data at the current moment and before, without introducing information from subsequent moments. Simultaneously, by progressively expanding the temporal perception range through dilated convolutions, the model can perceive both local abrupt changes between adjacent sampling moments and continuous trends throughout the entire validation upswing process.

[0045] Residual connections are used to directly transfer shallow features to deeper layers, reducing feature attenuation caused by multiple convolutions and preserving subtle anomalies during short-range validation lift. The multi-scale temporal convolution blocks enable each branch to extract both instantaneous anomaly responses and short-term continuous evolution features, making them more suitable for short-duration, rapidly changing grasping and judgment processes like validation lift.

[0046] Furthermore, after the three temporal convolutional branches output temporal features respectively, the feature fusion stage begins.

[0047] The feature fusion layer does not perform a single unified stitching, but rather performs three targeted fusions first, followed by a comprehensive fusion.

[0048] It should be noted that all three directional fusions are implemented using a pairwise gating interactive fusion method. First, the dimensions of the two temporal features involved in the fusion are aligned. Then, corresponding gating weights are generated based on the two temporal features. The gating weights are then used to adjust the two temporal features and combined with the interaction results of the two temporal features to obtain the corresponding judgment features.

[0049] The comprehensive fusion is achieved by weighted splicing fusion. First, the temporal features, bias judgment features, target failure judgment features, and channel failure judgment features output by all temporal convolution branches are aligned in dimension. Then, the fusion weight of each feature is calculated separately, and the features are spliced ​​after being weighted. Finally, a global judgment feature is generated through mapping.

[0050] Specifically, the first fusion process jointly processes the temporal features output from the first temporal convolution branch and the temporal features output from the second temporal convolution branch to obtain the offset determination features. Offset gripping states are usually manifested simultaneously by abnormal force distribution at the gripping end and abnormal robot motion following. Therefore, mapping the gripping state features to the motion state features is more conducive to identifying offset gripping states.

[0051] The second fusion process combines the temporal features output from the first and third temporal convolutional branches to obtain target failure determination features. Target failure typically manifests as the gripping end responding to an action, but the target object is not actually lifted sufficiently or fails to form a stable follow-up motion. Therefore, mapping the gripping state features to the target object's spatial response features is more effective in identifying target failure states.

[0052] The third fusion process combines the temporal features output from the second and third temporal convolutional branches to obtain channel failure determination features. Channel failure typically manifests as the robot having performed a lifting action, changes in joint load, and increased spatial occupancy in the withdrawal direction. Therefore, correlating motion state features with channel spatial response features is more effective in identifying channel failure states.

[0053] After completing three directional fusions, the temporal features output by all temporal convolution branches are then integrated with the offset judgment features, target failure judgment features, and channel failure judgment features to obtain global judgment features. Global judgment features reflect the overall joint response of the current grasping result in the gripping state, motion state, and spatial state.

[0054] The first-level decision layer receives global decision features and outputs a first-level decision result indicating whether the current crawling result belongs to a stable crawling state or an unstable crawling state. If the first-level decision layer outputs a stable crawling state, the second-level decision layer is not performed, and the result is output directly.

[0055] The secondary decision layer receives the offset determination features, target failure determination features, and channel failure determination features, and outputs a secondary decision result indicating whether the current capture result belongs to an offset clamping state, a target failure state, or a channel failure state. The secondary decision layer only participates in the fine-grained classification judgment when the current capture result is determined to be an unstable capture state, and further provides the specific classification type of the unstable capture state.

[0056] The model is trained using a hierarchical supervised training approach. The first-level decision layer uses stable and unstable grasping states from the labeled data as first-level supervision labels, and the training objective is to enable the model to reliably distinguish whether the current grasping result has met the stable grasping conditions. The second-level decision layer is trained only on historical state decision data labeled as unstable grasping states, and uses the corresponding offset gripping states, target failure states, and channel failure states as second-level supervision labels. The training objective is to enable the model to further distinguish the sources of different abnormal states under unstable grasping conditions.

[0057] The first-level decision layer uses a binary focus loss for supervised training, while the second-level decision layer uses a multi-class focus loss. The binary focus loss enhances the first-level decision layer's attention to hard-to-distinguish samples between stable and unstable grasping states, reducing the dominant role of easily distinguishable samples during training. The multi-class focus loss enhances the second-level decision layer's ability to distinguish hard-to-distinguish samples between biased gripping states, target failure states, and channel failure states. The binary focus loss and the multi-class focus loss are then summed according to set weights to obtain the joint training loss.

[0058] During training, historical state judgment data is input into the state judgment model to obtain first-level and second-level judgment results. The model output error is then calculated based on the joint training loss. Subsequently, backpropagation is used to update the parameters of each layer in the model, and an adaptive moment estimation optimization algorithm is used to adjust the parameter update magnitude, ensuring a relatively stable convergence speed across different parameter dimensions. As the number of training epochs increases, the model gradually develops the ability to recognize the temporal variation patterns in the historical state judgment data. Training stops when the joint training loss converges or when the preset number of training epochs is reached, resulting in the completed state judgment model.

[0059] Step S4: Call the state determination model, determine the state based on the verification data, and when the state determination result is in an unstable grasping state, generate countermeasures and update the grasping pose and withdrawal direction to re-grab the target object.

[0060] After completing the verification lift and obtaining the verification data, the trained state determination model is invoked to process the verification data during the current grasping process. The state determination model first extracts the temporal variation features in the input data, and then outputs the state determination result by combining the correlation between different data. The state determination result includes stable grasping state and unstable grasping state; when the determination result is an unstable grasping state, it further provides the offset clamping state, target failure state, or channel failure state.

[0061] Furthermore, when the state determination result is in an unstable grasping state, the robot is no longer allowed to continue to complete the retrieval directly in the current evacuation direction. Instead, it first generates corresponding countermeasures based on the state determination result.

[0062] First, when the state determination result is a misaligned gripping state, it means that the target object has been gripped by the end effector, but the current gripping position or gripping posture has deviated from the ideal grasping posture. If it continues to be taken out directly along the current state, it is easy for the gripping to loosen, the target object to rotate, or local collisions to occur during the subsequent evacuation process.

[0063] At this point, the end effector posture corresponding to the current grasping pose is used as the reference coordinate, and a local grasping coordinate is established in the plane perpendicular to the withdrawal direction. Then, based on the changes in torque at each joint during the lifting process, the direction of force change of the end effector in the local grasping coordinate is calculated using the robot's Jacobian matrix. According to the direction of force change, when the lateral force continuously increases on the same side, and the corresponding changes in gripper opening / closing, gripping force fluctuations, and drive current increments occur simultaneously on this side, that side is determined as the offset direction of the current grasping pose. The offset is not directly taken as an instantaneous value at a certain moment, but is determined based on the cumulative result of the lateral force changes during the lifting process, combined with the continuous displacement difference between the commanded lifting displacement and the actual lifting displacement.

[0064] After obtaining the offset direction and amount, the robot is controlled to release the target object and re-acquire the scene's 3D data. Since the target object may fall back, flip, or slide after release, the grasping pose before release is not directly used. Instead, the current exposed surface and current spatial position of the target object are re-identified in the re-acquired scene 3D data.

[0065] A local search region is established centered on the original grasping region, and the search region is moved in the opposite direction of the offset by the offset amount. Local point clouds are extracted in the moved region, and the surface point cloud of the target object is obtained based on the point cloud segmentation results. Then, the grasping reference coordinates are re-established by combining surface normal estimation and principal curvature analysis, and the grasping pose is regenerated by point cloud grasping pose detection.

[0066] The approach direction of the regenerated grasping pose is reversed as the initial withdrawal direction. Passable directions are then selected based on the point cloud occupancy around the target object, resulting in updated grasping pose and withdrawal direction. Finally, the robot re-grabs the target object using the updated grasping pose and withdrawal direction, and performs verification lifting again.

[0067] Second, when the state determination result is a target failure state, it indicates that the current source of the anomaly is not in the evacuation channel, but rather that the current grasping pose has failed to form an effective grasp. Specifically, this includes two situations: First, although the robot has completed the approach and gripping actions, the end effector has not formed a stable grip in the effective holding position of the target object; second, the end effector has already gripped the target object, but during the verification lifting process, the target object has not risen stably with the end effector, or it only rises briefly before slipping and falling back, thus failing to establish a stable follow-up relationship.

[0068] In the case of target failure, the problem lies in the failure to establish a reliable holding relationship at the current grasping position. If the grasping pose is repeatedly generated near the original contact area, it is easy to experience repeated instances of missed grasping, unstable grasping, or slippage after lifting. In this situation, first control the robot to release the target object, then re-acquire the scene's 3D data, and identify the target object in the re-acquired scene 3D data. Determine the avoidance area based on the contact area corresponding to the current grasping pose on the target object's surface, and then re-search for a grasping area on the exposed surface outside the avoidance area.

[0069] After determining the new grasping area, a local point cloud is extracted from the grasping area, and the surface data of the target object is re-separated from the local point cloud; then, the grasping reference coordinates are re-established according to the surface normal direction and principal curvature direction to generate a new grasping pose.

[0070] Subsequently, the approach direction of the new grasping pose is reversed as the initial evacuation direction, and the occupancy of the local space in front is counted along this direction. When there is enough space in front of this direction to allow passage for the current outer size of the target object, this direction is determined as the updated evacuation direction, thus obtaining the updated grasping pose and the updated evacuation direction.

[0071] The robot re-grasps the target object according to the updated grasping pose and the updated evacuation direction, and performs the verification lift again.

[0072] Third, when the status determination result is a channel failure state, it means that the robot has grasped the target object through the current grasping pose, but when the target object is taken out along the current withdrawal direction, there are problems such as increased occupancy of the channel ahead, enhanced local interference, or obstruction of the extraction path.

[0073] At this point, the current grasping pose is not changed, so that the holding relationship already established between the target object and the end effector is maintained, and the exception handling is focused on adjusting the withdrawal direction.

[0074] The data on changes in space occupancy along the evacuation direction is derived from local verification 3D data obtained during the verification lifting process. This data reflects changes in the local point cloud occupancy within the passageway in front of the target object. If the local space margin along the current evacuation direction decreases after verification lifting, or if a nearby target object enters the current evacuation path, it indicates a risk of obstruction along the current evacuation direction.

[0075] Based on this, multiple candidate evacuation directions are gradually generated within a preset deflection angle range on both sides of the current evacuation direction as the center; then, along each candidate evacuation direction, a channel region matching the current outer size of the target object is constructed within a preset extraction length range, and the local point cloud in the channel region is extracted from the local verification 3D data.

[0076] For each candidate evacuation direction, the local point cloud is first voxelized, then the number of voxels occupied in the corresponding channel area is counted, and the minimum clearance in that direction is calculated. Candidate evacuation directions with a minimum clearance smaller than the current cross-sectional size of the target object are first eliminated. Then, the direction with the smallest number of voxels occupied is selected from the remaining candidate evacuation directions as the updated evacuation direction. If no direction that meets the conditions is found within the current deflection angle range, the deflection angle range is expanded to continue the search until an updated evacuation direction is obtained.

[0077] Once the updated evacuation direction is determined, the robot maintains its current grasping posture and retrieves the target object along the updated evacuation direction.

[0078] It should be noted that even after implementing offset response measures, target failure response measures, or channel failure response measures, the current target object may still be temporarily unable to be stably removed. This situation indicates that the exposure or removal conditions of the current target object in the current stacking state still do not meet the requirements for stable grasping. For example, although the current target object has regenerated a grasping pose, its exposed surface is still small, or it is squeezed by surrounding target objects, and a stable holding relationship cannot be formed after re-grasping; or, although the current target object has been clamped, there are still channel obstructions along different evacuation directions, indicating that there are adjacent target objects blocking the removal path in front of or to the side of it.

[0079] At this point, the current target object can be released first, and other target objects with better exposure conditions around it can be grabbed first, or nearby target objects that block its extraction channel can be removed first. After the surrounding target objects are removed, the scene 3D data can be collected again and the grab pose and evacuation direction of the current target object can be regenerated.

[0080] Step S5: When the status determination result is in a stable grasping state, retrieve the target object according to the current evacuation direction.

[0081] When the state determination result is in a stable grasping state, it means that the target object has established a stable holding relationship with the end effector. Furthermore, during the verification lifting process, the lifting height of the target object changes synchronously with the actual lifting displacement, and the space occupancy change data in the evacuation direction does not show obvious obstruction characteristics. Therefore, the current grasping pose and the current evacuation direction both meet the conditions for continuing to retrieve the object.

[0082] The robot retrieves the target object following the current evacuation direction. During the retrieval process, it maintains its current grasping posture and continues to lift and move out along the direction verified at the end of the lifting process, allowing the target object to leave the current stacking area along the already verified path. Because the current evacuation direction has been dynamically verified during the lifting phase, the target object can leave the bin along a relatively unobstructed path during subsequent retrieval, thus reducing the possibility of collisions with adjacent target objects or the bin structure.

[0083] Once the target object detaches from the bin along the current evacuation direction, the robot moves the target object to the predetermined position and releases it, completing the grasping and removal of the current target object. Subsequently, the robot collects 3D scene data again to update the spatial distribution of the remaining target objects in the bin, and then proceeds to the next round of target object recognition, grasping pose generation, verification lifting, and state determination processes.

[0084] Thus, by setting up a verification lifting process before formal removal and executing corresponding countermeasures based on the state determination results, this invention enables the robot to first determine whether the current grasping result meets the stable removal conditions in disordered stacking scenarios, and then decide whether to continue removal or perform correction processing, thereby connecting the grasping pose generation, state determination and removal action into a continuous closed-loop grasping process.

[0085] Example 2, in an exemplary embodiment, also provides a robot disordered grasping system, including a data acquisition module, which acquires three-dimensional scene data of the work area and identifies target objects, and generates grasping pose and withdrawal direction.

[0086] The verification module controls the robot to grasp the target object according to the grasping posture, performs verification lifting and collects verification data, and obtains state judgment data based on the verification data.

[0087] The modeling module acquires historical data and builds and trains a state determination model based on that data.

[0088] The judgment correction module calls the state judgment model to make a state judgment based on the verification data. When the state judgment result is in an unstable grasping state, it generates a response measure and updates the grasping pose and withdrawal direction.

[0089] The execution module, when the status determination result is in a stable grasping state, completes the extraction of the target object according to the current evacuation direction.

[0090] If the above functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0092] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0093] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0094] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for unordered grasping by a robot, characterized in that, include: Step S1: Collect 3D scene data of the work area and identify target objects to generate grab pose and withdrawal direction; Step S2: Control the robot to grasp the target object according to the grasping posture, perform verification lifting and collect verification data, and obtain state judgment data based on the verification data; Step S3: Obtain historical data, and build and train a state determination model based on the historical data; Step S4: Call the state determination model, determine the state based on the verification data, and when the state determination result is in an unstable grasping state, generate countermeasures and update the grasping pose and withdrawal direction. Step S5: When the status determination result is in a stable grasping state, retrieve the target object according to the current evacuation direction.

2. The robot unordered grasping method as described in claim 1, characterized in that, Step S1 is as follows: Perform background separation on the 3D scene data, remove fixed scene areas, and retain entity data located in the grab space; The retained entity data is segmented to obtain the spatial contours of each target object; For each target object, extract the surface area exposed to the grasping space, calculate the center position and surface orientation of the surface area, and generate the grasping pose of the target object; Using the opposite direction of the surface orientation as the initial evacuation direction, the occupancy status in the scene's 3D data is detected along the initial evacuation direction; If there is interference in the initial evacuation direction, adjust the direction and re-detect until an interference-free evacuation direction is obtained; Output the grab pose and evacuation direction for each target object.

3. The robot unordered grasping method as described in claim 2, characterized in that, Step S2 is as follows: Set the commanded lifting displacement. After the robot finishes gripping the target object, it will perform a short-range lifting along the withdrawal direction. The stroke of the short-range lifting is the commanded lifting displacement. During the verification lifting process, verification data is collected, including end effector feedback data, robot motion response data, and local verification 3D data. The end effector feedback data includes the gripper opening / closing amount, gripping force, and drive current. The robot motion response data includes commanded lifting displacement, actual lifting displacement, and joint torque. The local verification 3D data refers to the local point cloud data collected in the grasping area during the verification lifting process; Based on the local verification of 3D data, determine the changes in the target object's lifting height and spatial occupancy in the evacuation direction; The end effector feedback data, robot motion response data, target object lifting height and space occupancy change data in the withdrawal direction are used as state determination data.

4. The robot unordered grasping method as described in claim 3, characterized in that, The acquisition of historical data and the establishment and training of a state determination model based on the historical data include: The historical data includes historical state determination data and corresponding annotation data, with the annotation data indicating the state determination result to which the historical state determination data belongs. The state determination results include stable grasping state and unstable grasping state. The unstable grasping state includes offset clamping state, target failure state and channel failure state. A state determination model is established, which includes a first temporal convolutional branch, a second temporal convolutional branch, a third temporal convolutional branch, a feature fusion layer, a first-level determination layer, and a second-level determination layer. The state determination model is trained using a hierarchical supervised training method. The first-level determination layer uses stable and unstable grasping states in the labeled data as first-level supervision labels. The secondary decision layer is trained under supervision only on historical state decision data labeled as unstable grasping states, and uses the offset clamping state, target failure state, and channel failure state in the labeled data as secondary supervision labels. Supervised training is performed using binary focus loss for the first-level decision layer and multi-class focus loss for the second-level decision layer. The weighted sum of the binary focus loss and the multi-class focus loss is used as the joint training loss. Input historical state determination data into the state determination model, and update the model parameters using backpropagation and adaptive moment estimation optimization algorithms based on the joint training loss. When the joint training loss converges or the preset number of training rounds is reached, training is stopped, and the state judgment model of the completed training is obtained.

5. The robot unordered grasping method as described in claim 4, characterized in that, The state determination model includes: The first temporal convolutional branch is used to extract temporal features from historical end effector feedback data; the second temporal convolutional branch is used to extract temporal features from historical robot motion response data; and the third temporal convolutional branch is used to extract temporal features from historical target object lifting height and spatial occupancy change data in the withdrawal direction. The temporal convolution branches all employ multi-scale temporal convolution blocks composed of one-dimensional causal dilated convolution and residual connections; The temporal features output from the first temporal convolution branch and the temporal features output from the second temporal convolution branch are fused together to obtain the offset determination features. The temporal features output from the first temporal convolutional branch and the temporal features output from the third temporal convolutional branch are fused together to obtain the target failure determination features. The temporal features output from the second temporal convolution branch are fused with the temporal features output from the third temporal convolution branch to obtain the channel failure determination features. Then, the temporal features output from all temporal convolution branches are integrated with the offset judgment features, target failure judgment features, and channel failure judgment features to obtain the global judgment features. The global judgment features are input into the first-level judgment layer, and the first-level judgment result is output, indicating whether the current grasping result belongs to a stable grasping state or an unstable grasping state. Input the offset determination features, target failure determination features, and channel failure determination features into the secondary determination layer, and output the secondary determination result of whether the current capture result belongs to the offset clamping state, target failure state, or channel failure state.

6. The robot unordered grasping method as described in claim 5, characterized in that, The countermeasures include: When the state determination result is a misalignment clamping state, misalignment countermeasures are generated; Based on the feedback data from the end effector and the robot's motion response data, the offset direction and amount of the current grasping pose are determined. The robot is then controlled to release the target object, and the scene's 3D data is re-acquired. The target object is identified based on the re-acquired scene 3D data, and the grasping pose generation process is corrected by combining the offset direction and amount. The grasping pose and withdrawal direction are regenerated, resulting in an updated grasping pose and withdrawal direction. The robot is then controlled to grasp the target object again according to the updated grasping pose and withdrawal direction and perform the verification lifting again. When the status determination result is that the target has failed, generate the target failure response measures; The robot is controlled to release the target object and re-acquire the scene's 3D data. The target object is identified based on the re-acquired scene 3D data. The avoidance area is determined based on the contact area on the target object's surface corresponding to the current grasping pose. The grasping pose and withdrawal direction are regenerated to obtain the updated grasping pose and withdrawal direction. The robot is then controlled to re-grasp the target object according to the updated grasping pose and withdrawal direction and perform the verification lifting again. When the status determination result is a channel failure status, generate channel failure response measures; Retain the current grasping pose, adjust the current withdrawal direction based on the spatial occupancy change data in the withdrawal direction to obtain the updated withdrawal direction, control the robot to maintain the gripping of the target object, and complete the removal of the target object along the updated withdrawal direction.

7. A robot unordered grasping system, applied to the robot unordered grasping method according to any one of claims 1 to 6, characterized in that, include, The acquisition module collects 3D scene data of the work area and identifies target objects, generating the grab pose and withdrawal direction; The verification module controls the robot to grasp the target object according to the grasping posture, performs verification lifting and collects verification data, and obtains state judgment data based on the verification data. The modeling module acquires historical data and builds and trains a state determination model based on the historical data. The judgment correction module calls the state judgment model, performs state judgment based on the verification data, and generates countermeasures and updates the grasp pose and withdrawal direction when the state judgment result is in an unstable grasping state. The execution module, when the status determination result is in a stable grasping state, completes the extraction of the target object according to the current evacuation direction.

Images