A robot grasping failure cause identification and adaptive regrasping method and system

Abstract

The present application relates to a kind of robot grabbing failure cause identification and adaptive regrasp method and system, the signal of gripper encoder opening, gripper motor current and robot joint current is collected in real time during grabbing execution, and discriminant feature is extracted;When determining failure, it is classified into preset cause category according to discriminant feature;Current scene visual information is collected and is different from reference scene before grabbing to relocate failed object, establish old frame to new frame space transformation and migrate failed grabbing pose;Surface attribute at the failed pose migrated to new frame is extracted, space penalty is constructed, and comprehensive score is obtained by fusing with network new frame score in the form of weight reduction;According to cause, regrasp strategy is selected and regrasp pose is determined by reordering according to comprehensive score, and after executing regrasp, it returns to determine closed loop.The present application can detect failure in real time without additional force / tactile sensor, does not require camera continuous visibility, can migrate experience across frame, generalization to similar surface, significantly improve the regrasp success rate under complex surface and incomplete point cloud.

Description

Technical Field

[0001] This invention belongs to the field of robot control and artificial intelligence technology, specifically relating to a method for monitoring and closed-loop recovery of grasping execution of embodied intelligent robots, and particularly to a method and system for identifying the cause of robot grasping failure and adaptive re-grasping based on arm and gripper proprioception and visual enhancement after failure. Background Technology

[0002] Point cloud-based six-DOF grasping pose detection networks can provide robots with a wealth of candidate grasping poses and their corresponding grasping scores. Downstream modules typically select the optimal candidate based on the score from highest to lowest for execution. This type of technology is relatively mature in generating "where to grasp and how to grasp," providing a basic capability for robot grasping. However, the closed-loop monitoring from grasping execution to confirmation of successful grasping is severely lacking in existing technologies. Most grasping pipelines are essentially open-loop, assuming success upon completion of the action, lacking real-time awareness of the execution process itself and effective recovery from failures.

[0003] In service scenarios such as intelligent wheelchair robots picking up fallen medicine bottles or water cups from the ground or grabbing items from a table and handing them to users, a single missed attempt or slip not only means task failure but may also lead to consequences such as spilled medicine or spilled hot water. Summarizing the situations that have occurred in the actual use of existing service robots on the market, the following shortcomings exist:

[0004] Existing methods for detecting grasping failures typically employ two approaches: one relies on a six-dimensional force / torque sensor on the wrist or a fingertip tactile array to sense the gripping state and slippage. These dedicated sensors increase hardware costs and are not commonly used in service robots. Another approach relies on a camera continuously tracking the object throughout the grasping process. However, in typical working poses, the robotic arm and gripper severely obstruct the camera's field of view, making continuous observation of the object before and after grasping unreliable. Even when a failure is detected, the same action is usually blindly retried in the original candidate pose, or the next highest-scoring candidate is simply selected, without analyzing or differentiating the root cause of the failure. This leads to repeated failures of the same type and a low success rate for re-grasping. Furthermore, when the object falls off or is moved due to a grasping failure, its position changes. In this case, the old candidate and the newly acquired data are not from the same set, rendering the method of selecting the next highest-scoring candidate within the same frame completely ineffective. Simply re-scoring the new point cloud from zero means that situations that occurred during previous failures cannot be utilized, and re-grasping becomes nothing more than blind trial and error. Under conditions of complex object surfaces (such as smooth surfaces or specular reflections) and incomplete point clouds, existing recapture methods, even with different candidates selected, may still repeatedly fail on the same object, making it difficult to improve the success rate of failure recovery.

[0005] In summary, existing technologies have shortcomings in areas such as failure detection during the grasping and execution process, handling of failures, re-grabbing after object displacement, and failure recovery under complex surface and defective cloud conditions. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a method and system for identifying the causes of robot grasping failures and adaptive re-grasping.

[0007] The technical solution of this invention is as follows:

[0008] A method for identifying the causes of failed grasping and adaptive re-grasping in a robot, wherein the robot is equipped with a gripper and a camera, and receives candidate grasping poses, grasping scores, and expected object widths output by a grasping detection network, the method includes:

[0009] Step S1. During the gripping process, collect three body sensing signals in real time: gripper opening, gripper motor current, and robotic arm joint current output by the gripper encoder.

[0010] Step S2. Extract discriminative features based on the three-channel body perception signals, and determine whether the capture was successful or failed.

[0011] Step S3. When a failure is determined, the failure is classified into one of the preset failure cause categories according to the discrimination features;

[0012] Step S4. After a failure is determined, the visual information of the current scene is collected and compared with the reference scene before the grasping execution to relocate the failed object. A spatial transformation from the old data frame to the new data frame is established, and the failed grasping pose recorded in the old data frame is transferred to the new data frame according to the spatial transformation.

[0013] Step S5. Extract the surface attributes of the object at the failed grasp pose when migrating to the new data frame, construct a spatial penalty including a position decay factor and a surface attribute matching factor, and use the spatial penalty to perform weighted fusion on the grasp score output by the grasp detection network on the new data frame to obtain a comprehensive score for each candidate grasp; the surface attribute matching factor generalizes the spatial penalty to regions on the object that have similar surface attributes to the failed grasp pose.

[0014] Step S6. Select the corresponding re-grabbing strategy according to the failure cause category, and re-sort the candidate grabs according to the comprehensive score to determine the re-grabbing pose. After re-grabbing, return to step S2 to determine success or failure again, thus forming an adaptive re-grabbing closed loop.

[0015] As a preferred technical solution of the present invention, in step S1, the collected gripper motor current and robotic arm joint current are smoothed by a sliding window; and before each gripping operation, the average steady-state current of the gripper is collected as an adaptive baseline and the average steady-state current of the joints is collected as an unloaded baseline in the unloaded static state of the robotic arm; the robotic arm joint current is converted into the load torque of each joint through a torque constant, and a scalar joint load characterization for discrimination is obtained by selecting load-sensitive joints or aggregating them according to a preset method; the discrimination feature in step S2 is based on the smoothed signal and is obtained after normalization relative to the adaptive baseline and the unloaded baseline.

[0016] As a preferred embodiment of the present invention, the discrimination features in step S2 include: the ratio of the steady-state opening after the gripper closes to the width of the expected object, i.e., the opening residual; the holding force level determined by the gripper motor current; the sudden drop in holding force during the lifting period determined by the gripper motor current; the joint load increment of the robotic arm; and the sudden drop in joint load during the lifting period; the joint load increment is used to normalize the sudden drop in joint load.

[0017] The determination is as follows: when the opening residual falls within a preset range, the clamping force level is not lower than a first threshold, the joint load increment is not lower than a load increment threshold, the clamping force drop is less than a second threshold, and the joint load drop is less than a third threshold, the grasping is determined to be successful; otherwise, it is determined to be a failure.

[0018] Furthermore, the failure causes described in step S3 include four categories: missing the target, incomplete clamping, slippage, and collision.

[0019] Among them: missing the target corresponds to low opening residual and clamping force level and no increase in joint load; incomplete clamping corresponds to abnormal opening and low clamping force level and joint load; slippage corresponds to normal holding period but sudden drop in clamping force and sudden drop in joint load during lifting period, both exceeding the corresponding threshold; collision and displacement corresponds to the situation where the discrimination features do not show the abnormal patterns of missing the target, incomplete clamping and slippage, the success criterion is not established, and the object's original position is confirmed to have shifted after failure by visual difference.

[0020] The failure classification is based on the feature vector constructed by the discriminative features and output by the classifier. The feature vector includes at least the opening residual, the compensation of the clamping force level, the joint load increment, the clamping force drop, and the joint load drop. When the difference between the maximum confidence and the second maximum confidence output by the classifier is less than the ambiguity threshold, the cause is determined to be uncertain and supplementary observation is triggered.

[0021] As a preferred technical solution of the present invention, the differential comparison and relocation in step S4 includes: first, transforming the point cloud of the current scene and the reference scene point cloud collected before the grasping execution to the same robot base coordinate system or world coordinate system, and then performing nearest neighbor difference; extracting the centroid displacement vector of the failed object in the region where the differential distance exceeds the displacement threshold, and determining whether the failed object has left the field of view; when it is determined that the failed object has left the field of view, selecting the view with the highest visibility of the area where the failed object is located from multiple candidate views that the robot arm can reach with the camera, controlling the camera to move to the view, re-collecting the point cloud and detecting the failed object to find it.

[0022] As a preferred embodiment of the present invention, step S4, establishing the spatial transformation and migration of the failed grasping pose, includes: performing rigid registration on the repositioned failed object to obtain the rigid body transformation from the old data frame to the new data frame and its registration confidence; when the registration confidence is not lower than the registration threshold, migrating the failed grasping pose to the new data frame according to the rigid body transformation; when the registration confidence is lower than the registration threshold, clearing the failed grasping pose and related cache of the failed object, and switching to determining the re-grabbing pose only based on the grasping score output by the grasping detection network on the new data frame.

[0023] As a preferred embodiment of the present invention, the surface properties in step S5 include the average curvature, reflectivity, and texture roughness of the object surface in the contact neighborhood of the failed grasping pose migrated to the new data frame; the spatial penalty is the sum of penalty terms for the historical failed grasping poses after migration, and each penalty term is obtained by multiplying the position attenuation factor, the orientation attenuation factor, and the surface property matching factor, and then multiplying by the penalty amplitude set according to the failure cause category;

[0024] The position decay factor decreases as the spatial distance between the candidate grab and the historical failed grab pose after migration increases; the direction decay factor decreases as the angle between their approach directions increases; and the surface attribute matching factor decreases as the difference between their surface attributes increases. Furthermore, the penalty magnitude corresponding to slip failure is greater than the penalty magnitude corresponding to miss failure.

[0025] As a preferred technical solution of the present invention, in step S5, after determining the re-grasping pose and before performing the re-grasping, a visual verification is also included: confirming on the newly acquired point cloud that the approach path of the re-grasping pose is unobstructed, the current orientation of the object matches the re-grasping pose, and the gripper width is consistent with the currently observed object thickness. If all three items pass, the re-grasping is performed.

[0026] As a preferred technical solution of the present invention, the re-grabbing strategy selected in step S6 according to the failure cause category includes: for missed grasp, after triggering supplementary observation, selecting the candidate grasp with the least spatial penalty weight reduction and the greatest difference from the original failed grasp pose; for incomplete gripping, adjusting the gripper opening width and gripping depth and fine-tuning the pose along the normal of the object surface; for slippage, increasing the gripping force setting value and reducing the acceleration during the lifting phase; for collision and displacement, fine-tuning the pose along the opposite direction of the object displacement and reducing the end-effector approach speed.

[0027] Furthermore, when the number of re-grabbing attempts reaches the upper limit, or the maximum value of the comprehensive score of each candidate grab is lower than the executable threshold, or the failed object is still not found after supplementary search and reliable registration cannot be established, the re-grabbing is terminated, the robotic arm is restored to a safe pose, and a voice prompt is given; when the failed object has been found but the registration confidence is lower than the registration threshold, the historical failure cache is cleared and the re-grabbing pose is determined only based on the grab score of the new data frame, instead of directly terminating the re-grabbing.

[0028] This invention also provides a robot grasping failure cause identification and adaptive re-grasping system. The robot is equipped with a gripper and a camera, and receives candidate grasping poses, grasping scores, and expected object widths output by a grasping detection network, including:

[0029] The signal acquisition unit is used to acquire three body sensing signals in real time during the grasping process: the opening degree of the gripper encoder, the gripper motor current, and the robotic arm joint current.

[0030] The state machine unit is used to extract discriminative features based on the three body perception signals and determine whether the capture is successful or unsuccessful.

[0031] The cause classification unit is used to classify the failure into one of a number of preset failure cause categories based on the discrimination features when the failure is determined.

[0032] The visual enhancement unit is used to collect visual information of the current scene after a failure is determined, perform differential comparison with the reference scene before the grasping execution to relocate the failed object, establish a spatial transformation from the old data frame to the new data frame, and transfer the failed grasping pose recorded in the old data frame to the new data frame according to the spatial transformation.

[0033] The semantic backfeeding unit is used to extract the surface attributes of the object at the failed grasp pose when it is migrated to the new data frame, construct a spatial penalty including a position decay factor and a surface attribute matching factor, and use the spatial penalty to perform weighted fusion on the grasp score output by the grasp detection network on the new data frame to obtain a comprehensive score for each candidate grasp. The surface attribute matching factor enables the spatial penalty to be generalized to the region on the object that has similar surface attributes to the failed grasp pose.

[0034] The closed-loop safety unit is used to select the corresponding re-grab strategy according to the failure cause category, and reorder the candidate grabs according to the comprehensive score to determine the re-grab pose. After the robot performs the re-grab, the state machine unit is handed over to determine the success or failure again, thus forming an adaptive re-grab closed loop.

[0035] According to the above-described solution, the beneficial effects of this invention are as follows:

[0036] This invention only needs to collect three inherent body perception signals of the robot itself—gripper opening, gripper motor current, and robotic arm joint current—in the success or failure determination stage to determine success or failure. Its success or failure perception capability is no longer based on wrist force / torque sensors or fingertip tactile arrays, nor on the continuous visibility of the grasped object by the camera. Therefore, it is not necessary to add force or tactile sensors to the robot to obtain the success or failure perception of the grasping execution process, reducing hardware costs and improving feasibility in service robots. Furthermore, under the conditions that the gripper and arm severely obstruct the camera's field of view in typical working poses and the grasped object cannot be continuously observed, the success or failure determination can still be carried out normally, overcoming the problem of failure due to occlusion when relying on visual online tracking.

[0037] This invention not only identifies failed capture events but also categorizes them into several preset cause categories based on extracted discriminative features, eliminating the limitation of existing technologies that can only detect failures but cannot determine the root cause. Furthermore, by selecting appropriate recapture strategies based on the failure cause category and reordering candidate captures according to a comprehensive score to determine the recapture pose, different actions can be taken for different root causes. Each recapture is followed by a new success / failure determination, forming a closed loop. The recovery effect can be instantly verified using the same set of ontology perception criteria, making adaptive recapture a convergent and verifiable process.

[0038] This invention also relocates failed objects by differentially comparing the current visual information with the reference scene before the grasping execution, and establishes a spatial transformation from the old frame to the new frame to transfer the failed grasping pose recorded in the old frame to the new frame. This realizes the mapping of the experience of "which position of the object will fail to grasp under this approach direction" from the old coordinate system to the new coordinate system. After re-collecting the point cloud, the experience of this failure is no longer discarded, providing spatial anchoring for evidence-based re-grasping on the new point cloud.

[0039] This invention introduces a surface attribute matching factor into the spatial penalty, so that the penalty not only applies to the spatial neighborhood of the failed pose, but also generalizes to other areas on the object that have similar surface attributes to the failed pose. Based on this, for cases where the grasp fails due to the smooth or slippery characteristics of the surface, even if the candidate grasp is far away from the original failure point in space, as long as its surface attributes are similar, it will be appropriately downweighted, thereby guiding the re-grab to avoid the easily failed surface and turn to reliable areas with different attributes.

[0040] In summary, this invention enables the robot to detect failures, identify root causes, transfer and generalize failure experience across frames, and then address the problems effectively even under adverse conditions such as camera obstruction, object displacement after failure, incomplete point clouds, and complex object surfaces. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method of the present invention;

[0042] Figures 2 to 5 The following are the normalized signal timing curves of the gripper opening, gripper motor current and robotic arm joint load under the following conditions: missed gripping, incomplete gripping, slippage, and collision / displacement.

[0043] Figure 6 A schematic diagram of feature space decision-making for four types of failure causes;

[0044] Figure 7 This is a bar chart comparing the re-capture success rates of the method of this invention with those of existing technologies;

[0045] Figure 8 This is a bar chart comparing the recapture success rates of the method of this invention and the blind retry method for four types of causes;

[0046] Figure 9 This is a system structure block diagram of the present invention. Detailed Implementation

[0047] To better understand the purpose, technical solution, and technical effects of this invention, the invention will be further explained and described below in conjunction with the accompanying drawings and embodiments. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. It is also stated that the embodiments described below are only for explaining this invention and are not intended to limit this invention.

[0048] like Figure 1 As shown, Figure 1 The first three steps in the diagram represent the execution phase (online real-time), showcasing the data flow from the grasping execution through step S1 (arm and claw three-way signal acquisition), step S2 (grasping state machine determination), to the successful output or failure transition to step S3 (cause four-class classification); the last three steps in the diagram represent the post-failure stage (offline analysis), showcasing step S4 (scene differentiation, object search, object relocation), step S5 (surface attribution, semantic backfeeding and reordering), and step S6 (closed-loop control and safe stop).

[0049] A method for identifying the causes of failed grasping and adaptive re-grasping is implemented on a robot equipped with a gripper and a camera. Before grasping, the robot receives candidate grasping poses, grasping scores, and expected object widths output by a grasping detection network, and selects candidates for execution based on their scores from highest to lowest. The method runs as a post-execution monitoring and failure recovery process after the execution actions of an existing grasping pipeline. It includes six sequential steps: signal acquisition and preprocessing, grasping state determination, failure cause classification, post-failure visual enhancement analysis, semantic enhancement feedback and reordering, and closed-loop control and safe stopping. After each re-grasping, it returns to the state determination step, thus forming a convergent adaptive re-grasping closed loop where the results can be reconfirmed. In a typical implementation scenario, the robot is an intelligent wheelchair robot equipped with a six-DOF robotic arm and a wrist depth camera, used to pick up medicine bottles, water cups, or tabletop items dropped on the ground and hand them to the user. During the grasping execution, the wrist camera's field of view is obstructed by the gripper and arm, preventing continuous observation of the grasped object.

[0050] Step S1: In the signal acquisition and preprocessing stage, the robot acquires three inherent body perception signals in real time during the grasping process, namely the current opening read by the gripper encoder. gripper motor current The load torque of each joint is obtained by converting the motor current of each joint of the robotic arm into torque constants. For subsequent judgment, the joint most sensitive to the current grasping load can be selected, or the load torques of each joint can be aggregated according to preset weights to obtain a scalar equivalent joint load. The sampling period is approximately 20 milliseconds. Since these three signals are all taken from the robot's existing encoder and motor driver, this step does not require adding wrist force or torque sensors, fingertip tactile arrays, or continuous camera observation of the object being grasped throughout the grasping process. This ensures that failure detection during grasping does not increase the cost of dedicated sensing hardware and can function normally even when the camera's field of view is severely obstructed. (Refer to...) Figures 2 to 5 These are the normalized signal timing curves of three signals: gripper opening, gripper motor current, and robotic arm joint load, respectively, under four causes: missed grip, incomplete gripping, slippage, and collision.

[0051] It should be noted that, Figures 2 to 5The three curves in the figure are visualized and normalized for ease of display. Their purpose is only to compare the relative trends of each channel and do not replace the specific definitions of the discriminant features in formulas 2 to 9. When plotting, the baseline mean of each channel can be collected under an unloaded, stationary state before the grasping operation begins, and then normalized according to the normal operating range of each channel to obtain dimensionless curves. The opening residual, clamping force level, joint load increment, and sudden drop, which are actually used by the algorithm for success / failure determination and cause classification, are calculated according to the formulas described later and the corresponding baselines, respectively. The horizontal axis in the figure represents the temporal progression direction during the grasping operation, used to characterize the relationship between each signal and time.

[0052] To suppress high-frequency sampling noise in the motor current and short-time spikes caused by gear friction, a sliding window low-pass smoothing is applied to the gripper motor current sequence. The formula is as follows: (Formula 1); where L is the width of the smoothing window, and l is the sample index within the window. Let L be the gripper motor current value at time t-1. In one specific embodiment, L=5, corresponding to a time window of approximately 100ms, with a sampling period of approximately 20ms; the same sliding window is applied to the scalar equivalent joint load sequence for smoothing to obtain... .

[0053] Step S2: To eliminate the drift of motor characteristics under different ambient temperatures and the deviation caused by different working conditions, before each gripping operation, two baselines are automatically collected and updated in the unloaded and stationary state of the robotic arm. One of them is the average steady-state current after the grippers are fully closed, which is the adaptive baseline. The second is the average joint torque when the robotic arm is stationary, i.e., the no-load baseline. All subsequent discrimination parameters involving current and torque are normalized relative to these two baselines. After the above smoothing and baseline adaptation processing, the signal-to-noise ratio of the gripper current during normal gripping can be improved by about 12 dB, providing a clean input for subsequent threshold determination that is unaffected by operating condition drift. This step supports the reliable foundation of real-time detection during execution.

[0054] In the grasping state determination stage, five-dimensional discriminative features are extracted based on the smoothed and normalized three-channel signals to determine success or failure. The first dimension is the gripper closing opening residual r, which is equal to the steady-state opening after the gripper closing action is completed. The expected object width is provided along with the capture and detection network when outputting the candidate pose. The ratio, that is (Formula 2).

[0055] The second dimension is the holding force level during the retention period. It is equal to the normalized value of the mean smooth current relative to the adaptive baseline during the holding period from the completion of closure to the start of the lifting action, i.e. (Formula 3); where, This refers to the holding period after the closure is completed and before the lifting motion begins. To achieve an adaptive baseline, the average steady-state current is taken after the grippers are fully closed, thus avoiding misjudgments caused by fixed thresholds under different operating conditions.

[0056] The third dimension is the sudden drop in clamping force. The value is determined by taking the negative of the minimum value of the first-order difference of the smoothed current during the lifting period, relative to the larger of the value after adaptive baseline normalization and zero. This quantity is the core signal for the cause of slippage. (Formula 4); where, To smooth the first-order difference of the current, To improve the motion time range, a reduction rate normalized relative to the baseline is used instead of an absolute reduction to avoid adaptation issues caused by different object weights.

[0057] The fourth dimension is the joint load increment. It is equal to the mean of the smoothed scalar equivalent joint load during the lifting period minus the no-load baseline, reflecting whether the object has been actually lifted, i.e. (Formula 5); where The original joint torque (observed torque) at time t after unloaded baseline compensation. The baseline is the scalar equivalent joint load mean value collected when the robotic arm is stationary before gripping. The joint load increment is used both to determine whether an actual lifting load has been established and to normalize sudden drops in joint load.

[0058] The fifth dimension is the sudden drop in joint load. (Joint-side signal of object detachment during lifting) is the larger of the negative of the minimum value of the first-order difference of the smoothed joint torque during the lifting period, and the normalized joint load increment, and zero. This is a robust form that introduces a very small positive constant to avoid division by zero, reflecting whether the object detached from the joint side during lifting. That is, (Formula 6); where, To avoid division by zero constants.

[0059] The final success criterion is the joint threshold determination of the above five-dimensional features, that is, when the opening residual falls within the preset interval. to The holding force level during the retention period is not lower than the threshold. The joint load increment is not lower than the load increment threshold. The sudden drop in clamping force is less than the threshold. And the sudden drop in joint load is less than the threshold. The crawl is considered successful if any condition is not met; otherwise, it is considered a failure. (Formula 7). Wherein, The calibration value, determined by statistics of no-load noise, can be higher than the upper limit of no-load load fluctuation; the remaining thresholds are determined by statistics of the calibration set. In one specific embodiment, , , , , .

[0060] Step S3: In the failure cause classification stage, when the state is determined to be a failure, the following five-dimensional discriminant features used for cause discrimination—opening residual, clamping force level compensation, joint load increment, clamping force drop, and joint load drop—are normalized by subtracting the calibration set mean and then dividing by the standard deviation to synthesize the failure feature vector Φ. (Formula 8). Therefore, Φ is a 5-dimensional normalized eigenvector; By incorporating classification input, the criterion "joint load not increased / low" used to distinguish between missed catch, incomplete clamping and other causes has a direct corresponding quantity in the classifier input.

[0061] The data is then fed into a lightweight linear softmax classifier for cause identification: c = argmax softmax(W·φ + b), c {miss, partial, slip, shift} (Formula 9). Where W is the weight matrix and b is the bias; when Φ is 5-dimensional and the output has 4 classes, it can be taken as... , There are a total of 24 learnable parameters (excluding the mean and standard deviation required for normalization), based on the actual dimensions of the linear model. {miss, partial, slip, shift} indicates that category c is taken from: miss (missing the target), partial (incomplete gripping), slip (slipping off), and shift (moving away from the target).

[0062] The four causes are distinguished by their distribution patterns in the feature space: Miss is characterized by an extremely low opening residual, meaning the gripper is almost completely closed, the clamping force is extremely low, and the joint load does not increase, indicating that the gripper never contacted the object; Partial is characterized by an abnormal opening, while the clamping force and joint load are low, indicating that the object is only partially clamped; Slip is characterized by normal closing and holding periods, but during the lifting period, the sudden drop in clamping force and the sudden drop in joint load simultaneously exceed the corresponding thresholds. This simultaneous drop in both channels is the core discrimination signal that distinguishes it from the other three types; Shift is characterized by the absence of typical abnormal patterns of Miss, Partial, and Slip in each discrimination feature, yet the success criterion is still not established, and after failure, the visual difference detects that the object has shifted from its original position, indicating that the object was touched and shifted by the gripper before closing and then slid out of the clamping range.

[0063] Causal confidence of classifier output Take the maximum value of softmax, that is When the difference in confidence between the second largest class and the largest class is less than the ambiguity threshold If the cause is determined to be uncertain, a supplementary observation process is triggered to make a final determination based on the visual difference results. Preferably, Because this normalized index classifier uses ontology-aware discriminative features as input, its single inference time is less than 0.1 milliseconds, and the average classification accuracy for the four causes is approximately 88%. Among them, the slippage category has the highest accuracy, approximately 94.5%, due to the strong discriminative power of the dual-channel sudden drop signal. Through this process, the system not only learns of the capture failure but also the root cause of the failure, providing direct evidence for subsequent targeted recovery. (Refer to...) Figure 6 A schematic diagram of the feature space decision for four types of failure causes. The horizontal and vertical axes represent two different dimensions of normalized features, which are dimensionless. This diagram shows the distribution of the four causes in the normalized feature space and the softmax decision boundary. The dashed circles mark the ambiguity areas: the sub-class interval. If so, then supplementary observations are needed.

[0064] Step S4: In the post-failure visual enhancement analysis stage, after the robotic arm returns to a safe pose to restore the camera's field of view to unobstructed, it re-acquires the depth point cloud of the current scene. Then, scene difference, object search and object relocalization and registration are performed in sequence.

[0065] Scene Differentiation: To avoid camera or robotic arm pose changes being mistaken for object displacement, the current point cloud is first analyzed using robot forward kinematics / camera extrinsic parameters. Point cloud of the reference scene collected before execution of the capture process Transform to the same robot base coordinate system or world coordinate system, then perform nearest neighbor difference. For each current point, calculate its distance to its nearest neighbor in the reference point cloud. (Formula 10); where, Let p be the nearest neighbor of point p in the reference point cloud. For the difference distance... Exceeding the displacement threshold The centroid displacement vector of the object that failed to be extracted from the region. This information is used to determine whether the failed object has left the field of vision.

[0066] When it is determined that the failed object has left the field of view, the object search begins: among multiple candidate viewpoints accessible by the robotic arm and camera, the viewpoint with the highest visibility to the area where the failed object is located is selected according to the visibility maximization criterion. : (Formula 11); where This is the set of candidate viewpoints accessible to the camera driven by the robotic arm. This is the area where the object disappears. Let v be the visibility indicator function for point p at a given viewpoint v. After the robotic arm moves the camera to this optimal viewpoint, it re-acquires the point cloud and detects the object to locate it.

[0067] Then, object relocation and registration are performed. For the recovered failed objects, rigid registration is performed, and the rigid body transformation that minimizes the sum of squared distances from the transformed object point set in the old frame to its nearest neighbor in the new frame is solved: (Formula 12); and the registration confidence level is obtained by combining the inlier ratio and the residual. When the registration confidence level is not lower than the registration threshold At that time, that is According to this rigid body transformation The failed capture poses recorded in the old frame are transferred to the new frame coordinate system, that is: (Formula 13); When the registration confidence level is lower than the registration threshold, i.e. Instead of performing the pose transfer, the historical failure cache of the object is cleared, and the re-grasp pose is determined solely based on the grasp score of the grasp detection network on the new point cloud.

[0068] In one specific embodiment, With an interior point distance threshold of 0.5 cm, the registration success rate was approximately 92%, the average registration residual was approximately 0.21 cm, and the deviation between the migrated failed pose and the corresponding position of the actual failed point in the new frame was approximately 0.35 cm, much smaller than the object radius. This step maps the failure experience from the coordinate system of the old data frame to the coordinate system of the new data frame after re-acquiring the point cloud. This ensures that the experience gained in this failure is not discarded and can still be used even if the object falls off or shifts, or if the old candidate and the new data no longer belong to the same set. At the same time, it provides a clear backtracking path for boundary cases where reliable relocation is not possible.

[0069] Step S5: In the semantic enhancement and reordering stage, for the failed capture poses that have been reliably migrated to the current new data frame. Visual properties of the object surface are extracted within its contact neighborhood, and a surface property vector is constructed: (Formula 14a); where, This is the normalized value of the average curvature of the local point cloud; This is a normalized value for intensity or reflectivity; This is the normalized value for surface texture roughness.

[0070] Subsequently, a semantically enhanced spatial penalty is constructed based on the migrated historical failed grasp poses and their surface attributes, and a weighted deduction is applied to the scores of each candidate grasp on the new point cloud: (Equation 14b). This represents the pose of the kth historical failed capture pose after being migrated to the current new data frame via Formula 13, avoiding direct calculation of distance and direction between the old and new coordinate systems.

[0071] Among them, the position attenuation factor , This represents the spatial distance between the candidate crawl point and the k-th historically failed crawl point after migration within the current data frame. It decreases as the distance increases.

[0072] Directional attenuation factor , This represents the angle between the candidate approach direction and the historical failed approach direction after the k-th migration. The effect decreases as the angle increases.

[0073] Surface property matching factor The penalty decreases as the difference in surface attributes between the candidate pose and the historical failed pose after migration increases. Therefore, the penalty acts not only on the spatial neighborhood of the failed pose, but also on regions with similar surface attributes in the current data frame. The penalty amplitude is set according to the cause, and the penalty amplitude corresponding to the slip failure is greater than the penalty amplitude corresponding to the miss failure.

[0074] The spatial penalty is weighted and fused with the original grasping score output by the grasping detection network on the new point cloud to obtain the comprehensive score after refeeding: (Formula 15); where This is the network's original capture score for this candidate on the current new point cloud. Since the penalty term is independent of the network score, this step can be integrated into the existing capture pipeline without modifying or retraining the upstream capture detection network.

[0075] Through this process, the experience of failure is improved from spatial-level avoidance of a single failed pose to surface-level avoidance of similar surface attribute regions. The re-grabbing is naturally guided to avoid the entire class of easily failed surfaces and turn to reliable regions with different attributes. In one specific embodiment, the highest-scoring candidate located on a smooth surface is penalized, and its overall score drops from about 0.85 to about 0.63. The second-highest-scoring candidate located in the rough label area is almost unpenalized, and its overall score is about 0.69. In the re-sorting, it jumps to the top and finally succeeds in grabbing.

[0076] Step S6: In the closed-loop control and safety stop phase, select the corresponding re-grab strategy according to the failure cause category, and re-rank the candidate grabs according to the comprehensive score to determine the re-grab pose. For missed grabs, after triggering supplementary observations to update the point cloud, select the candidate grab with the least spatial penalty and the largest difference from the original failed grab pose; for incomplete gripping, adjust the gripper opening width and gripping depth parameters according to the current observations, and fine-tune along the normal direction of the object surface near the original candidate pose; for slippage, increase the gripping force setting value and reduce the acceleration during the lifting stage. In one specific embodiment, increase the preset gripping current limit by 20% to 40%, reduce the lifting acceleration by 30% to 50%, and prioritize candidate grabs in the area near the object's center of mass; for collision and displacement, apply pose fine-tuning along the opposite direction of the object displacement on the original candidate pose and reduce the end-effector approach velocity. In one specific embodiment, reduce it by about 40%.

[0077] After the re-fetch is executed, it re-enters the state determination stage, using the same set of ontology perception criteria to instantly verify the success or failure of this re-fetch. The re-fetch process is constrained by three safety termination criteria: when the number of re-fetches reaches the upper limit; when the comprehensive score of all candidates is lower than the minimum executable threshold; or when the object is still not found after supplementary search and a reliable registration cannot be established. It needs to be distinguished from the rollback logic of Formula 13: if the object has been found but the registration confidence is lower than the threshold, then the historical cache is cleared and the process rolls back to the pure network scoring mode, without directly terminating the re-fetch.

[0078] (Formula 16); where, =1 indicates that the object loss marker has not been cleared after supplementary search; when the object is not found and effective registration cannot be performed, the marker can be... Set to 0. If =0 but If so, then execute the cache clearing and pure network score rollback described in Formula 13, instead of triggering the termination item.

[0079] After termination, a safety stop is executed, and the robotic arm returns to a predefined safe pose away from the user and the items on the table, and reminds the user via voice that the item was not picked up. This step ensures that the system does not get bogged down in endless trial and error; even if the object has been found but the registration confidence is insufficient, and therefore the system reverts to the pure network scoring mode according to Formula 13, the cause-specific strategy itself can still continue to provide recovery gains.

[0080] As an example of a complete process, in a scenario where an intelligent wheelchair robot picks up a cylindrical medicine bottle from the ground, which may slip, the robot grasps the bottle from above. During execution, the wrist camera is obstructed, and the system completes real-time detection solely based on three-channel body perception signals. When the gripping force and joint load suddenly drop simultaneously during the lifting process, the state determination step fails, and the cause classification step identifies it as slippage. After the robotic arm returns to a safe position, the visual enhancement analysis step uses scene differential positioning to locate the new position of the fallen bottle, and if necessary, uses object search to retrieve the rolled-out object. The medicine bottle is captured, and the failed grasp pose is transferred to the new frame through rigid registration. In the semantic enhancement and refeeding stage, a penalty is constructed based on the surface properties of the smooth cylindrical surface of the medicine bottle, so that each smooth area of ​​the medicine bottle is generalized and downweighted. After reordering, the candidate in the rough label area jumps to the first place. After visual verification before re-grabbing, the command is issued. In the closed-loop control stage, a special strategy of increasing the gripping force and reducing the acceleration is adopted to prevent slippage. The gripper closes in the rough label area and obtains a gripping force of about 1.7 times that of the smooth area. The second grasp is successful. If the re-grabbing is still unsuccessful after continuous attempts, it will stop safely when the upper limit of attempts is reached and a voice prompt will be given.

[0081] Through the aforementioned collaboration, in an evaluation of approximately 900 capture attempts and 310 failures in a mixed task, the final success rate of this method after recapture was approximately 81%, significantly higher than the approximately 34% success rate of blindly retrying in the original pose and the approximately 52% success rate of uniformly selecting the second-highest-scoring candidate without distinguishing the causes. (Refer to...) Figure 7 The figure shows a bar chart comparing the re-grasp success rate of the method of this invention with existing technologies (blind retries, causeless identification unified retries, and ablation: surface attribution removal). Breaking down the failure causes—missed grasp, incomplete grip, slippage, and collision—the re-grasp success rate of the method of this invention is approximately 76%, 82%, 79%, and 85%, respectively. The most significant improvement is in the re-grasp success rate for slippage, which increased from 22% in blind retries to approximately 79%. Figure 8 In the figure, column a represents blind retries, and column b represents the method of the present invention.

[0082] In the control method where surface attribute matching factors are removed and only spatial decay penalty is retained, the overall recapture success rate drops from about 81% to about 68%, indicating that surface attribute matching factors contribute an independent 13 percentage points. For the extreme case of about 8% where the registration confidence is insufficient due to objects rolling out of the field of view or falling into invisible areas, the system automatically clears the cache and falls back to the pure network scoring mode. After falling back, the recapture success rate is about 48%, which is still higher than blind retries.

[0083] like Figure 9 As shown, the present invention also provides a robot grasping failure cause identification and adaptive re-grasping system that implements the above method. The system is organized and deployed in the robot's onboard embedded computing unit in the form of a two-stage pipeline, including a signal acquisition unit, a state machine unit, a cause classification unit, a vision enhancement unit, a semantic feedback unit, and a closed-loop safety unit.

[0084] The signal acquisition unit is used to acquire three body sensing signals in real time during the gripping process: gripper opening, gripper motor current and robotic arm joint current, which are read by the gripper encoder. It also performs sliding window smoothing and baseline adaptation. Specifically, it subscribes to the data streams of robotic arm joint state and gripper state, executes at a frequency of about 50 Hz, and configures a smoothing window width of 5.

[0085] The state machine unit is used to extract five-dimensional discriminative features based on three-way body perception signals, including opening residual, holding period clamping force level, clamping force drop, joint load increment and joint load drop. Based on these features, the grasp is determined to be successful or unsuccessful according to the joint threshold criterion. Its configuration parameters include opening residual range, clamping force level threshold, joint load increment threshold, clamping force drop threshold and joint load drop threshold.

[0086] The cause classification unit is used to construct a five-dimensional normalized feature vector by combining the opening residual, clamping force level compensation, joint load increment, clamping force drop, and joint load drop when a failure is determined. The classifier classifies the failure into one of the following: missed grasp, incomplete clamping, slippage, and collision, and outputs the cause confidence. When the difference between the maximum and the second maximum confidence is less than the ambiguity threshold, the cause is determined to be uncertain.

[0087] The visual enhancement unit is used to acquire visual information of the current scene after a failure is determined, perform scene subtraction, object search view selection, rigid registration and cross-frame migration of the failed grasp pose, and clear the failure cache and roll back to pure network scoring when the registration confidence is insufficient. At the same time, it extracts the surface attributes of the object at the failed grasp pose. Its configured displacement threshold is 0.03 meters and the registration confidence threshold is 0.6.

[0088] The semantic refeeding unit is used to construct a semantically enhanced spatial penalty based on the historical failed capture poses migrated to the current data frame. This penalty includes a positional attenuation factor, an orientation attenuation factor, and a surface attribute matching factor. The weighted score is then fused with the capture score output by the network to obtain a comprehensive score. Additionally, three visual checks are performed before recapture: path accessibility, orientation matching, and width consistency.

[0089] The closed-loop safety unit is used to select the corresponding re-grasping strategy based on the failure cause category, re-rank the candidate grasping positions according to the comprehensive score to determine the re-grasping pose, control the robot to execute the re-grasping, and then hand it over to the state machine unit for success or failure determination again. It terminates the re-grasping when one of the three safety termination criteria is met, restores the robotic arm to a safe pose, and provides a voice prompt. Its configured maximum number of retries is 3, and the minimum executable score threshold is 0.35. Intermediate results are passed between the two stages via shared memory. Failure experience is cached in a hash table with the object identifier as the key to support cross-frame persistence. The system runs as a post-execution monitoring plugin for the existing grasping pipeline. The input interface subscribes to the robot joint state, gripper state, camera data stream, and the candidate list output by the grasping detection network. The output interface issues re-grasping commands containing target pose, gripper width, approach speed, and gripping force limit to the motion planning and execution module, or issues a safe pose command and triggers a voice broadcast upon termination, without requiring modification to the upstream grasping detection network.

[0090] In one specific embodiment, the execution time for ontology perception processing and state determination is less than 0.5 milliseconds per frame, and the entire visual enhancement calculation after failure takes approximately 73 milliseconds, far less than the 1.5 to 3 seconds typically required for the robotic arm to return to a safe pose and re-move. For the 5-dimensional input, 4-class output linear softmax classifier described in Equation 9, there are a total of 24 learnable parameters (excluding normalized statistics) for W and b, which is consistent with the parameter count of the actual model structure; the total system memory increment is less than 5 megabytes, demonstrating good airborne deployment economy.

[0091] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the aforementioned method; and a computer-readable storage medium storing a computer program thereon, wherein the computer program is executed by the processor to implement the steps of the aforementioned method.

[0092] It should be noted that the above are merely specific embodiments of the present invention. The sampling period, smoothing window width, various discrimination thresholds, displacement thresholds, registration confidence thresholds, attenuation kernel bandwidths, causal penalty amplitudes, maximum number of retry attempts, minimum executable score thresholds, and various experimental values ​​listed are all examples and do not constitute a limitation on the present invention. Equivalent substitutions, parameter adjustments, or combinations and omissions of some features made by those skilled in the art based on the technical solutions of the present invention, which do not depart from the spirit and scope of the technical solutions of the present invention, should all be covered within the protection scope of the present invention.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for identifying the causes of robot grasping failures and adaptive re-grasping, wherein the robot is equipped with a gripper and a camera, and receives candidate grasping poses, grasping scores, and expected object widths output by a grasping detection network, characterized in that... The methods include: Step S1. During the gripping process, collect three body sensing signals in real time: gripper opening, gripper motor current, and robotic arm joint current output by the gripper encoder. Step S2. Extract discriminative features based on the three-channel body perception signals, and determine whether the capture was successful or failed. Step S3. When a failure is determined, the failure is classified into one of the preset failure cause categories according to the discrimination features; Step S4. After a failure is determined, the visual information of the current scene is collected and compared with the reference scene before the grasping execution to relocate the failed object. A spatial transformation from the old data frame to the new data frame is established, and the failed grasping pose recorded in the old data frame is transferred to the new data frame according to the spatial transformation. Step S5. Extract the surface attributes of the object at the failed grasp pose when migrating to the new data frame, construct a spatial penalty including a position decay factor and a surface attribute matching factor, and use the spatial penalty to perform weighted fusion on the grasp score output by the grasp detection network on the new data frame to obtain a comprehensive score for each candidate grasp; the surface attribute matching factor generalizes the spatial penalty to regions on the object that have similar surface attributes to the failed grasp pose. Step S6. Select the corresponding re-grabbing strategy according to the failure cause category, and re-sort the candidate grabs according to the comprehensive score to determine the re-grabbing pose. After re-grabbing, return to step S2 to determine success or failure again, thus forming an adaptive re-grabbing closed loop.

2. The robot grasping failure cause identification and adaptive re-grasping method according to claim 1, characterized in that, In step S1, a sliding window smoothing is applied to the collected gripper motor current and robotic arm joint current respectively; and before each gripping operation, the average steady-state current of the gripper is collected as the adaptive baseline and the average steady-state current of the joint is collected as the no-load baseline in the no-load static state of the robotic arm. The joint current of the robotic arm is converted into the load torque of each joint through the torque constant, and a scalar joint load characterization for discrimination is obtained by selecting load-sensitive joints or aggregating them according to a preset method. The discriminative features described in step S2 are based on the smoothed signal and are obtained after normalization relative to the adaptive baseline and the empty baseline.

3. The robot grasping failure cause identification and adaptive re-grasping method according to claim 1, characterized in that, The discrimination features mentioned in step S2 include: the ratio of the steady-state opening after the gripper closes to the width of the expected object, i.e., the opening residual; the holding force level determined by the gripper motor current; the sudden drop in holding force during the lifting period determined by the gripper motor current; the joint load increment of the robotic arm; and the sudden drop in joint load during the lifting period; the joint load increment is used to normalize the sudden drop in joint load. The determination is as follows: when the opening residual falls within a preset range, the clamping force level is not lower than a first threshold, the joint load increment is not lower than a load increment threshold, the clamping force drop is less than a second threshold, and the joint load drop is less than a third threshold, the grasping is determined to be successful; otherwise, it is determined to be a failure.

4. The robot grasping failure cause identification and adaptive re-grasping method according to claim 3, characterized in that, The failure causes mentioned in step S3 include four categories: missing the target, incomplete clamping, slippage, and collision. Among them: missing the target corresponds to low opening residual and clamping force level and no increase in joint load; incomplete clamping corresponds to abnormal opening and low clamping force level and joint load; slippage corresponds to normal holding period but sudden drop in clamping force and sudden drop in joint load during lifting period, both exceeding the corresponding threshold; collision and displacement corresponds to the situation where the discrimination features do not show the abnormal patterns of missing the target, incomplete clamping and slippage, the success criterion is not established, and the object's original position is confirmed to have shifted after failure by visual difference. The failure classification is based on the feature vector constructed by the discriminative features and output by the classifier. The feature vector includes at least the opening residual, the compensation of the clamping force level, the joint load increment, the clamping force drop, and the joint load drop. When the difference between the maximum confidence and the second maximum confidence output by the classifier is less than the ambiguity threshold, the cause is determined to be uncertain and supplementary observation is triggered.

5. The robot grasping failure cause identification and adaptive re-grasping method according to claim 1, characterized in that, The differential comparison and relocalization in step S4 includes: first, transforming the point cloud of the current scene and the reference scene point cloud collected before the grasping execution to the same robot base coordinate system or world coordinate system, and then performing nearest neighbor differential; extracting the centroid displacement vector of the failed object in the region where the differential distance exceeds the displacement threshold, and determining whether the failed object has left the field of view; when it is determined that the failed object has left the field of view, selecting the view with the highest visibility of the area where the failed object is located from multiple candidate views that the robot arm can reach with the camera, controlling the camera to move to the view, re-collecting the point cloud, and detecting the failed object to find it.

6. The robot grasping failure cause identification and adaptive re-grasping method according to claim 1, characterized in that, Step S4, establishing the spatial transformation and migration of the failed grasping pose, includes: performing rigid registration on the relocated failed object to obtain the rigid body transformation from the old data frame to the new data frame and its registration confidence; when the registration confidence is not lower than the registration threshold, migrating the failed grasping pose to the new data frame according to the rigid body transformation; when the registration confidence is lower than the registration threshold, clearing the failed grasping pose and related cache of the failed object, and switching to determining the re-grabbing pose only based on the grasping score output by the grasping detection network on the new data frame.

7. The robot grasping failure cause identification and adaptive re-grasping method according to claim 1, characterized in that, The surface properties mentioned in step S5 include the average curvature, reflectivity, and texture roughness of the object surface in the contact neighborhood of the failed grasp pose migrated to the new data frame; the spatial penalty is the sum of penalty terms for the migrated historical failed grasp poses, and each penalty term is obtained by multiplying the position attenuation factor, orientation attenuation factor, and surface property matching factor, and then multiplying by the penalty magnitude set according to the failure cause category. The position decay factor decreases as the spatial distance between the candidate grab and the historical failed grab pose after migration increases; the direction decay factor decreases as the angle between their approach directions increases; and the surface attribute matching factor decreases as the difference between their surface attributes increases. Furthermore, the penalty magnitude corresponding to slip failure is greater than the penalty magnitude corresponding to miss failure.

8. The robot grasping failure cause identification and adaptive re-grasping method according to claim 1, characterized in that, In step S5, after determining the re-grasp pose but before executing the re-grasp, a visual verification is also included: confirming on the latest acquired point cloud that the approach path of the re-grasp pose is unobstructed, the current orientation of the object matches the re-grasp pose, and the gripper width is consistent with the currently observed object thickness. Only after all three items pass the verification can the re-grasp be executed.

9. The robot grasping failure cause identification and adaptive re-grasping method according to claim 1, characterized in that, The re-grab strategy selected in step S6 based on the failure cause category includes: for missed grabs, after triggering supplementary observation, selecting the candidate grab with the least spatial penalty weight reduction and the greatest difference from the original failed grab pose; for incomplete gripping, adjusting the gripper opening width and gripping depth and fine-tuning the pose along the normal of the object surface; for slippage, increasing the gripping force setting value and reducing the acceleration during the lifting phase; for collision and displacement, fine-tuning the pose along the opposite direction of the object displacement and reducing the end-effector approach velocity. Furthermore, when the number of re-grabbing attempts reaches the upper limit, or the maximum value of the comprehensive score of each candidate grab is lower than the executable threshold, or the failed object is still not found after supplementary search and reliable registration cannot be established, the re-grabbing is terminated, the robotic arm is restored to a safe pose, and a voice prompt is given; when the failed object has been found but the registration confidence is lower than the registration threshold, the historical failure cache is cleared and the re-grabbing pose is determined only based on the grab score of the new data frame, instead of directly terminating the re-grabbing.

10. A robot grasping failure cause identification and adaptive re-grasping system, wherein the robot is equipped with a gripper and a camera, and receives candidate grasping positions, grasping scores, and expected object widths output by a grasping detection network, characterized in that, include: The signal acquisition unit is used to acquire three body sensing signals in real time during the grasping process: the opening degree of the gripper encoder, the gripper motor current, and the robotic arm joint current. The state machine unit is used to extract discriminative features based on the three body perception signals and determine whether the capture is successful or unsuccessful. The cause classification unit is used to classify the failure into one of a number of preset failure cause categories based on the discrimination features when the failure is determined. The visual enhancement unit is used to collect visual information of the current scene after a failure is determined, perform differential comparison with the reference scene before the grasping execution to relocate the failed object, establish a spatial transformation from the old data frame to the new data frame, and transfer the failed grasping pose recorded in the old data frame to the new data frame according to the spatial transformation. The semantic backfeeding unit is used to extract the surface attributes of the object at the failed grasp pose when it is migrated to the new data frame, construct a spatial penalty including a position decay factor and a surface attribute matching factor, and use the spatial penalty to subtract and fuse the grasp score output by the grasp detection network on the new data frame to obtain a comprehensive score for each candidate grasp. The surface attribute matching factor enables the spatial penalty to be generalized to the region on the object that has similar surface attributes to the failed grasp pose. The closed-loop safety unit is used to select the corresponding re-grab strategy according to the failure cause category, and reorder the candidate grabs according to the comprehensive score to determine the re-grab pose. After the robot performs the re-grab, the state machine unit is handed over to determine the success or failure again, thus forming an adaptive re-grab closed loop.

Images