Self-recovery method and system for picking robot after claw slipping

Abstract

The application relates to the automatic grabbing technical field of picking robots, and proposes a self-regrasping method and system for a picking robot after the slipping of a gripper, wherein the four sensing data of the joint angle, pressure distribution, vibration acceleration and external force and torque of the gripper are comprehensively analyzed, subtle changes generated when slipping occurs are captured from multiple dimensions, and the signal of slipping is accurately separated from complex interference information; the accuracy of slipping detection is greatly improved, whether slipping occurs and the degree of slipping are accurately judged, and the problem that a single sensor is prone to misjudgment is avoided; meanwhile, according to the recognized slipping type and severity, a hierarchical response strategy is adopted, slight slipping is remedied by slightly adjusting the clamping force and posture, and serious slipping directly gives up regrasping; and the effect of each regrasping is monitored and evaluated, so that the success rate of picking operation and the intact rate of fruits are improved.

Description

Technical Field

[0001] This application relates to the field of automatic grasping technology for harvesting robots, and in particular to a method and system for autonomous re-grasping after the gripper of a harvesting robot slips off. Background Technology

[0002] With the rapid development of agricultural automation, harvesting robots are increasingly being used in orchards. Harvesting robots typically pick fruit by gripping it with claws and pulling or rotating it. However, due to the irregular shape and varying smoothness of the fruit's surface, coupled with the fragility of the peel, the claws are prone to slipping during the grasping process—the fruit slips out of the gripper before being picked. Therefore, improving the accuracy of claw slippage detection in harvesting robots and the adaptability and effectiveness of re-grip strategies have become problems to be solved. Summary of the Invention

[0003] This application provides a method and system for autonomous re-grasping of a picking robot after the gripper slips out, the main purpose of which is to solve the problem of low automatic grasping efficiency of picking robots.

[0004] To achieve the above objectives, this application provides a method for autonomous re-grasping of a picking robot after the gripper slips off, comprising: S1, acquiring multimodal sensing data of the picking robot during the grasping process.

[0005] S2. Based on multimodal sensing data, construct a slip deviation feature map.

[0006] S3. Based on the slip deviation feature map, analyze the slip state category and slip severity score of the current grasping action.

[0007] S4. Determine the re-grabbing strategy based on the slip state category and slip severity score.

[0008] S5. Execute the re-capture strategy and reacquire multimodal sensing data for feedback monitoring after execution.

[0009] In a preferred embodiment, the multimodal sensing data includes gripper joint angle data, pressure distribution data, vibration acceleration data, and external force and torque data.

[0010] In a preferred embodiment, the step of constructing a slip deviation feature map based on multimodal sensing data includes: constructing an angle deviation time series, a pressure distribution deviation feature series, a vibration frequency domain deviation feature series, and an external force and torque deviation feature series, and then aligning and splicing them in the time dimension to generate a slip deviation feature map.

[0011] In a preferred embodiment, the external force and torque deviation characteristic sequence includes: setting the first... The normal grasping force component at each sampling time is The torque component in the rotational direction is The sampling time interval is The rate of change of the normal force component and the rate of change of torque component in the direction of rotation The calculation formulas are as follows: ; ; The characteristic sequence of external force and torque deviation consists of the rate of change of the normal force component and the rate of change of the torque component in the rotation direction at each moment.

[0012] In a preferred embodiment, the step of aligning and stitching along the time dimension to generate a slippage deviation feature map includes: setting the angle deviation time series as... The pressure distribution deviation characteristic sequence is The vibration frequency domain deviation characteristic sequence is The characteristic sequence of external force and torque deviation is as follows Then the slip deviation characteristic map The formula is: ; in 4 represents the total number of sampling times and the number of feature dimensions.

[0013] In a preferred embodiment, the step of analyzing the slip deviation feature map to identify the slip state category and slip severity score of the current grasping action includes: S601, flattening the slip deviation feature map into a one-dimensional feature vector and inputting it into a pre-trained slip state recognition model to obtain the slip state category of the current grasping action.

[0014] S602, calculate the symbolic distance from the one-dimensional feature vector to the classification decision boundary of the slip state recognition model, and map the symbolic distance to the slip severity score.

[0015] In a preferred embodiment, the step of calculating the symbolic distance from the one-dimensional feature vector to the classification decision boundary of the slippage state recognition model, and mapping the symbolic distance to a slippage severity score, includes: the calculation formula for the multi-class decision function is as follows: ; in The predicted slip state category is 0, which represents no slip, 1 represents slight slip, and 2 represents complete slip. The number of support vectors; For the first The multipliers of the support vectors; For the first The class labels of the support vectors; It is a radial basis kernel function; For the first Support vectors; For bias terms; The input is a one-dimensional feature vector.

[0016] In a preferred embodiment, determining the re-grabbing strategy based on the slippage state category and slippage severity score includes: S801, identifying the slippage state category, and when the slippage state category is no slippage, maintaining the current gripping state and executing the subsequent picking and handling process.

[0017] S802, when the slippage state category is micro-slippage, dynamic compensation control of the gripper force is initiated based on the slippage severity score, gradually increasing the gripper force to the preset upper limit value, and at the same time, the end posture of the robotic arm is finely adjusted according to the direction of the angle deviation to obtain the first-level heavy gripping strategy.

[0018] S803, when the slippage state category is complete slippage, generates a secondary re-grasping strategy that abandons the current gripping action and re-identifies the fruit position and posture based on the visual feedback system, and plans a new gripping point and gripping path.

[0019] In a preferred embodiment, the step of executing the re-capture strategy and re-acquiring multimodal sensing data for feedback monitoring after execution includes: S901, performing a corresponding recovery operation according to the re-capture strategy.

[0020] S902, after execution, acquires the re-captured multimodal sensing data, and re-identifies the slip state category and the slip severity score after re-capture.

[0021] S903: Calculate the change between the slip severity score before re-grabbing and the slip severity score after re-grabbing. If the change is greater than or equal to a preset effective threshold, the re-grabbing is deemed invalid and an alarm is triggered. If the change is less than the preset effective threshold, the re-grabbing is deemed successful and the subsequent picking task continues.

[0022] To address the aforementioned issues, this application also provides a system for an autonomous re-grasping method after the gripper of a harvesting robot slips out. The system includes a multimodal sensor data acquisition module for acquiring multimodal sensor data of the harvesting robot during the grasping process.

[0023] The slip deviation feature map construction module constructs slip deviation feature maps based on multimodal sensing data.

[0024] The slippage deviation feature map analysis module analyzes the slippage deviation feature map to identify the slippage state category and the severity score of the current grasping action.

[0025] The re-grabbing strategy determination module determines the re-grabbing strategy based on the slip state category and slip severity score.

[0026] The feedback monitoring module is used to execute the re-capture strategy and, after execution, reacquire multimodal sensing data for feedback monitoring.

[0027] Compared with the prior art, this application has the following beneficial effects: 1. This application provides a method and system for autonomous re-grasping of a harvesting robot gripper after slippage. By comprehensively analyzing four types of sensor data—grip joint angle, pressure distribution, vibration acceleration, and external force and torque—it captures subtle changes that occur when slippage occurs from multiple dimensions. This is equivalent to equipping the gripper with a "multi-sensory system," which can accurately isolate the slippage signal from complex interference information. As a result, the accuracy of slippage detection is greatly improved, accurately determining whether slippage has occurred and to what extent, avoiding the problem of misjudgment that is easily caused by a single sensor. At the same time, a graded response strategy is adopted according to the identified slippage type and severity. For minor slippage, the gripping force and posture are finely adjusted to remedy the situation, while for severe slippage, the gripping is abandoned and re-grasped. Moreover, the entire process is a closed-loop feedback, and the effect of each re-grasping is monitored and evaluated, which can improve the success rate of harvesting operations and the fruit integrity rate.

[0028] 2. This application sets up a system that records the entire process after each slippage, regardless of whether the re-grabbing is successful or not. This includes the raw data collected by the sensor, the analyzed deviation characteristics, the decisions made, and the final results, all of which are stored and fed back to the model. This system fully stores information on which fruits are prone to slipping and what force is just right, and its self-iterative capability makes the entire system more and more convenient to use in practice, without the need for manual parameter tuning and optimization. Attached Figure Description

[0029] Figure 1 A flowchart illustrating a method for autonomous re-grabbing of a picking robot after the gripper slips out, provided in an embodiment of this application; Figure 2 A functional block diagram of a system for an autonomous re-grabbing method of a picking robot after the gripper slips out, provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the process of constructing the slip deviation feature map in this application. Figure 4 This is a schematic diagram showing the layout of the thin-film pressure sensor array on the inner surface of the gripper in this application; Figure 5 This is a schematic diagram illustrating the harvesting robot simulating the harvesting of tree fruits in this application; Figure 6 This is a schematic diagram of the harvesting robot in this application; Figure 7 This is a schematic diagram illustrating the picking robot simulating the picking of fallen fruit in this application; The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0030] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0031] This application provides a method for autonomously re-grasping a harvesting robot's gripper after it slips out. The executing entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method for autonomously re-grasping a harvesting robot's gripper after it slips out can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0032] Reference Figure 1 The diagram shown is a flowchart illustrating an autonomous re-grasping method for a harvesting robot after its gripper slips out, according to an embodiment of this application. In this embodiment, the autonomous re-grasping method for a harvesting robot after its gripper slips out includes: S1, acquiring multimodal sensing data of the harvesting robot during the gripping process.

[0033] In this embodiment of the application, the multimodal sensing data includes gripper joint angle data, pressure distribution data, vibration acceleration data, and external force and torque data.

[0034] It should be noted that the real-time joint angle data of the gripper during the grasping process is obtained through the angle sensor at the gripper joint; the real-time pressure distribution data of each pressure acquisition point is obtained through the thin-film pressure sensor array on the inner surface of the gripper; the real-time vibration acceleration data of the gripper during the grasping process is obtained through the triaxial accelerometer installed on the gripper base; and the external force and torque data of the gripper during the grasping process is obtained through the six-dimensional force / torque sensor installed at the end of the robotic arm. The real-time joint angle data, real-time pressure distribution data, real-time vibration acceleration data, and external force and torque data are time-stamped and aligned to obtain synchronized multimodal sensing data.

[0035] It should be noted that the thin-film pressure sensor array consists of multiple pressure sensing units arranged in a grid pattern. Each pressure acquisition point covers the area on the inner surface of the gripper that contacts the fruit, used to collect pressure values ​​at each contact point in real time. A triaxial accelerometer is used to collect the vibration acceleration components of the gripper in the X, Y, and Z axes, with a sampling frequency set to no less than 200Hz to capture the high-frequency vibration signal generated at the moment of slippage. A six-dimensional force / torque sensor is used to collect the force components and torque components in three directions experienced by the gripper during the grasping process.

[0036] Furthermore, timestamp alignment is used to eliminate data asynchrony issues caused by differences in sampling clocks and startup times between different sensors. Timestamp alignment refers to using the time axis of the sensor with the highest sampling frequency as the reference time axis, and aligning the data collected by other sensors using nearest neighbor interpolation, so that each sensor has a corresponding data value at every reference time point.

[0037] It should be noted that the nearest neighbor interpolation method refers to: setting a certain moment on the reference time axis as the moment to be aligned, finding the sampling moment with the smallest time difference from the moment to be aligned on the original time axis of the sensor to be synchronized, and using the data value corresponding to the sampling moment as the data value of the moment to be aligned. The calculation formula for timestamp alignment is as follows: ; ; in The first on the reference time axis At that moment, The first time axis of the sensor to be synchronized Each sampling time, For the sensor to be synchronized in The raw data values ​​collected at any time To align Data value at any given time.

[0038] S2. Based on multimodal sensing data, construct a slip deviation feature map.

[0039] In this embodiment of the application, the step of constructing a slip deviation feature map based on multimodal sensing data includes: constructing an angle deviation time series, a pressure distribution deviation feature series, a vibration frequency domain deviation feature series, and an external force and torque deviation feature series, and then aligning and splicing them in the time dimension to generate a slip deviation feature map.

[0040] It should be noted that: A1 calculates the angle deviation value at each moment based on real-time joint angle data and the preset expected joint angle trajectory, and constructs an angle deviation time series; A2 extracts the average pressure and pressure center offset at each pressure acquisition point based on real-time pressure distribution data, and constructs a pressure distribution deviation feature series; A3 extracts vibration frequency domain features by performing a fast Fourier transform on real-time vibration acceleration data, calculates the change in the distribution ratio of vibration energy in the low-frequency and high-frequency bands, and constructs a vibration frequency domain deviation feature series; A4 extracts the component change rate of the gripping force in the gripper normal direction and the component change rate of the torque in the gripper rotation direction based on external force and torque data, and constructs an external force and torque deviation feature series.

[0041] It should be noted that the preset expected joint angle trajectory refers to a preset standard curve showing the change of the gripper joint angle over time under ideal grasping conditions. Time-by-time comparison refers to calculating the difference between the real-time joint angle value at each sampling moment and the corresponding expected joint angle value. The angle deviation values ​​at each moment are arranged in chronological order to form an angle deviation time series.

[0042] Furthermore, the angle deviation value is calculated as follows: Set the first... The real-time joint angle at each sampling time is The preset desired joint angle is Then the angle deviation value The calculation formula is: ; in This represents the total number of sampling times; the angle deviation values ​​at each time point constitute the angle deviation time series. .

[0043] It should be noted that the average pressure value refers to the arithmetic mean of the pressure values ​​of all pressure acquisition points in the thin-film pressure sensor array at the same sampling time; the pressure center offset refers to the offset distance and direction of the pressure distribution center of each pressure acquisition point relative to the geometric center of the inner surface of the gripper at the current time.

[0044] Furthermore, the average pressure is calculated as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] There are a total of sampling times. There are 1 pressure sampling point, and the pressure value at each sampling point is... Then the average pressure The calculation formula is: ; Setting the first There are a total of sampling times. The pressure sampling point, the first The coordinates of the pressure acquisition points are: The weighted coordinates of the pressure center The calculation formula is: ; Set the geometric center coordinates of the inner surface of the gripper as The formula for calculating the offset of the pressure center is: ; in For the first The pressure center offset at each sampling time represents the Euclidean distance between the pressure distribution center and the geometric center of the gripper's inner surface at that current moment. The pressure distribution deviation characteristic sequence is composed of the pressure mean and pressure center offset at each time moment.

[0045] It should be noted that the vibration acceleration signal in the time domain is converted into a spectrum signal in the frequency domain; the low-frequency band refers to the frequency range of 0 to 50 Hz, and the high-frequency band refers to the frequency range of 50 Hz to 200 Hz. The change in the distribution ratio refers to the change in the ratio of the high-frequency vibration energy to the low-frequency vibration energy at the current moment relative to the preset standard grasping state.

[0046] Furthermore, the vibration frequency domain deviation characteristic sequence is constructed as follows: The first... The vibration acceleration signal at each sampling time point yields the low-frequency energy. High-frequency energy is Then the current distribution ratio for: ; The standard distribution ratio under the preset standard crawling state is: The change in the distribution ratio for: ; The changes in the distribution ratio at each time point constitute the vibration frequency domain deviation characteristic sequence.

[0047] In this embodiment of the application, the external force and torque deviation characteristic sequence includes: setting the first... The normal grasping force component at each sampling time is The torque component in the rotational direction is The sampling time interval is The rate of change of the normal force component and the rate of change of torque component in the direction of rotation The calculation formulas are as follows: ; ; The characteristic sequence of external force and torque deviation consists of the rate of change of the normal force component and the rate of change of the torque component in the rotation direction at each moment.

[0048] It should be noted that the rate of change of the gripping force component in the normal direction of the gripper refers to the rate of change of the normal gripping force at the current moment relative to the normal gripping force at the previous moment. The rate of change of the torque component in the rotational direction of the gripper refers to the rate of change of the rotational torque at the current moment relative to the rotational torque at the previous moment.

[0049] In this embodiment of the application, the step of aligning and stitching along the time dimension to generate a slippage deviation feature map includes: setting the angle deviation time series as... The pressure distribution deviation characteristic sequence is The vibration frequency domain deviation characteristic sequence is The characteristic sequence of external force and torque deviation is as follows Then the slip deviation characteristic map The formula is: ; in 4 represents the total number of sampling times and the number of feature dimensions.

[0050] It should be noted that the angle deviation time series, pressure distribution deviation feature series, vibration frequency domain deviation feature series, and external force and torque deviation feature series are horizontally combined according to the same sampling time order to generate a two-dimensional matrix form of slip deviation feature map.

[0051] S3. Based on the slip deviation feature map, analyze the slip state category and slip severity score of the current grasping action.

[0052] In this embodiment of the application, the step of analyzing the slip deviation feature map to identify the slip state category and slip severity score of the current grasping action includes: S601, flattening the slip deviation feature map into a one-dimensional feature vector and inputting it into a pre-trained slip state recognition model to obtain the slip state category of the current grasping action.

[0053] S602, calculate the symbolic distance from the one-dimensional feature vector to the classification decision boundary of the slip state recognition model, and map the symbolic distance to the slip severity score.

[0054] It should be noted that the slippage state recognition model is a classification model pre-trained based on the support vector machine algorithm. The training process of this model is as follows: a large number of sample data of normal grasping and slippage grasping under different types of fruits, different gripping postures, and different grasping forces are collected; slippage deviation feature maps of each sample data are extracted; each slippage deviation feature map is flattened into a one-dimensional feature vector as a training sample; each training sample is manually labeled with three categories: no slippage, slight slippage, and complete slippage; the labeled training samples are used to train the support vector machine classifier to obtain the slippage state recognition model.

[0055] Furthermore, flattening refers to concatenating the two-dimensional matrix form of the slip deviation feature map row by row in row-major order, converting it into a one-dimensional vector form. The slip deviation feature map is then defined. ,in Let 4 be the total number of sampling times and 4 be the number of feature dimensions. Then, the flattened one-dimensional feature vector... .

[0056] It should be noted that the slip state categories include three types: no slip, slight slip, and complete slip. The classification decision process of the slip state recognition model is as follows: a one-dimensional feature vector is input into a trained support vector machine classifier, and the confidence score of the feature vector belonging to each category is calculated through a multi-class decision function. The category with the highest confidence score is taken as the slip state category of the current grasping action.

[0057] In this embodiment of the application, the step of calculating the symbolic distance from the one-dimensional feature vector to the classification decision boundary of the slippage state recognition model, and mapping the symbolic distance to a slippage severity score, includes: the calculation formula for the multi-class decision function is as follows: ; in The predicted slip state category is 0, which represents no slip, 1 represents slight slip, and 2 represents complete slip. The number of support vectors; For the first The multipliers of the support vectors; For the first The class labels of the support vectors; It is a radial basis kernel function; For the first Support vectors; For bias terms; The input is a one-dimensional feature vector.

[0058] It should be noted that the category with the highest confidence score is selected as the identification result for the slip state category; the slip severity score is used to quantify the severity of the slip phenomenon in the current grasping action. This score is calculated by measuring the signed distance from the one-dimensional feature vector to the classification decision boundary and mapping this distance to a value between 0 and 1.

[0059] Furthermore, the formula for calculating the severity score of slippage is as follows: ; in The severity of slippage is scored, with a value ranging from 0 to 1. The higher the value, the more severe the slippage. One-dimensional feature vector The signed distance to the classification decision boundary; It is a natural constant.

[0060] Furthermore, symbolic distance The calculation method is as follows: the classification decision function value output by the slip state recognition model is subtracted from the boundary value of each classification decision boundary, and the difference with the smallest absolute value is taken as the symbol distance; the sign of this distance is used to determine which side of the decision boundary the feature vector is located on. The larger the absolute value of the distance, the farther the feature vector is from the decision boundary, and the higher the classification confidence.

[0061] S4. Determine the re-grabbing strategy based on the slip state category and slip severity score.

[0062] In this embodiment of the application, the step of determining the re-grabbing strategy based on the slip state category and the slip severity score includes: S801, identifying the slip state category, and when the slip state category is no slip, maintaining the current gripping state and executing the subsequent picking and handling process.

[0063] S802, when the slippage state category is micro-slippage, dynamic compensation control of the gripper force is initiated based on the slippage severity score, gradually increasing the gripper force to the preset upper limit value, and at the same time, the end posture of the robotic arm is finely adjusted according to the direction of the angle deviation to obtain the first-level heavy gripping strategy.

[0064] S803, when the slippage state category is complete slippage, generates a secondary re-grasping strategy that abandons the current gripping action and re-identifies the fruit position and posture based on the visual feedback system, and plans a new gripping point and gripping path.

[0065] It should be noted that the adaptive decision rule is a graded response rule based on the slip state category and slip severity score. It is used to select differentiated re-grab recovery paths according to different types and degrees of slip, avoiding ineffective operations or fruit damage caused by a single fixed strategy.

[0066] Furthermore, when the slippage status category is non-slippage, it is determined that the current grasping action is in a normal state, the gripper has sufficient clamping force and the fruit is stably held, and there is no need to perform any re-grabbing operation. The current grasping state is maintained and the subsequent picking and handling process continues.

[0067] It should be noted that the first-level re-grip strategy is used to deal with the slight slip state, which means that the fruit has a slight tendency to slip but has not yet completely slipped out of the gripper; the core of the first-level re-grip strategy is to restore a stable grip by fine-tuning compensation, rather than abandoning the current gripping action.

[0068] Furthermore, the specific process of dynamic compensation control is as follows: Starting with a preset base clamping force, the clamping force increment is calculated based on the severity score of slippage. The base clamping force and the clamping force increment are added together to obtain the target clamping force. The grippers are then controlled to gradually increase the clamping force to the target clamping force. The clamping force increment is positively correlated with the slippage severity score; that is, the higher the slippage severity score, the greater the clamping force increment. The increment step of the clamping force is set to not exceed a preset maximum step size to prevent sudden changes in clamping force that could damage the fruit.

[0069] Furthermore, the formula for calculating the clamping force increment is as follows: ; in This represents an increase in clamping force. The severity of slippage is scored, with a value ranging from 0 to 1; An upper limit value is preset for the clamping force, which is preset according to the fruit type and ripeness; This is the preset basic clamping force.

[0070] Furthermore, fine-tuning the robotic arm's end effector posture based on the direction of the angle deviation refers to: extracting the current angle deviation value from the angle deviation time series and determining its positive or negative direction; if the angle deviation is positive, it indicates that the actual opening and closing angle of the gripper is greater than the desired angle, and the robotic arm's end effector is controlled to fine-tune in the direction of decreasing the angle; if the angle deviation is negative, it indicates that the actual opening and closing angle of the gripper is less than the desired angle, and the robotic arm's end effector is controlled to fine-tune in the direction of increasing the angle. The fine-tuning amplitude is positively correlated with the absolute value of the angle deviation.

[0071] It should be noted that the secondary re-grip strategy is used to deal with the complete slippage state, which indicates that the fruit has completely escaped from the grippers, the current gripping action is invalid, and a complete re-grip plan is required. The core idea of ​​the secondary re-grip strategy is to abandon the current invalid action and re-plan the gripping scheme.

[0072] Furthermore, the specific execution process of the secondary gripping strategy is as follows: First, control the gripper to retract to a safe position to avoid collisions with other fruits or branches during movement; activate the visual feedback system to reacquire the image data of the current fruit; reposition the fruit's position and orientation using an image recognition algorithm to obtain the fruit's coordinates and orientation information in three-dimensional space; based on the repositioned fruit position and orientation, plan a new gripping point and gripping path; finally, control the robotic arm to move to the target gripping point according to the new gripping path and perform the secondary gripping action.

[0073] Furthermore, the preset upper limit value refers to the maximum allowable clamping force of the grippers. This value is preset based on the fruit type, maturity, and skin hardness. It is used to constrain the range of increase in clamping force during dynamic compensation control, preventing excessive clamping force from causing fruit damage or breakage. Examples of preset upper limits are shown in the table below: ; Furthermore, the basic gripping force refers to the initial gripping force applied by the grippers of the harvesting robot to the fruit under normal grasping conditions. This value is preset based on the average weight of the fruit and the surface friction coefficient to ensure that the fruit is stably gripped without causing damage. Examples of basic gripping forces are shown in the table below: ; S5. Execute the re-capture strategy and reacquire multimodal sensing data for feedback monitoring after execution.

[0074] In this embodiment of the application, the step of executing the re-capture strategy and re-acquiring multimodal sensing data for feedback monitoring after execution includes: S901, performing a corresponding recovery operation according to the re-capture strategy.

[0075] S902, after execution, acquires the re-captured multimodal sensing data, and re-identifies the slip state category and the slip severity score after re-capture.

[0076] S903: Calculate the change between the slip severity score before re-grabbing and the slip severity score after re-grabbing. If the change is greater than or equal to a preset effective threshold, the re-grabbing is deemed invalid and an alarm is triggered. If the change is less than the preset effective threshold, the re-grabbing is deemed successful and the subsequent picking task continues.

[0077] It should be noted that the recovery operation is a specific action within either the primary or secondary re-grip strategy. When the re-grip strategy is primary, the recovery operation includes initiating dynamic compensation control of the gripper's holding force and fine-tuning the robotic arm's end effector posture. When the re-grip strategy is secondary, the recovery operation includes retrieving the gripper to a safe position, re-identifying the fruit's position and posture, planning a new gripping point and path, and performing a secondary gripping.

[0078] Furthermore, the specific process of performing the corresponding recovery operation according to the re-grasping strategy is as follows: When the first-level re-grasping strategy is selected, the gripping force of the gripper is gradually increased to the target gripping force with a preset step increment. At the same time, the attitude of the end of the robotic arm is finely adjusted according to the direction of the angle deviation. During the fine-tuning process, the change of the slippage severity score is continuously monitored. When the second-level re-grasping strategy is selected, the gripper is first fully opened and retracted to a safe position. Then, the visual feedback system is activated to re-acquire the fruit image. The three-dimensional spatial coordinates and orientation information of the fruit are obtained through the image recognition algorithm. Based on the repositioned fruit state, a new gripping point coordinate and robotic arm movement path are planned. Finally, the robotic arm is controlled to move along the planned path to the new gripping point to perform a second gripping.

[0079] It should be noted that re-execution refers to restarting the entire slip detection process after the recovery operation is completed: multimodal sensing data is re-acquired from each sensor after the re-grasp; a new slip deviation feature map is reconstructed based on the new sensing data; this new slip deviation feature map is input into the slip state recognition model to obtain the slip state category and the slip severity score after the re-grasp. This process incorporates the results of the re-grasp operation into a closed-loop feedback system to verify whether the re-grasp operation effectively restored the normal grasping state.

[0080] It should be noted that the change refers to the difference in the slip severity score before and after re-grabbing, which is used to quantify the degree to which the re-grabbing operation improves the slip condition.

[0081] Furthermore, the formula for calculating the change in slip severity score before and after re-grasp is as follows: ; in The change in the severity score of slippage before and after re-grasp; The severity of the slippage before re-grabbing was assessed; The severity of slippage after re-grabbing is scored.

[0082] It should be noted that the preset effective threshold is a preset negative constant used to determine whether the re-grab operation effectively reduced the severity of slippage. If the change is less than this threshold, meaning the slippage severity score after re-grab is significantly lower than the slippage severity score before re-grab, it indicates that the slippage state has been effectively improved, and the re-grab is considered successful. If the change is greater than or equal to this threshold, meaning the slippage severity score after re-grab is not significantly reduced, it indicates that the re-grab operation has not effectively improved the slippage state, and the re-grab is considered invalid.

[0083] Furthermore, the preset effective threshold is set to a range of -0.3 to -0.1. When the change is less than -0.2, the slip severity score is considered to have decreased by at least 0.2, indicating a significant re-grab effect, and is thus considered successful. When the change is greater than or equal to -0.2, the slip severity score is considered not to have been sufficiently improved, and the re-grab is deemed invalid.

[0084] It should be noted that triggering an alarm means that when multiple consecutive re-grabbing attempts are deemed invalid, the system outputs an alarm signal, prompting the operator that the current fruit or environmental state exceeds the range of autonomous re-grabbing capabilities, and that it is necessary to switch to manual intervention mode or skip the current target fruit.

[0085] Furthermore, the entire process data from this capture, including the original multimodal sensing data, slip deviation feature map, slip state recognition results, recapture strategy selection results, recapture execution results, and validity judgment results, will be added as training samples to the training dataset of the slip state recognition model for iterative updates and optimization of the model.

[0086] It should be noted that, based on the content of this application, the following experiment was designed: 1. Experimental conditions: Test subjects: apples (semi-ripe), peaches (fully ripe), and strawberries (fully ripe); each fruit was subjected to 200 grasping actions, including 100 normal grasping actions and 100 slip grasping actions; slip grasping actions included artificially induced micro-slipping and complete slipping. 2. Comparison schemes: Scheme 1 (this application): a four-dimensional fusion scheme based on angle + pressure + vibration + force and torque; Scheme 2 (single angle): using only joint angle data, judging slipping through angle deviation thresholds; Scheme 3 (single pressure): using only pressure distribution data, judging slipping through changes in the average pressure; Scheme 4 (single vibration): using only vibration acceleration data, judging slipping through high-frequency energy mutations. 3. Evaluation indicators: Slip detection accuracy: the proportion of times the slip state (including no slip, slight slip, and complete slip) is correctly identified out of the total number of tests; False alarm rate: the proportion of times a normal grasp is misjudged as a slip out of the total number of normal grasps; Missed alarm rate: the proportion of times a slip grasp is misjudged as a normal grasp out of the total number of slip grasps.

[0087] The experimental results are shown in the table below: ; like Figure 2 The diagram shown is a functional block diagram of a system for an autonomous re-grabbing method of a picking robot after the gripper slips out, according to an embodiment of this application.

[0088] The system 100 of the autonomous re-grasping method for a harvesting robot gripper after slippage, as described in this application, can be installed in an electronic device. Depending on the functions implemented, the system 100 may include a multimodal sensor data acquisition module 101, a slippage deviation feature map construction module 102, a slippage deviation feature map analysis module 103, a re-grasping strategy determination module 104, and a feedback monitoring module 105. The multimodal sensor data acquisition module is responsible for collecting raw data from various sensors, such as joint angles, pressure distribution, vibration signals, and force conditions during the gripper's grasping process, and then transmitting this data to the slippage deviation feature map construction module. This module compares these four types of data with a standard grasping trajectory, calculates the difference, and combines the deviation features of the four dimensions. The slippage deviation feature map analysis module then analyzes this data to draw two conclusions: first, whether slippage has occurred, and if so, whether it is a slight or complete slippage; second, the severity of the slippage, assigning a score from 0 to 1. The re-grabbing strategy determination module makes decisions based on this result: if there is no slippage, it continues normal detachment; if there is slight slippage, it adjusts the gripping force and posture; if there is complete slippage, it abandons the current action and re-grasps. Finally, the feedback monitoring module executes the operation, and after execution, it triggers the data acquisition module to collect and analyze data again to check the operation effect. If the effect is unsatisfactory, it readjusts and tries again; if the effect is satisfactory, it continues.

[0089] The module described in this application can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0090] In this embodiment, the functions of each module / unit are as follows: The multimodal sensor data acquisition module is used to acquire multimodal sensor data of the picking robot during the grasping process.

[0091] The slip deviation feature map construction module constructs slip deviation feature maps based on multimodal sensing data.

[0092] The slippage deviation feature map analysis module analyzes the slippage deviation feature map to identify the slippage state category and the severity score of the current grasping action.

[0093] The re-grabbing strategy determination module determines the re-grabbing strategy based on the slip state category and slip severity score.

[0094] The feedback monitoring module is used to execute the re-capture strategy and, after execution, reacquire multimodal sensing data for feedback monitoring.

[0095] By comprehensively analyzing four types of sensor data—grip joint angle, pressure distribution, vibration acceleration, and external force and torque—the system captures subtle changes that occur when slippage occurs from multiple dimensions, accurately separating slippage signals from complex interference information. This significantly improves the accuracy of slippage detection, accurately determining whether slippage has occurred and its severity, avoiding the problem of misjudgment that can easily occur with a single sensor. Furthermore, based on the identified slippage type and severity, a graded response strategy is adopted: minor slippage is remedied by fine-tuning the gripping force and posture, while severe slippage is abandoned and re-grabbed. The effect of each re-grab is monitored and evaluated, improving the success rate of harvesting operations and the rate of fruit integrity.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0097] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0099] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application.

[0100] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method for autonomously re-grabbing a harvesting robot after its gripper has slipped, characterized in that, The method includes: S1. Acquire multimodal sensor data of the picking robot during the grasping process; S2. Construct a slip deviation feature map based on multimodal sensing data; S3. Based on the slip deviation feature map, analyze the slip state category and slip severity score of the current grasping action; S4. Determine the re-grabbing strategy based on the slip state category and slip severity score; S5. Execute the re-capture strategy and reacquire multimodal sensing data for feedback monitoring after execution.

2. The autonomous re-grasping method for a harvesting robot after the gripper slips out, as described in claim 1, is characterized in that... The multimodal sensing data includes gripper joint angle data, pressure distribution data, vibration acceleration data, and external force and torque data.

3. The autonomous re-grasping method for a harvesting robot after the gripper slips out, as described in claim 1, is characterized in that... The construction of a slip deviation feature map based on multimodal sensing data includes: The time series of angle deviation, the characteristic series of pressure distribution deviation, the characteristic series of vibration frequency domain deviation, and the characteristic series of external force and torque deviation are constructed, and then aligned and spliced ​​in the time dimension to generate a slip deviation characteristic map.

4. The autonomous re-grasping method for a harvesting robot after the gripper slips out, as described in claim 3, is characterized in that... The characteristic sequence of external force and torque deviation includes: Setting the first The normal grasping force component at each sampling time is The torque component in the rotational direction is The sampling time interval is The rate of change of the normal force component and the rate of change of torque component in the direction of rotation The calculation formulas are as follows: ； ； The characteristic sequence of external force and torque deviation consists of the rate of change of the normal force component and the rate of change of the torque component in the rotation direction at each moment.

5. The autonomous re-grasping method for a harvesting robot after the gripper slips out, as described in claim 3, is characterized in that... The process of aligning and stitching along the time dimension to generate a slippage bias feature map includes: Set the angle deviation time series as The pressure distribution deviation characteristic sequence is The vibration frequency domain deviation characteristic sequence is The characteristic sequence of external force and torque deviation is as follows Then the slip deviation characteristic map The formula is: ； in 4 represents the total number of sampling times and the number of feature dimensions.

6. The autonomous re-grasping method for a harvesting robot after the gripper slips out, as described in claim 1, is characterized in that... The analysis based on the slip deviation feature map identifies the slip state category and slip severity score of the current grasping action, including: S601, flatten the slip deviation feature map into a one-dimensional feature vector, input it into the pre-trained slip state recognition model, and obtain the slip state category of the current grasping action; S602, calculate the symbolic distance from the one-dimensional feature vector to the classification decision boundary of the slip state recognition model, and map the symbolic distance to the slip severity score.

7. The autonomous re-grasping method for a harvesting robot after the gripper slips out, as described in claim 6, is characterized in that... The calculation of the symbolic distance from the one-dimensional feature vector to the classification decision boundary of the slippage state recognition model, and mapping the symbolic distance to a slippage severity score, includes: The formula for calculating the multi-class decision function is as follows: ； in The predicted slip state category is 0, which represents no slip, 1 represents slight slip, and 2 represents complete slip. The number of support vectors; For the first The multipliers of the support vectors; For the first The class labels of the support vectors; It is a radial basis kernel function; For the first Support vectors; For bias terms; The input is a one-dimensional feature vector.

8. The autonomous re-grasping method for a harvesting robot after the gripper slips out, as described in claim 1, is characterized in that... The re-grabbing strategy, determined based on slippage state category and slippage severity score, includes: S801, identify the slip state category. When the slip state category is no slip, maintain the current grasping state and execute the subsequent picking and handling process; S802, when the slippage state category is micro slippage, dynamic compensation control of the gripper's clamping force is initiated based on the slippage severity score, gradually increasing the clamping force to the preset upper limit value, and at the same time, the robot arm's end posture is finely adjusted according to the angular deviation direction to obtain a first-level heavy gripping strategy; S803, when the slippage state category is complete slippage, generates a secondary re-grasping strategy that abandons the current gripping action and re-identifies the fruit position and posture based on the visual feedback system, and plans a new gripping point and gripping path.

9. The autonomous re-grasping method for a harvesting robot after the gripper slips out, as described in claim 1, is characterized in that... The execution of the re-capture strategy, and the subsequent acquisition of multimodal sensing data for feedback monitoring after execution, includes: S901, executes the corresponding recovery operation according to the re-capture strategy; S902, after execution, acquires the re-captured multimodal sensing data, and re-identifies the re-captured slip state category and the re-captured slip severity score; S903: Calculate the change between the slip severity score before re-grabbing and the slip severity score after re-grabbing. If the change is greater than or equal to a preset effective threshold, the re-grabbing is deemed invalid and an alarm is triggered. If the change is less than the preset effective threshold, the re-grabbing is deemed successful and the subsequent picking task continues.

10. A system for implementing the autonomous re-grasping method of a picking robot gripper after slippage according to any one of claims 1-9, characterized in that, include: The multimodal sensing data acquisition module is used to acquire multimodal sensing data of the picking robot during the grasping process; The slip deviation feature map construction module constructs a slip deviation feature map based on multimodal sensing data; The slip deviation feature map analysis module analyzes the slip deviation feature map to identify the slip state category and slip severity score of the current grasping action; The re-grabbing strategy determination module determines the re-grabbing strategy based on the slip state category and slip severity score; The feedback monitoring module is used to execute the re-capture strategy and, after execution, reacquire multimodal sensing data for feedback monitoring.

Images