Robot intelligent grabbing control method and system based on ToF camera and deep learning

Abstract

This invention provides a robot intelligent grasping control method and system based on a ToF camera and deep learning, relating to the field of intelligent robot technology. The method includes: acquiring the target pose information of the target object in the world coordinate system and the current joint state of the robot; generating a first grasping action sequence based on the current joint state and the target pose information using a first path generation method; generating a second grasping action sequence based on the target pose information and / or current environment information using a second path generation method, wherein the second path generation method differs from the first path generation method; performing a consistency judgment on the first and second grasping action sequences, and correcting the second grasping action sequence according to the judgment result to obtain a final grasping action sequence; and controlling the robot to execute the final grasping action sequence to complete the grasping of the target object. This invention solves the problems of low accuracy and poor robustness in existing robot grasping methods in complex environments.

Description

Technical Field

[0001] This invention relates to the field of industrial robot technology, and in particular to a robot intelligent grasping control method and system based on ToF camera and deep learning, which is suitable for automated production lines, intelligent logistics sorting and other scenarios with high requirements for grasping accuracy, robustness and adaptability. Background Technology

[0002] With the development of modern technology, robots are increasingly widely used in precision assembly, flexible sorting, and human-robot collaboration. Achieving accurate targeting, stable grasping, and correct placement of objects by robots is a core aspect of improving the intelligence and flexibility of production lines. Currently, mainstream robot grasping solutions generally adopt a technical path of visual perception, pose estimation, trajectory planning, and grasping execution. However, in practical industrial applications, robot grasping still faces many shortcomings: (1) Low perception accuracy and reliability: Traditional robot grasping systems mostly use fixed-mounted RGB-D cameras (such as structured light cameras) for global perception. However, this grasping method is easily affected by ambient light interference, and during the robot's movement, the robotic arm itself is very likely to obstruct the target's field of vision, resulting in blind spots and failing to provide continuous and high-precision target pose feedback during the grasping approach phase. Although mounting the camera at the end of the robotic arm (Eye-in-Hand) can alleviate the obstruction, the decrease in the accuracy of the depth sensor at close range and the secondary obstruction of the field of vision by the end effector itself have not been effectively solved.

[0003] (2) Poor planning intelligence and model credibility: With the development of deep learning, data-driven grasping strategy networks (such as those trained through imitation learning and reinforcement learning) can output seemingly intelligent grasping trajectories. However, these black-box models have limited generalization ability outside the training scenario, and their output trajectories have unpredictable deviations. In actual execution, due to the lack of an online evaluation mechanism for the credibility of the model's output trajectory, the system can only execute blindly. Once the model produces erroneous outputs due to changes in the environment or differences in the target object, it will directly lead to grasping failure or collision, lacking basic fault tolerance and safety guarantees.

[0004] (3) Poor lag and adaptability of error compensation: Existing technologies typically generate a fixed trajectory during the planning phase or rely on simple visual servoing (such as position-based visual servoing PBVS) for position correction during execution. These methods cannot systematically distinguish and compensate for multiple sources of error, including estimation errors of the perception model itself, output errors of the data-driven policy network, and kinematic calibration errors of the robot itself. More importantly, there is a lack of a system that can dynamically balance the intelligence of the data-driven model with the reliability of the traditional kinematic model. When the output of the learning model is unreliable, the system cannot intelligently enhance the weights of the baseline trajectory based on classical kinematic principles, resulting in poor correction effects and poor robustness in dealing with unstructured scenarios and uncertainties.

[0005] It is evident that existing robot grasping methods have significant shortcomings in many aspects, such as high-precision near-field perception, online reliability assessment of intelligent planning models, and adaptive fusion compensation of multi-source errors, which limit the robot's ability to achieve high success rate and high precision grasping in complex dynamic environments. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a robot intelligent grasping control method and system based on a ToF camera and deep learning. During the robot grasping process, the reliability of the data-driven model output trajectory is judged online, and classical kinematic reference trajectories are dynamically and adaptively fused accordingly. This enables intelligent compensation for multi-source errors, thereby solving the problems of low grasping accuracy and poor robustness in existing robot grasping methods under uncertain and complex environments.

[0007] On one hand, the present invention provides a robot intelligent grasping control method based on a ToF camera and deep learning, the method comprising: Obtain the target pose information of the target object in the world coordinate system, and obtain the current joint state of the robot; Based on the current joint state and the target pose information, a first grasping action sequence is generated using a first path generation method; based on the target pose information and / or the current environment information, a second grasping action sequence is generated using a second path generation method, and the second path generation method is different from the first path generation method. The consistency of the first grasping action sequence and the second grasping action sequence is determined, and the second grasping action sequence is corrected according to the determination result to obtain the final grasping action sequence; The robot is controlled to execute the final grasping action sequence to complete the grasping of the target object.

[0008] Furthermore, the first path generation method is a path generation method based on the robot's kinematics model; the second path generation method is a path generation method based on a data-driven model.

[0009] Furthermore, the path generation method based on the robot kinematics model specifically includes: Based on forward kinematics, the current pose of the robot end effector is calculated according to the current joint state; based on inverse kinematics, the target joint angle is solved according to the target pose information; and interpolation is performed between the current pose of the robot end effector and the target joint angle to generate the first grasping action sequence.

[0010] Furthermore, the path generation method based on the data-driven model specifically includes: The observation information, including the target pose information, is input into a grasping policy network pre-trained through imitation learning or reinforcement learning, and the second grasping action sequence is output.

[0011] Furthermore, a consistency judgment is performed on the first grasping action sequence and the second grasping action sequence, and the second grasping action sequence is corrected based on the judgment result, specifically including: Calculate the deviation statistics of the second grasping action sequence relative to the first grasping action sequence; When the deviation statistic exceeds a preset threshold, a correction amount is generated to adjust the second grasping action sequence to the first grasping action sequence based on the ratio between the deviation statistic and the preset threshold. The second grasping action sequence is adjusted based on the correction amount to obtain the final grasping action sequence.

[0012] Furthermore, the deviation statistic is obtained based on the variance or standard deviation of the joint angle difference sequence between the first grasping action sequence and the second grasping action sequence.

[0013] Furthermore, the final grasping action sequence is obtained as follows: , ; in, This is the final grasping action sequence; This is the second grasping action sequence; This is the first grasping action sequence; To correct the gain coefficient, used to control the correction magnitude. ; This is a correction amount.

[0014] Furthermore, the target pose information of the target object in the world coordinate system is obtained, including: Three-dimensional point cloud data of the target object are acquired by a ToF camera installed on the robot's end effector; The three-dimensional point cloud data is input into a neural network for feature extraction and pose transformation, and the target pose information is output.

[0015] Furthermore, the method also includes: after the end effector contacts the target object, switching to a gripping force closed-loop control mode based on a tactile sensor, and dynamically adjusting the gripping force based on the feedback contact force information until the gripping is stable.

[0016] On the other hand, the present invention provides a robot intelligent grasping control system based on a ToF camera and deep learning, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of any of the methods described above.

[0017] In summary, this invention provides a robot intelligent grasping control method and system based on a ToF camera and deep learning. Compared with the prior art, the technical solution conceived in this invention can achieve the following beneficial effects: (1) By introducing a consistency judgment between the first grasping action sequence and the second grasping action sequence, and dynamically correcting the trajectory based on the judgment result, this invention fundamentally overcomes the technical defects of a single planning source, especially the unreliable output and lack of safety guarantee of the data-driven model in unknown scenarios, and significantly improves the overall robustness, safety and task success rate of robot grasping control in unstructured and dynamic environments.

[0018] (2) The correction method proposed in this invention, which is based on the comparison between the deviation statistic and the preset threshold, can quantify the degree of anomaly in the output trajectory of the data-driven model. By dynamically converting the ratio of the deviation statistic to the preset threshold into a correction value and using a weighted formula for fusion, the model can retain its intelligence when outputting reliable results and automatically enhance the weight of the reference trajectory based on classical kinematics principles when outputting unreliable results. This achieves the optimal balance between high accuracy and robustness, as well as adaptive compensation for various model errors and environmental disturbances.

[0019] (3) This invention combines high-precision end-effector ToF visual perception, deep learning-based pose estimation, intelligent trajectory planning, real-time dynamic correction and tactile force control into a unified adaptive grasping control method. This not only ensures the positioning accuracy in the grasping approximation stage, but also ensures the stability and compliance in the grasping contact stage through tactile feedback. This enables the system to safely and accurately handle a variety of target objects, from rigid industrial parts to fragile and flexible objects, greatly expanding the application range of robots in complex industrial and service scenarios. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the method steps of a robot intelligent grasping control method and system based on ToF camera and deep learning provided by the present invention; Figure 2 This invention provides a schematic diagram of a robot end effector for a robot intelligent grasping control method and system based on a ToF camera and deep learning. Figure 1 ; Figure 3 This invention provides a schematic diagram of a robot end effector for a robot intelligent grasping control method and system based on a ToF camera and deep learning. Figure 2 . Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0023] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method, step, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the method, step, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the method, step, or apparatus that includes said element.

[0024] To address the issues of low grasping accuracy and poor robustness in existing robotic grasping methods operating in uncertain and complex environments, this invention provides a robot intelligent grasping control method and system based on a ToF camera and deep learning. During the robot grasping process, the reliability of the data-driven model's output trajectory is judged online, and classical kinematic reference trajectories are dynamically and adaptively fused accordingly. This enables intelligent compensation for multi-source errors, improving compensation capabilities and achieving high precision, robustness, and scene adaptability in grasping target objects.

[0025] Specifically, such as Figure 1 As shown, the method of the present invention includes: S100: Obtain the target pose information of the target object in the world coordinate system, and obtain the current joint state of the robot.

[0026] As an example, obtaining the target pose information of the target object in the world coordinate system includes: acquiring three-dimensional point cloud data of the target object through a ToF camera installed on the robot's end effector; inputting the three-dimensional point cloud data into a neural network for feature extraction and pose transformation, and outputting the target pose information.

[0027] Furthermore, 3D point cloud data is constructed based on depth image information and camera parameters.

[0028] Specifically, the robotic arm is controlled to move so that the ToF camera is aimed at the target object. A relatively complete depth image of the target object's surface is acquired. At the same time, combined with the intrinsic and extrinsic parameters of the ToF camera, the depth value of each pixel in the depth image is converted into the point cloud coordinates (X, Y, Z) of the three-dimensional spatial point, thereby obtaining the three-dimensional point cloud data of the target object and realizing the three-dimensional shape perception of the target object.

[0029] The intrinsic data includes focal length and principal point coordinates, which can be obtained through camera calibration; the extrinsic data refers to the transformation relationship between the camera coordinate system and the world coordinate system, which can be determined through hand-eye calibration.

[0030] It should be noted that the ToF camera is fixedly mounted at the geometric center of the end effector body, and its optical axis is parallel to or coincides with the central axis of the end effector in the grasping direction. For example, a small ToF camera (such as the Sipeed MaixSense A075V) can be mounted in the middle of the robot's end effector, such as in the palm of the dexterous hand or the center of the gripper, to reduce the occlusion of the camera's field of view by the robot body. Compared with structured light cameras and ordinary RGB cameras, using a ToF camera located at the center of the end effector to acquire depth information is not affected by ambient light, has a wide measurement range, and is less likely to be obstructed by itself or the environment. It can accurately acquire the 3D point cloud data of the target object dynamically throughout the entire grasping process. In addition, even in a completely dark environment, basic grasping functions can still be achieved using only the ToF camera at the end effector.

[0031] To improve the quality of subsequent processing, the 3D point cloud data can be further preprocessed. This preprocessing includes at least statistical filtering to remove outliers and downsampling using a voxel mesh method.

[0032] Statistical filtering removes outliers by calculating the average distance between each point and its k nearest neighbors. Assuming this distance follows a Gaussian distribution, outliers whose mean distance exceeds a specified multiple (e.g., 2 times) of the standard deviation are removed to eliminate noise.

[0033] Downsampling is used to create a voxel grid of a preset size (e.g., 3mm). The centroid of all points within each voxel is used to approximate the points in the voxel, so as to reduce the number of points while ensuring the shape of the point cloud and improve processing efficiency.

[0034] The process involves inputting 3D point cloud data into a neural network for feature extraction and pose transformation, and outputting target pose information. Specifically, this includes: inputting 3D point cloud data into a pre-trained neural network, using the neural network to extract the geometric features of the target object, such as shape, contour, and spatial distribution, and then outputting the target pose information of the target object in the world coordinate system through coordinate transformation algorithms (such as ICP algorithm and pose regression based on deep learning).

[0035] In this embodiment, the neural network can be a convolutional neural network (CNN), and its network architecture can be any of PointNet, PointNet++, or VoteNet, including a multilayer perceptron (MLP) and an ensemble abstraction layer for extracting multi-scale geometric features. The multi-scale geometric features of the point cloud are extracted through the MLP and ensemble abstraction layer, and finally, the 6-DOF target pose information of the target object is regressed and output through a fully connected layer. Furthermore, a large amount of point cloud data with real pose labels can be used to train the pose estimation network through supervised learning (such as the Smooth L1 loss function) to improve the accuracy of the target pose information.

[0036] It should be noted that target pose information refers to the precise position and orientation of the target object in the world coordinate system (a global reference system with the robot's base as the origin). It is a 6-DOF parameter, a 4x4 homogeneous transformation matrix, including the target object's position vector and rotation matrix in three-dimensional space. The position vector represents the (X, Y, Z) coordinates of the target object's center or a specific grasping point in the world coordinate system. The rotation matrix represents the target object's orientation relative to the world coordinate system (e.g., rotation angles around the X, Y, and Z axes, or expressed as quaternions, rotation matrices, etc.).

[0037] S200: Based on the current joint state and target pose information, generate a first grasping action sequence using a first path generation method; based on the target pose information and / or current environment information, generate a second grasping action sequence using a second path generation method, wherein the second path generation method is different from the first path generation method.

[0038] As an example, the first path generation method is a path generation method based on the robot's kinematics model; the second path generation method is a path generation method based on the data-driven model.

[0039] Furthermore, the path generation method based on the robot kinematics model specifically includes: calculating the current pose of the robot end effector based on the current joint state according to the forward kinematics; solving the target joint angle based on the target pose information according to the inverse kinematics; and interpolating between the current pose of the robot end effector and the target joint angle to generate the first grasping action sequence.

[0040] It should be noted that the first grasping action sequence is a smooth, collision-free baseline trajectory (collision detection has been performed during planning) from the starting point to the target point, i.e., the joint state of the baseline action sequence.

[0041] Specifically, reading the current joint state of the robot Specifically, it includes: Through positive kinematic function Calculate the current pose of the end effector: .

[0042] Based on the current joint state Using the seed, the inverse kinematics function Solve the target pose Corresponding target joint state: .

[0043] Within the joint space, in the current joint state Target joint angle state Linear interpolation is performed between them to generate the first grabbing action sequence containing N+1 path points. ,in, , ; .

[0044] Specifically, the first grasping action sequence The calculation method is as follows: ; in, express The first grasping action sequence at any given moment; Indicates the current joint state of the robot's end effector; Indicates the time sequence number. , This is the initial time of the correction startup; Represents the discrete total number of steps in the action sequence for grasping the target object; Indicates the state of the target joint.

[0045] Furthermore, the path generation method based on the data-driven model specifically includes: inputting observation information including target pose information into a grasping policy network pre-trained through imitation learning or reinforcement learning, and outputting a second grasping action sequence.

[0046] It should be noted that the current observation information includes target pose information. Current joint status And supplementary RGB image features provided by the global RGB camera.

[0047] As an example, training the grasping policy network includes: collecting joint state sequences and synchronized visual observation data during the grasping task performed by a human-operated robot as a training dataset; using a Transformer-based behavioral cloning model (such as ACT) to supervise the training dataset to obtain the grasping policy network; the input information of the grasping policy network is the current joint state and its corresponding visual observation data, and the output information is the joint state at the next moment. Furthermore, during training, a large number of grasping scenarios can be simulated in a simulation environment (such as Issac Lab, Gazebo, PyBullet) to accelerate training convergence.

[0048] During the execution of a specific grasping task, by inputting the joint state of the robot's end effector and the timing input from the vision sensor, a real-time updated sequence of target actions for several future moments can be obtained, i.e., the second grasping action sequence. .

[0049] Specifically, the second grasping action sequence This represents a more efficient or better-adapted grasping strategy learned from data, specifically tailored to the shape of a particular object; among which, , .

[0050] Compared to existing grasping controllers based on simple models, this invention trains different grasping controllers based on different models, enabling the grasping controller to output multiple grasping trajectories according to the target object's pose. It can dynamically correct the grasping trajectory based on the actual pose of the target object, effectively compensating for model errors such as camera calibration errors and robotic arm kinematic errors, as well as environmental errors such as small object displacements, thus improving grasping accuracy, especially suitable for small and complex shaped target objects.

[0051] S300: Perform a consistency judgment on the first grasping action sequence and the second grasping action sequence, and correct the second grasping action sequence according to the judgment result to obtain the final grasping action sequence.

[0052] As one embodiment, a consistency judgment is performed on the first grasping action sequence and the second grasping action sequence, and the second grasping action sequence is corrected according to the judgment result, specifically including: S301: Calculate the deviation statistics of the second grasping action sequence relative to the first grasping action sequence.

[0053] It should be noted that the deviation statistic is the joint angle difference vector for each path point; the joint angle difference vector The calculation method is as follows: , .

[0054] To quantify the overall deviation, the total variance of all joints across all path points is calculated as the deviation statistic. As an example, the deviation statistic is obtained based on the variance or standard deviation of the joint angle difference sequence between the first and second grasping action sequences.

[0055] Furthermore, the deviation statistic is calculated as follows: , ; in, Indicates the first The variance of the joint angle differences; Indicates the first The difference in joint angles; Indicates the first The mean of the differences between the joints .

[0056] S302: When the deviation statistic exceeds a preset threshold, a correction amount is generated to adjust the second grasping action sequence to the first grasping action sequence based on the ratio between the deviation statistic and the preset threshold.

[0057] The calculated With preset threshold In comparison, specifically: like If the data-driven model output is considered reliable, then the final grasping action sequence will be directly output. ; like If the output of the data-driven model is significantly anomaly, it is considered to require correction. Specifically, the correction amount is the variance coefficient, calculated as follows: .

[0058] It should be noted that the preset threshold can be dynamically calibrated based on the robot's repeatability and the average error of the grasping strategy network in simulation tests, and can also be fine-tuned based on the estimated attributes of the target object, such as mass and friction coefficient.

[0059] Furthermore, throughout the control process, the method also includes: calculating the total variance of the entire joint over P consecutive control cycles (e.g., P=3). All are greater than the preset threshold If the current scene (e.g., extreme occlusion, strong light interference, or unseen objects) is deemed to have exceeded the reliable operating range of the data-driven model, a safety mode will be triggered: either pausing motion and issuing an alarm, or automatically switching to a degraded mode based on pure kinematics planning; that is, ignoring the second grasping action sequence. The first grabbing action sequence is used directly. As the final grabbing action sequence, it ensures safety while guaranteeing basic functionality.

[0060] S303: Adjust the second grasping action sequence based on the correction amount to obtain the final grasping action sequence.

[0061] The corrected final grasping action sequence is generated using a weighted fusion formula. As an example, the final grasping action sequence is obtained as follows: , ; in, This is the final grasping action sequence; This is the second grasping action sequence; This is the first grasping action sequence; To correct the gain coefficient, used to control the correction magnitude. ; This is a correction amount.

[0062] It should be noted that the correction gain coefficient can be adaptively adjusted based on the estimated weight or surface friction coefficient of the target object; specifically, for target objects with large estimated weight or low friction coefficient, a smaller gain coefficient is used. Value; for target objects with small estimated weight or high coefficient of friction, a larger value is used. Value. Correction amount. The larger the value, the less reliable the output of the data-driven model; the more likely the correction will be. To a greater extent towards reliability To move closer.

[0063] By combining robot kinematics models and data-driven models, it compensates for camera calibration errors, robotic arm kinematic errors, and environmental errors, enabling dynamic optimization of the grasping trajectory. This significantly improves the stability and success rate of grasping, thus adapting to complex and diverse grasping scenarios.

[0064] S400: Controls the robot to execute the final grasping action sequence to complete the grasping of the target object.

[0065] It should be noted that the manipulator is controlled to execute the final grasping action sequence . During the execution process, after every M steps (M < N) are executed, it returns to step S100, and the ToF camera is used to obtain the latest target pose information of the target object in the world coordinate system. This is continuously repeated to perform a new round of pose estimation, trajectory generation, and correction, thereby forming a dynamic closed loop.

[0066] When the position vector of the current pose of the end effector is calculated through forward kinematics and the position vector of the target pose the Euclidean distance between them is less than or equal to the grasping-in-place distance threshold (for example, 5 mm), that is: , it is considered that it has approached in place and enters the contact grasping stage.

[0067] In addition, in order to avoid object misalignment, damage, or end effector overload caused by hard impacts, as another embodiment, the method further includes: after the end effector contacts the target object, it switches to a tactile grasping force control mode based on a tactile sensor, and dynamically adjusts the clamping force until the grasping is stable through the feedback contact force information.

[0068] That is to say, in the final stage of grasping, that is, after the end effector contacts the object, the piezoresistive fingertip tactile sensor, based on the piezoresistive effect, converts physical quantities such as the contact force, deformation, and slip between the fingertip and the object into quantifiable electrical signals in real time, providing a grasping force closed-loop control feedback for the grasping end effector to achieve lossless and stable adaptive grasping.

[0069] Furthermore, after the end effector contacts the target object, it switches to a grasping force closed-loop control mode based on a tactile sensor. The grasping force closed-loop control mode specifically includes: Using the tactile sensors embedded in the fingertips of each finger of the end effector to continuously collect the pressure distribution signals of the contact surface. After filtering and noise reduction processing, the average contact force and the position of the pressure center are calculated; Taking the preset contact force as the target, a digital PID controller is used to calculate the grasping force adjustment amount; Converting the grasping force adjustment amount into a control instruction for the end effector to dynamically adjust the grasping force; at the same time, by monitoring the movement of the pressure center position, the object slip is detected in real time, and the pose adjustment or an increase in the clamping force is triggered to prevent slip; When the average contact force stabilizes within a safe range and lasts for a certain period of time, and no slip is detected, it is determined that the grasping is stable, and the entire grasping action is completed. Among them, the safe range is: .

[0070] Specifically, the calculation method of the grasping force adjustment amount is as follows: ; ; in, This represents the proportional gain, used to amplify the current force error and provide an immediate control response. The larger the error, the greater the control input. The larger; Indicates force error; Indicates the preset contact force; Indicates the current Average contact force at time; This represents the integral gain, which amplifies the integral of the force error to eliminate steady-state error (a small error that has been present for a long time). This represents the differential gain, which is used to amplify the rate of change of force error (error derivative), suppress overshoot and oscillation, and improve stability.

[0071] Furthermore, if the average contact force is less than the minimum safe range, it indicates insufficient gripping force and a risk of the object slipping. In this case, it is necessary to gradually increase the joint driving force of each finger at the end of the actuator to improve the gripping force. If the average contact force is greater than the maximum safe range, it indicates excessive gripping force, which may cause deformation or damage to the target object. In this case, the control unit appropriately reduces the joint driving force of the fingers to reduce the gripping force. At the same time, based on the deviation between the pressure center position and the contact point position fed back by the tactile sensor, the posture of each finger is finely adjusted to ensure that the fingers are in close contact with the object surface and to prevent the object from shifting or rotating during the gripping process.

[0072] This invention accurately obtains the adjustment amount of the gripping force of the robot's end effector by combining a proportional term (instantaneous response), an integral term (eliminating steady-state error), and a derivative term (suppressing oscillations). It adjusts the gripping force and finger posture in real time until a preset stable gripping state is achieved, including a stable average contact force within a safe range for a certain period of time, a stable contact area, and no object displacement. At this point, an adaptive gripping action is completed, enabling the handling and placement of objects. This allows for compliant gripping of fragile objects such as eggs and glass, ensuring that the control force does not exceed the safety threshold; stable gripping of rigid objects such as metal blocks, ensuring that the force is large enough to overcome gravity / friction; and force constraint control for assembly tasks such as inserting pins, controlling the contact force within an appropriate range to avoid being too tight or too loose.

[0073] It should be noted that after trajectory correction and error compensation, the end effector gradually approaches and contacts the target object according to the corrected precise trajectory. After the end effector contacts the object, the tactile sensor detects a pressure signal, switching from visual trajectory correction mode to tactile grasping force control mode. It no longer relies on pose feedback from visual imaging, but primarily focuses on adjusting the grasping force after contact. Centered on the end effector, the grasping action is executed, and the grasping force is adjusted in real time through feedback from the tactile sensor located on the end effector, achieving precise, stable, soft, and safe grasping of the target object. Furthermore, the tactile sensor ensures a precise closed-loop grasping force. By providing feedback on the grasping force, the actuator's stroke and attitude are dynamically adjusted, stabilizing the grasping force within a threshold that prevents slippage and damage, thus adapting to irregularly shaped, fragile, and flexible target objects.

[0074] On the other hand, this invention proposes a robot intelligent grasping control system based on a ToF camera and deep learning, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement any of the above-mentioned methods. The technical features of the system are consistent with the technical features of the method, and will not be repeated here.

[0075] As an example, the method of the present invention can be further implemented in a control system, which includes: a six-degree-of-freedom robotic arm, an end effector, a ToF camera mounted on the central base of the end effector gripper, a global RGB camera fixed above the work scene, a piezoresistive tactile sensor array embedded in the gripper's fingertips, and an industrial computer as the control core. The industrial computer communicates with and controls all the above hardware through the ROS operating system.

[0076] It should be noted that robotic arms can take many forms, such as Figure 2-3 As shown, it can be installed on the robot arm or directly on the machine's fixed platform. The end effector is mounted on the end of the robot arm, and the ToF camera is installed in the center of the end effector, such as the center of the palm or the center of the gripper, so that the ToF camera's field of view is not blocked by the robot arm, while the RGB camera is placed in a fixed position off-site.

[0077] In summary, this invention fundamentally overcomes the technical shortcomings of single planning sources, especially data-driven models, which suffer from unreliable output and lack of safety guarantees in unknown scenarios, by introducing a consistency judgment between the first and second grasping action sequences and dynamically correcting the trajectory based on the judgment result. This significantly improves the overall robustness, safety, and task success rate of robot grasping control in unstructured and dynamic environments. Furthermore, this invention can be widely applied to various scenarios such as industrial automated production lines (e.g., parts grasping and assembly), intelligent logistics sorting (e.g., e-commerce parcel grasping), and service robots (e.g., household item grasping), demonstrating strong versatility.

[0078] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0079] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed methods or systems can be implemented in other ways. For example, the embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0084] Those skilled in the art will understand that all or part of the circuits in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0085] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0086] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robot intelligent grasping control method based on ToF camera and deep learning, characterized in that, The method includes: Obtain the target pose information of the target object in the world coordinate system, and obtain the current joint state of the robot; Based on the current joint state and the target pose information, a first grasping action sequence is generated using a first path generation method; based on the target pose information and / or the current environment information, a second grasping action sequence is generated using a second path generation method, and the second path generation method is different from the first path generation method. The consistency of the first grasping action sequence and the second grasping action sequence is determined, and the second grasping action sequence is corrected according to the determination result to obtain the final grasping action sequence; The robot is controlled to execute the final grasping action sequence to complete the grasping of the target object.

2. The robot intelligent grasping control method based on ToF camera and deep learning according to claim 1, characterized in that, The first path generation method is a path generation method based on the robot's kinematics model; the second path generation method is a path generation method based on a data-driven model.

3. The intelligent grasping control method for robots based on a ToF camera and deep learning according to claim 2, characterized in that, The path generation method based on the robot kinematics model specifically includes: Based on forward kinematics, the current pose of the robot end effector is calculated according to the current joint state; based on inverse kinematics, the target joint angle is solved according to the target pose information; and interpolation is performed between the current pose of the robot end effector and the target joint angle to generate the first grasping action sequence.

4. The robot intelligent grasping control method based on ToF camera and deep learning according to claim 2, characterized in that, The path generation method based on the data-driven model specifically includes: The observation information, including the target pose information, is input into a grasping policy network pre-trained through imitation learning or reinforcement learning, and the second grasping action sequence is output.

5. The robot intelligent grasping control method based on ToF camera and deep learning according to claim 1, characterized in that, A consistency check is performed on the first grasping action sequence and the second grasping action sequence, and the second grasping action sequence is corrected based on the check result, specifically including: Calculate the deviation statistics of the second grasping action sequence relative to the first grasping action sequence; When the deviation statistic exceeds a preset threshold, a correction amount is generated to adjust the second grasping action sequence to the first grasping action sequence based on the ratio between the deviation statistic and the preset threshold. The second grasping action sequence is adjusted based on the correction amount to obtain the final grasping action sequence.

6. The intelligent grasping control method for robots based on a ToF camera and deep learning according to claim 5, characterized in that, The deviation statistic is obtained based on the variance or standard deviation of the joint angle difference sequence between the first grasping action sequence and the second grasping action sequence.

7. The intelligent grasping control method for robots based on a ToF camera and deep learning according to claim 5, characterized in that, The final grasping action sequence is obtained as follows: ， ； in, This is the final grasping action sequence; This is the second grasping action sequence; This is the first grasping action sequence; To correct the gain coefficient, used to control the correction magnitude. ; This is a correction amount.

8. The intelligent grasping control method for robots based on a ToF camera and deep learning according to claim 1, characterized in that, Obtain the target pose information of the target object in the world coordinate system, including: Three-dimensional point cloud data of the target object are acquired by a ToF camera installed on the robot's end effector; The three-dimensional point cloud data is input into a neural network for feature extraction and pose transformation, and the target pose information is output.

9. A robot intelligent grasping control method based on a ToF camera and deep learning according to claim 8, characterized in that, The method further includes: after the end effector contacts the target object, switching to a gripping force closed-loop control mode based on tactile sensors, and dynamically adjusting the clamping force based on the feedback contact force information until the gripping is stable.

10. A robot intelligent grasping control system based on a ToF camera and deep learning, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images