A method, device, and medium for robot hand-eye localization based on a 3D camera

Abstract

This application discloses a robot hand-eye localization method, device, and medium based on a 3D camera. The method includes: real-time scanning of the scene where the target object is located using a 3D camera to determine the point cloud data of the target object; predicting the motion state of the target object according to a pre-determined prediction model to obtain a predicted trajectory; matching the predicted trajectory with the real-time scanned point cloud data to determine the prediction difference; adjusting the prediction model according to the prediction difference; if the target object is occluded, determining the historical point cloud data of the target object; adjusting the prediction model according to the historical point cloud data; re-predicting the target object using the adjusted prediction model to obtain a prediction result; determining a pre-set hand-eye matrix; mapping the prediction result to the hand-eye matrix to dynamically generate the path of the target object; and controlling the robot's end effector to reach the corresponding predicted position according to the path to achieve grasping at the predicted position.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method, device and medium for robot hand-eye localization based on a 3D camera. Background Technology

[0002] In robot grasping or interaction scenarios, when the target object is in motion, even after hand-eye matrix calibration, obtaining the object's real-time pose still requires data collected by a camera and accurately mapped to the robot's coordinate system. However, traditional static mapping methods have significant drawbacks. When the object is moving at high speed, there is a time lag between the camera's data acquisition and the robot's execution, and this time asynchrony error accumulates continuously. Because traditional methods cannot adapt to rapid changes in the object's position and posture in real time and dynamically, relying on static mapping results to guide robot operations when facing dynamic targets that move at high speed or have frequently changing postures can easily lead to grasping failures or significant delays during the grasping process. This severely affects the efficiency and accuracy of robot operations and fails to meet the requirements for efficient and precise operation of dynamic targets. Summary of the Invention

[0003] To address the aforementioned issues, this application proposes a robot hand-eye localization method based on a 3D camera, comprising: real-time scanning of the scene where the target object is located using a 3D camera to determine the point cloud data of the target object, wherein the point cloud data includes the three-dimensional coordinate information of each point on the surface of the target object; predicting the motion state of the target object according to a pre-determined prediction model to obtain a predicted trajectory; matching the predicted trajectory with the real-time scanned point cloud data to determine the prediction difference, wherein the prediction difference includes distance difference and shape difference; adjusting the prediction model according to the prediction difference; determining whether the target object is occluded; if the target object is occluded, determining the historical point cloud data of the target object; adjusting the prediction model according to the historical point cloud data; re-predicting the target object using the adjusted prediction model to obtain a prediction result; determining a pre-set hand-eye matrix; mapping the prediction result to the hand-eye matrix to dynamically generate the path of the target object; and controlling the robot's end effector to reach the corresponding predicted position according to the path to achieve grasping at the predicted position.

[0004] In one example, the motion state of the target object is predicted according to a pre-determined prediction model to obtain a predicted trajectory. Specifically, this includes: determining an initial state based on the current point cloud data of the target object, the initial state including position and velocity; using the historical point cloud data of the target object and related motion features as input to train an LSTM network, thereby outputting the position information of the target object at future time, and thus obtaining the predicted trajectory.

[0005] In one example, the method further includes: mounting the 3D camera on the end effector of the robot to construct a hand-eye system; placing a calibration object in the workspace of the robot; and controlling the robot to drive the 3D camera to take pictures of the calibration object in multiple poses to obtain multiple sets of point cloud data of the calibration object and corresponding end effector pose data.

[0006] In one example, the method further includes: determining a hand-eye transformation matrix according to a pre-set hand-eye calibration equation, performing hand-eye calibration on the robot according to the hand-eye transformation matrix, thereby obtaining the 3D camera coordinate system, and determining the transformation relationship between the 3D camera coordinate system and the end effector coordinate system.

[0007] In one example, the method further includes: acquiring point cloud data of the target object using the 3D camera; preprocessing the acquired point cloud data, the preprocessing including denoising, filtering, and downsampling; identifying the preprocessed point cloud data to determine the category of the target object; and registering the point cloud data with a pre-established target object pose model to obtain the pose information of the target object in the coordinate system of the 3D camera.

[0008] In one example, the method further includes: calibrating the 3D camera according to a pre-set calibration board to obtain the hand-eye matrix of the 3D camera, the hand-eye matrix including an intrinsic parameter matrix and an extrinsic parameter matrix, the intrinsic parameter matrix including the focal length and principal point coordinates of the 3D camera, and the extrinsic parameter matrix including the transformation relationship between the 3D camera coordinate system and the end effector coordinate system; and optimizing the intrinsic parameter matrix and extrinsic parameter matrix using the least squares method to improve the calibration accuracy of the 3D camera.

[0009] In one example, adjusting the prediction model based on the historical point cloud data specifically includes: analyzing the geometric information of the historical point cloud data, reconstructing the prediction model based on the geometric information and the motion law of the target object, and determining the minimum reconstruction ratio of the prediction model based on the category of the target object and the grasping angle.

[0010] In one example, dynamically generating the path of the target object specifically includes: generating a dynamic grasping path based on a dynamic mapping relationship, and determining the predicted motion trajectory of the target object based on the dynamic grasping path, so that the robot's end effector reaches the corresponding predicted position in advance to achieve accurate grasping operation.

[0011] On the other hand, this application also proposes a robot hand-eye localization device based on a 3D camera, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the robot hand-eye localization device based on a 3D camera to perform: real-time scanning of the scene where the target object is located using a 3D camera to determine the point cloud data of the target object, the point cloud data including the three-dimensional coordinate information of each point on the surface of the target object; predicting the motion state of the target object according to a predetermined prediction model to obtain a predicted trajectory; and comparing the predicted trajectory with... Real-time scanned point cloud data is matched to determine prediction differences, including distance and shape differences. The prediction model is adjusted based on these differences. It is then determined whether the target object is occluded. If so, historical point cloud data of the target object is determined, and the prediction model is adjusted based on this data. The adjusted prediction model is used to re-predict the target object to obtain a prediction result. A pre-set hand-eye matrix is ​​determined, and the prediction result is mapped to this matrix to dynamically generate a path for the target object. The robot's end effector is then controlled to reach the corresponding predicted position based on this path, enabling grasping at the predicted position.

[0012] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions, which are configured to: perform real-time scanning of the scene where the target object is located using a 3D camera to determine the point cloud data of the target object, the point cloud data including the three-dimensional coordinate information of each point on the surface of the target object; predict the motion state of the target object according to a pre-determined prediction model to obtain a predicted trajectory, match the predicted trajectory with the real-time scanned point cloud data to determine the prediction difference, the prediction difference including distance difference and shape difference, and adjust the prediction model according to the prediction difference; determine whether the target object is occluded, if the target object is occluded, determine the historical point cloud data of the target object, and adjust the prediction model according to the historical point cloud data; re-predict the target object using the adjusted prediction model to obtain a prediction result, determine a pre-set hand-eye matrix, map the prediction result to the hand-eye matrix to dynamically generate the path of the target object, and control the robot's end effector to reach the corresponding predicted position according to the path to achieve grasping at the predicted position.

[0013] This application utilizes a predictive model to predict the motion state of a target object. By matching the model with real-time point cloud data to identify discrepancies and adjust the model accordingly, it can adapt to changes in object motion in a timely manner, improving prediction accuracy, especially in complex motion scenarios. When the target object is occluded, the predictive model is adjusted using historical point cloud data to avoid positioning failures due to occlusion, enhancing system robustness and ensuring stable operation in complex environments. A hand-eye system is constructed, using multiple sets of calibration object point cloud data and end effector pose data, combined with hand-eye calibration equations to determine the hand-eye transformation matrix, clarifying the transformation relationship between the 3D camera coordinate system and the end effector coordinate system, laying the foundation for subsequent precise control. The target object category is identified and its pose information is registered, providing accurate basis for subsequent path planning and grasping operations. The 3D camera hand-eye matrix is ​​obtained through a calibration board, and the intrinsic and extrinsic parameter matrices are optimized using the least squares method to improve camera calibration accuracy and make positioning more accurate. A dynamic grasping path is generated based on the dynamic mapping relationship, combined with the predicted motion trajectory, allowing the robot's end effector to reach the predicted position in advance, achieving accurate grasping and improving grasping efficiency and success rate. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0015] Figure 1 This is a flowchart illustrating a robot hand-eye localization method based on a 3D camera, as described in an embodiment of this application.

[0016] Figure 2 This is a schematic diagram of a robot hand-eye positioning device based on a 3D camera, as described in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0019] like Figure 1 As shown, in order to solve the above problems, this application provides a robot hand-eye localization method based on a 3D camera, the method including:

[0020] S101. The scene where the target object is located is scanned in real time by a 3D camera to determine the point cloud data of the target object, wherein the point cloud data includes the three-dimensional coordinate information of each point on the surface of the target object.

[0021] In robotic operation scenarios, advanced 3D cameras are used to perform real-time scanning of the complex environment in which the target object is located. The 3D camera can capture scene information from all directions and multiple angles. During the scanning process, it utilizes specific optical principles and technologies to acquire rich geometric data of the target object's surface point by point. Through precise calculation and analysis, the point cloud data of the target object is finally determined. This point cloud data contains the three-dimensional coordinate information of every point on the target object's surface, accurate to the micrometer level. Each coordinate point is interconnected and mutually supportive, jointly outlining the complete three-dimensional shape of the target object. With this precise point cloud data, the robot can clearly "see" the shape and position of the target object, providing solid and reliable data support for subsequent precise operations, such as grasping and handling.

[0022] S102. Predict the motion state of the target object according to a predetermined prediction model to obtain a predicted trajectory. Match the predicted trajectory with real-time scanned point cloud data to determine the prediction difference, which includes distance difference and shape difference. Adjust the prediction model according to the prediction difference.

[0023] In robot intelligent perception and decision-making systems, a predictive model, pre-built and trained and optimized with extensive historical data, accurately predicts the motion state of a target object. This predictive model integrates various kinematic principles, dynamic characteristics, and potential influencing factors of the target object's environment. By inputting various state parameters of the target object at the current moment, such as velocity vector, acceleration components, and forces, it uses a recurrent neural network architecture in deep learning algorithms, such as LSTM, to iteratively calculate the motion trend of the target object over a future period, thereby generating a continuous and smooth predicted trajectory. This trajectory consists of a series of three-dimensional spatial coordinate points arranged in chronological order, accurately depicting the expected motion path of the target object.

[0024] Simultaneously, a 3D camera is used to perform high-frequency, high-precision real-time scanning of the scene containing the target object, acquiring point cloud data of the target object at every moment. This point cloud data contains the three-dimensional coordinate information of a large number of discrete points on the surface of the target object, which can realistically reflect the actual shape and position of the target object in real space. Subsequently, the predicted trajectory is rigorously matched with the point cloud data obtained from the real-time scanning. The matching process employs advanced point cloud registration algorithms, such as the Iterative Closest Point (ICP) algorithm, which accurately determines the prediction difference by calculating the distance error between each point on the predicted trajectory and the corresponding area points in the real-time point cloud data, as well as the shape similarity index. The prediction difference specifically includes distance difference, namely the Euclidean distance deviation between the predicted trajectory points and the actual point cloud data points; and shape difference, which is measured by calculating the Hausdorff distance and other indices between the geometric shape enclosed by the predicted trajectory and the object shape presented by the actual point cloud data.

[0025] Based on the determined prediction discrepancies, an adaptive adjustment algorithm is used to dynamically optimize the prediction model. Adjustment strategies include, but are not limited to, correcting the motion parameter weights in the model, updating the connection weights of the neural network, and optimizing the combination of input features. This improves the accuracy and robustness of the prediction model in predicting the motion state of the target object, ensuring that the robot can accurately track and manipulate the target object in complex and ever-changing environments.

[0026] S103. Determine whether the target object is occluded. If the target object is occluded, determine the historical point cloud data of the target object and adjust the prediction model based on the historical point cloud data.

[0027] In dynamic scenarios of robot operations, to ensure accurate positioning and manipulation of target objects, it is necessary to determine in real time whether the target object is occluded. By comprehensively analyzing the features of point cloud data acquired in real time by a 3D camera and the distribution of depth information in the scene, an occlusion detection algorithm based on deep learning and geometric feature fusion is used to determine whether the target object is occluded. The algorithm first preprocesses the point cloud data at the current moment to extract local geometric features of the target object region, such as surface curvature and normal direction. At the same time, it combines the gradient change information of the depth image to construct a 3D feature descriptor of the target object. This descriptor is compared with a pre-established standard feature library of target objects in the unoccluded state. If the matching degree between the current feature descriptor and the standard feature library is lower than a preset threshold, and there are obvious depth abrupt changes or missing features in the depth image of the target object region, then the target object is determined to be occluded.

[0028] Once the target object is determined to be occluded, the system immediately retrieves its historical point cloud data from the storage module. This historical point cloud data encompasses the target object's three-dimensional geometric information at different times and in different postures, recording its motion patterns and morphological changes. Based on this historical point cloud data, a combination of time series analysis and machine learning is used to adjust the prediction model. Specifically, time series modeling is performed on the historical point cloud data to extract dynamic features such as the target object's velocity and acceleration, and clustering algorithms are used to classify different motion patterns of the target object. Based on the classification results, the dynamic parameters in the prediction model are retrained to optimize the model's ability to predict the target object's motion trend. Simultaneously, the structure of the prediction model is adaptively adjusted based on the target object's geometric features and motion patterns, such as increasing or decreasing the number of hidden layer nodes in the neural network and adjusting hyperparameters like the learning rate, to improve the robustness and accuracy of the prediction model when the target object is occluded, ensuring that the robot can continue to accurately track and manipulate the target object based on the adjusted prediction model.

[0029] S104. The target object is re-predicted using the adjusted prediction model to obtain a prediction result. A pre-set hand-eye matrix is ​​determined, and the prediction result is mapped to the hand-eye matrix to dynamically generate the path of the target object. The robot's end effector is controlled to reach the corresponding predicted position according to the path to achieve grasping at the predicted position.

[0030] After targeted adjustments to the prediction model, it was put into operation to accurately grasp the future motion of the target object. This model is built upon a deep learning framework deeply integrated with kinematic principles. It takes the current point cloud features, motion state parameters, and environmental constraints of the target object as input, and employs a multi-layer neural network for high-dimensional feature extraction and nonlinear mapping. Internally, a Long Short-Term Memory (LSTM) network unit captures the temporal dependencies of the target object's motion, and combined with fully connected layers, iteratively predicts the target object's position, orientation, and other state variables at future time steps. The final output includes prediction results containing detailed information such as the target object's three-dimensional spatial coordinates and rotation angles over multiple future time steps.

[0031] Simultaneously, a pre-set hand-eye matrix is ​​read from the system storage module. The determination of the hand-eye matrix involves a complex calibration process. This is achieved by controlling the robot to drive a 3D camera to capture multi-pose images of the calibration object, collecting a large amount of point cloud data of the calibration object and the corresponding pose data of the robot's end effector. The hand-eye matrix is ​​then solved using hand-eye calibration algorithms, such as the Tsai two-step method or methods based on nonlinear optimization. This hand-eye matrix accurately describes the spatial transformation relationship between the 3D camera coordinate system and the robot's end effector coordinate system, and is a key parameter for achieving precise robot operation.

[0032] The predicted results are mapped to the hand-eye matrix, and matrix multiplication is used to transform the pose information of the target object in the 3D camera coordinate system to the robot's end effector coordinate system. Based on the transformed pose information, a path planning algorithm, such as the Fast Extended Random Tree Algorithm (RRT*) or a model predictive control-based path planning method, is used to dynamically generate a grasping path for the target object. This path planning process fully considers the robot's kinematic constraints, dynamic characteristics, and obstacle information in the workspace to ensure that the generated path is smooth, feasible, and efficient.

[0033] Based on the generated path, the robot control system sends control commands to each joint of the robot. Through real-time feedback control algorithms, such as PID control or model-based control algorithms, it precisely adjusts the joint angles, driving the robot's end effector to move along the planned path and ultimately reach the corresponding predicted position. At the predicted position, the robot's end effector, according to a preset grasping strategy, adjusts its grasping posture and force to achieve stable grasping of the target object, ensuring the accuracy and reliability of the entire grasping process.

[0034] In one embodiment, a 3D camera is mounted on the robot's end effector to construct a hand-eye system; a calibration object is placed in the robot's workspace, and the robot is controlled to drive the 3D camera to take pictures of the calibration object in different poses, thereby acquiring multiple sets of point cloud data of the calibration object in the 3D camera coordinate system and the corresponding pose data of the robot's end effector.

[0035] Based on the hand-eye calibration equation AX = XB, where A is the transformation matrix of the robot end effector between different poses, B is the transformation matrix of the 3D camera observing the calibration object under different poses, and X is the hand-eye transformation matrix to be solved, the hand-eye transformation matrix X is solved by singular value decomposition (SVD) or other numerical methods to achieve robot hand-eye calibration and obtain the accurate transformation relationship between the 3D camera coordinate system and the robot end effector coordinate system.

[0036] In one embodiment, a robot is controlled to move a 3D camera to the area where the target object is located, and the 3D camera acquires point cloud data of the target object. The acquired point cloud data is preprocessed, including denoising, filtering, and downsampling, to improve the quality of the point cloud data and the efficiency of subsequent processing. Feature extraction or deep learning-based methods are used to identify the target object's category from the preprocessed point cloud data. Simultaneously, a point cloud registration algorithm (such as the Iterative Closest Point (ICP) algorithm) is used to register the target object's point cloud data with a pre-established target object model point cloud data to obtain the precise pose information of the target object in the 3D camera coordinate system.

[0037] In one embodiment, a calibration board is used to calibrate a 3D camera, obtaining the camera's intrinsic and extrinsic parameter matrices. The intrinsic parameter matrix contains internal parameters of the 3D camera, such as focal length and principal point coordinates, while the extrinsic parameter matrix describes the transformation relationship between the 3D camera coordinate system and the world coordinate system. Using imaging data of the calibration board in the 3D camera at multiple positions and attitudes, the intrinsic and extrinsic parameter matrices are solved and optimized using the least squares method or other optimization algorithms to improve calibration accuracy.

[0038] In one embodiment, when a target object is occluded, a local object motion model is reconstructed based on historical point cloud data of the target object using methods based on geometric features, motion patterns, or deep learning models. The method based on geometric features reconstructs the model by analyzing the shape, size, and other geometric information of the historical point cloud data and combining it with the object's motion patterns. The method based on motion patterns infers the possible motion of the object during the occlusion period based on the object's trajectory and velocity change patterns at historical moments. The method based on deep learning models uses a pre-trained neural network, inputs historical point cloud data, and outputs a reconstructed object model.

[0039] The minimum reconstruction ratio of the motion model is determined based on the actual object category and the gripping angle. Different categories of objects have different motion characteristics and structural features, and the gripping angle will also affect the completeness requirements of the object model. These factors are taken into account to determine the minimum reconstruction ratio in order to reduce the amount of computation while ensuring the effectiveness of the model.

[0040] In one embodiment, a dynamic mapping relationship between the hand-eye matrix and the motion trajectory is established based on the reconstructed object model and the predicted motion trajectory. This dynamic mapping relationship considers the changes in the object's position and posture at different times, as well as the calibration parameters of the hand-eye system, ensuring that the robot can adjust its operations according to the real-time movement of the object. Based on the established dynamic mapping relationship, a dynamic grasping path is generated. This dynamic grasping path, based on the predicted motion trajectory of the target object, enables the robotic arm's end effector to reach the predicted position in advance, thereby achieving accurate grasping operations.

[0041] like Figure 2 As shown in the illustration, this application also provides a robot hand-eye positioning device based on a 3D camera, comprising:

[0042] At least one processor; and,

[0043] A memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the 3D camera-based robot hand-eye positioning device to perform the following:

[0045] The scene in which the target object is located is scanned in real time by a 3D camera to determine the point cloud data of the target object, the point cloud data including the three-dimensional coordinate information of each point on the surface of the target object;

[0046] The motion state of the target object is predicted according to a predetermined prediction model to obtain a predicted trajectory. The predicted trajectory is matched with real-time scanned point cloud data to determine the prediction difference, which includes distance difference and shape difference. The prediction model is then adjusted according to the prediction difference.

[0047] Determine whether the target object is occluded; if the target object is occluded, determine the historical point cloud data of the target object and adjust the prediction model based on the historical point cloud data.

[0048] The target object is re-predicted using the adjusted prediction model to obtain a prediction result. A pre-set hand-eye matrix is ​​determined, and the prediction result is mapped to the hand-eye matrix to dynamically generate the path of the target object. The robot's end effector is then controlled to reach the corresponding predicted position based on the path to achieve grasping at the predicted position.

[0049] This application embodiment also provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0050] The scene in which the target object is located is scanned in real time by a 3D camera to determine the point cloud data of the target object, the point cloud data including the three-dimensional coordinate information of each point on the surface of the target object;

[0051] The motion state of the target object is predicted according to a predetermined prediction model to obtain a predicted trajectory. The predicted trajectory is matched with real-time scanned point cloud data to determine the prediction difference, which includes distance difference and shape difference. The prediction model is then adjusted according to the prediction difference.

[0052] Determine whether the target object is occluded; if the target object is occluded, determine the historical point cloud data of the target object and adjust the prediction model based on the historical point cloud data.

[0053] The target object is re-predicted using the adjusted prediction model to obtain a prediction result. A pre-set hand-eye matrix is ​​determined, and the prediction result is mapped to the hand-eye matrix to dynamically generate the path of the target object. The robot's end effector is then controlled to reach the corresponding predicted position based on the path to achieve grasping at the predicted position.

[0054] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0055] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0056] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0060] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0061] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0062] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0063] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0064] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A 3D camera based robot hand-eye positioning method, characterized in that, The method comprises the following steps: Real-time scanning the scene where the target object is located by a 3D camera to determine the point cloud data of the target object, the point cloud data comprising the three-dimensional coordinate information of each point on the surface of the target object; Predicting the motion state of the target object according to a predetermined prediction model to obtain a predicted trajectory, matching the predicted trajectory with the real-time scanned point cloud data to determine a prediction difference, the prediction difference comprising a distance difference and a shape difference, and adjusting the prediction model according to the prediction difference; Determining whether the target object is blocked, and if the target object is blocked, determining the historical point cloud data of the target object, and adjusting the prediction model according to the historical point cloud data; Re-predicting the target object by the adjusted prediction model to obtain a prediction result, determining a pre-set hand-eye matrix, mapping the prediction result with the hand-eye matrix to dynamically generate a path of the target object, and controlling the end effector of the robot to reach a corresponding predicted position according to the path to achieve grasping at the predicted position.

2. The method of claim 1, wherein, The method further comprises the following steps: Determining an initial state according to the point cloud data of the target object at the current time, the initial state comprising a position and a velocity; Taking the historical point cloud data of the target object and related motion features as inputs to train an LSTM network to output the position information of the target object at a future time, and then obtain a predicted trajectory.

3. The method of claim 1, wherein, The method further comprises the following steps: Installing the 3D camera on the end effector of the robot to construct a hand-eye system; Placing a calibration object in the workspace of the robot, and controlling the robot to drive the 3D camera to capture the calibration object at multiple poses to obtain multiple sets of point cloud data of the calibration object and corresponding end effector pose data.

4. The method of claim 1, wherein, The method further comprises the following steps: Determining a hand-eye transformation matrix according to a pre-set hand-eye calibration equation, performing hand-eye calibration on the robot according to the hand-eye transformation matrix to obtain the 3D camera coordinate system, and determining the conversion relationship between the 3D camera coordinate system and the end effector coordinate system.

5. The method of claim 1, wherein, The method further comprises the following steps: Collecting the point cloud data of the target object by the 3D camera, and pre-processing the collected point cloud data, the pre-processing comprising denoising, filtering, and downsampling; Identifying the pre-processed point cloud data to determine the category of the target object; Registering the point cloud data with a pre-established target object pose model to obtain the pose information of the target object in the coordinate system of the 3D camera.

6. The method of claim 1, wherein, The method further comprises the following steps: Calibrating the 3D camera according to a pre-set calibration board to obtain a hand-eye matrix of the 3D camera, the hand-eye matrix comprising an intrinsic matrix and an extrinsic matrix, the intrinsic matrix comprising the focal length and the principal point coordinates of the 3D camera, and the extrinsic matrix comprising the conversion relationship between the 3D camera coordinate system and the end effector coordinate system. The least square method is used to optimize the internal parameter matrix and the external parameter matrix, so as to improve the calibration accuracy of the 3D camera.

7. The method of claim 1, wherein, The prediction model is adjusted according to the historical point cloud data, specifically including: The geometric information of the historical point cloud data is analyzed, the prediction model is reconstructed according to the geometric information and the motion law of the target object, and the minimum reconstruction ratio of the prediction model is determined according to the category and the grabbing angle of the target object.

8. The method of claim 6, wherein, The path of the target object is dynamically generated, specifically including: The dynamic grabbing path is generated according to the dynamic mapping relationship, the prediction motion trajectory of the target object is determined according to the dynamic grabbing path, so that the end effector of the robot reaches the corresponding prediction position in advance, so as to realize accurate grabbing operation.

9. A 3D camera based robot hand-eye positioning device, characterized in that, Including: At least one processor; And The memory is in communication connection with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the 3D camera-based robot hand-eye positioning device to perform: Real-time scanning of the scene where the target object is located by the 3D camera to determine the point cloud data of the target object, the point cloud data including three-dimensional coordinate information of each point on the surface of the target object; According to a predetermined prediction model, the motion state of the target object is predicted to obtain a prediction trajectory, the prediction trajectory is matched with the real-time scanned point cloud data to determine a prediction difference, the prediction difference including a distance difference and a shape difference, and the prediction model is adjusted according to the prediction difference; Determine whether the target object is blocked, if the target object is blocked, determine the historical point cloud data of the target object, and adjust the prediction model according to the historical point cloud data; The target object is re-predicted by the adjusted prediction model to obtain a prediction result, a pre-set hand-eye matrix is determined, the prediction result is mapped with the hand-eye matrix to dynamically generate a path of the target object, and the end effector of the robot is controlled to reach the corresponding prediction position according to the path to realize grabbing at the prediction position.

10. A non-transitory computer storage medium storing computer-executable instructions that, when executed, cause a computer to perform: The computer executable instructions are set as: Real-time scanning of the scene where the target object is located by the 3D camera to determine the point cloud data of the target object, the point cloud data including three-dimensional coordinate information of each point on the surface of the target object; According to a predetermined prediction model, the motion state of the target object is predicted to obtain a prediction trajectory, the prediction trajectory is matched with the real-time scanned point cloud data to determine a prediction difference, the prediction difference including a distance difference and a shape difference, and the prediction model is adjusted according to the prediction difference; Determine whether the target object is blocked, if the target object is blocked, determine the historical point cloud data of the target object, and adjust the prediction model according to the historical point cloud data; The target object is re-predicted by the adjusted prediction model to obtain a prediction result, a preset hand-eye matrix is determined, the prediction result is mapped with the hand-eye matrix, a path of the target object is dynamically generated, and an end effector of a robot is controlled to reach a corresponding predicted position according to the path, so that grabbing is realized at the predicted position.

Images