Robotic vision positioning calibration method and system based on dexterous hand haptics
By combining visual and tactile perception and using tactile feedback from a dexterous hand for calibration, the problem of insufficient accuracy caused by camera distortion and changes in lighting in robot visual positioning technology has been solved, achieving high-precision robot visual positioning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUTONG SENSATION CONTROL (BEIJING) TECH CO LTD
- Filing Date
- 2026-04-14
- Publication Date
- 2026-06-23
Smart Images
Figure CN122008256B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robot vision positioning technology, and in particular to a robot vision positioning calibration method and system based on dexterous hand tactile sense. Background Technology
[0002] As a high-degree-of-freedom robotic end effector that mimics the functions of the human hand, the robotic dexterous hand plays an irreplaceable core role in fields such as industrial precision assembly, hazardous environment operations, and especially minimally invasive surgical robots. One of its core functions is to model the spatial information of the task scene through a vision system, thereby achieving precise positioning and operation. Currently, existing robot visual positioning technologies mainly rely on cameras to acquire image data and use image processing algorithms to construct 3D models to achieve perception and understanding of the surrounding environment.
[0003] While this vision-based positioning method can achieve a certain level of accuracy in ideal environments, in practical applications, factors such as camera distortion, calibration errors, and data processing errors lead to significant deviations in image-based spatial information modeling. Furthermore, drastic changes in ambient light and poor lighting conditions can interfere with visual image information, further affecting positioning accuracy. These factors collectively limit the accuracy of robot vision positioning, making it difficult to meet the requirements of high-precision operation and assembly.
[0004] Therefore, there is an urgent need for a method that can effectively improve the accuracy of robot visual positioning in order to overcome the shortcomings of existing technologies. Summary of the Invention
[0005] The purpose of this application is to provide a robot visual positioning calibration method and system based on dexterous hand tactile sense, so as to solve at least one of the above-mentioned technical problems.
[0006] To achieve the above objectives, firstly, this application proposes a robot visual positioning calibration method based on dexterous hand tactile feedback, the method comprising:
[0007] An initial task scene model is constructed based on environmental image data, and the target calibration point and its initial coordinates are determined based on the initial task scene model.
[0008] Based on the target calibration point and the initial coordinates, the target force point and motion calibration strategy of the dexterous hand are determined, and the dexterous hand is controlled to execute the motion calibration strategy.
[0009] When the dexterous hand comes into contact with the target calibration point through the target force point, the actual coordinates of the target calibration point are determined based on the real-time movement coordinates of the dexterous hand;
[0010] Based on the initial coordinates and the actual coordinates, an error correction value is calculated, and the initial task scenario model is corrected based on the error correction value to obtain the real task scenario model;
[0011] Specifically, before determining the actual coordinates of the target calibration point based on the real-time action coordinates of the dexterous hand when the dexterous hand contacts the target force point, the method further includes: acquiring tactile feature information of the target calibration point, wherein the tactile feature information includes that the force point and force area do not change with the touch direction of the dexterous hand when the dexterous hand contacts the target calibration point; when the target force point of the dexterous hand provides a contact signal, determining that the verification action strategy corresponding to the tactile feature information is to change the touch direction of the dexterous hand's fingertips, and controlling the dexterous hand to change the touch direction of the fingertips multiple times; acquiring the tactile signals fed back by the dexterous hand when it changes the touch direction of the fingertips multiple times, and determining that the dexterous hand contacts the target calibration point when the force point and force area of the dexterous hand do not change by analyzing the tactile signals.
[0012] In some implementations, determining the target force points and motion calibration strategy of the dexterous hand based on the target calibration points and the initial coordinates, and controlling the dexterous hand to execute the motion calibration strategy, includes:
[0013] Based on the target calibration point and the initial coordinates, determine the target force point of the dexterous hand and the initial calibration action;
[0014] Control the dexterous hand to perform the preliminary calibration action, and determine the actual force point when the dexterous hand provides a contact signal;
[0015] Based on the target force point and the actual force point, determine the positional offset of the dexterous hand;
[0016] Based on the position offset, the target calibration action of the dexterous hand is determined, and the dexterous hand is controlled to perform the target calibration action so that the dexterous hand contacts the target calibration point through the target force point.
[0017] In some implementations, there are multiple target calibration points, and the calculation of the error correction value based on the initial coordinates and the actual coordinates includes:
[0018] A set of point pairs is formed based on the initial coordinates and corresponding actual coordinates of each target calibration point;
[0019] The set of point pairs is used as input to a preset spatial transformation model to obtain the error correction value.
[0020] In some embodiments, the dexterous hand includes multiple fingers, each finger having a target force-bearing point. The step of determining the actual coordinates of the target calibration point based on the real-time movement coordinates of the dexterous hand when the dexterous hand contacts the target calibration point through the target force-bearing point includes:
[0021] When each finger of the dexterous hand comes into contact with the target calibration point through the corresponding target force point, the actual coordinates of the target calibration point are determined based on the real-time movement coordinates of each finger.
[0022] The calculation of the error correction value based on the initial coordinates and the actual coordinates includes:
[0023] Based on the initial coordinates and the actual coordinates corresponding to each finger, a single correction value is calculated for each finger.
[0024] The error correction value is calculated based on the individual correction value corresponding to each finger.
[0025] In some implementations, before acquiring an initial task scene model constructed based on environmental image data, and determining the target calibration point and its initial coordinates based on the initial task scene model, the method further includes:
[0026] The environmental image data collected by the binocular camera is acquired, and the environmental image data is corrected based on the correction parameters obtained from the pre-test.
[0027] A disparity map is obtained by performing stereo matching on the corrected environmental image data using a semi-global block matching algorithm.
[0028] The depth information of each pixel in the disparity map is obtained, and each pixel is back-projected into three-dimensional space based on the depth information of each pixel to form a three-dimensional point cloud.
[0029] The three-dimensional point cloud is connected into a three-dimensional mesh, and the color information of the environmental image data is mapped onto the three-dimensional mesh to generate an initial task scene model.
[0030] In some embodiments, the method further includes:
[0031] Upon receiving an operation task, obtain the operation scenario model corresponding to the operation task;
[0032] Based on the operation scenario model, the initial operation coordinates of the operation target in the operation task are determined;
[0033] Based on the error correction value and the initial operation coordinates, the actual operation coordinates are determined, and the operation task is executed based on the actual operation coordinates.
[0034] Secondly, to achieve the above objectives, this application also proposes a robot visual positioning calibration system based on dexterous hand tactile sense, the system comprising: a robot controller for acquiring an initial task scene model;
[0035] The motion control module is used to determine the target calibration point, the initial coordinates of the target calibration point, the target force point of the dexterous hand, and the motion calibration strategy based on the initial task scenario model.
[0036] A dexterous hand controller for controlling a dexterous hand to execute the motion calibration strategy, wherein the dexterous hand is equipped with a tactile sensor;
[0037] A tactile data processing module is used to analyze the tactile signals fed back by the tactile sensor. When it is determined based on the tactile signals that the dexterous hand is in contact with the target calibration point through the target force point, the module determines the actual coordinates of the target calibration point based on the real-time action coordinates of the dexterous hand, and calculates an error correction value based on the initial coordinates and the actual coordinates. Specifically, it acquires tactile feature information of the target calibration point, including that the force point and force area do not change with the touch direction of the dexterous hand when it contacts the target calibration point. When the target force point of the dexterous hand feeds back a contact signal, it determines that the verification action strategy corresponding to the tactile feature information is to change the fingertip touch direction of the dexterous hand and controls the dexterous hand to change the fingertip touch direction multiple times. It acquires the tactile signals fed back by the dexterous hand when it changes the fingertip touch direction multiple times, and determines that the dexterous hand is in contact with the target calibration point through the target force point when the force point and force area of the dexterous hand do not change by analyzing the tactile signals.
[0038] A robot controller is used to correct the initial task scenario model based on the error correction value to obtain a real task scenario model.
[0039] In some embodiments, the system further includes:
[0040] The task module is used to receive visual positioning calibration tasks;
[0041] A robot controller is used to control the binocular cameras to acquire environmental image data;
[0042] The visual information control module is used to construct an initial task scene model based on the environmental image data.
[0043] Compared with the prior art, the beneficial effects of this application include:
[0044] By combining visual and tactile perception, the robot can use tactile feedback to calibrate itself when faced with interference factors such as changes in ambient light and camera distortion, thereby improving the robot's visual positioning accuracy. Attached Figure Description
[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.
[0046] Figure 1 This is a flowchart illustrating a robot visual positioning calibration method based on dexterous hand tactile feedback in one embodiment;
[0047] Figure 2 This is a global schematic diagram of a dexterous hand touching target calibration point A in different touch directions in one embodiment;
[0048] Figure 3 This is a schematic diagram of the target force points when a dexterous hand touches target calibration point A in different touch directions in one embodiment;
[0049] Figure 4 This is a detailed flowchart illustrating how, in one embodiment, the target force points and motion calibration strategies of the dexterous hand are determined based on the target calibration points and the initial coordinates, and the dexterous hand is controlled to execute the motion calibration strategies.
[0050] Figure 5 This is a schematic diagram of the actual force point when a dexterous hand touches target calibration point A in one embodiment;
[0051] Figure 6 This is a schematic diagram showing the actual position of the target calibration point when it is in contact with the target force point in one embodiment.
[0052] Figure 7 This is a partial flowchart of a robot visual positioning calibration method based on dexterous hand tactile feedback in one embodiment;
[0053] Figure 8 This is a schematic diagram of the target force points when a dexterous hand touches target calibration point B in different touch directions in one embodiment;
[0054] Figure 9 This is a partial flowchart of a robot visual positioning calibration method based on dexterous hand tactile feedback in another embodiment;
[0055] Figure 10 This is a flowchart illustrating a robot visual positioning calibration method based on dexterous hand tactile feedback in another embodiment;
[0056] Figure 11 This is a schematic diagram illustrating the execution of an operation task in one embodiment;
[0057] Figure 12 This is a schematic diagram of the functional modules of a robot vision positioning and calibration system based on dexterous hand tactile feedback in one embodiment. Detailed Implementation
[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0059] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.
[0060] For example, the terms "first," "second," etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.
[0061] For example, the terms "comprising" or "including" used in this application indicate the presence of features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.
[0062] As mentioned earlier, while existing robot visual positioning technologies can achieve a certain level of positioning accuracy under ideal conditions, in practical applications, factors such as camera distortion, calibration errors, and data processing errors lead to significant deviations in image-based spatial information modeling. Furthermore, drastic changes in ambient light and poor lighting conditions can interfere with visual image information, further affecting positioning accuracy. These factors collectively limit the accuracy of robot visual positioning, making it difficult to meet the requirements of high-precision operation and assembly. Therefore, there is an urgent need for a method to effectively improve the accuracy of robot visual positioning to overcome the shortcomings of existing technologies. To this end, this application proposes a robot visual positioning calibration method and system based on dexterous hand tactile feedback, which can perform calibration through tactile feedback, thereby improving the accuracy of robot visual positioning.
[0063] like Figure 1 As shown in the figure, this application provides a robot visual positioning calibration method based on dexterous hand tactile feedback, the method including the following steps:
[0064] Step S10: Obtain an initial task scene model constructed based on environmental image data, and determine the target calibration point and its initial coordinates based on the initial task scene model.
[0065] In this embodiment, environmental image data refers to the raw image information of the surrounding environment collected by the robot's vision system (such as a binocular camera), including left and right eye images, used to construct a 3D model of the task scene, i.e., the initial task scene model. The initial task scene model is a 3D model generated based on the environmental image data through algorithmic processing, representing the spatial structure of the robot's task scene. However, due to visual errors (such as camera distortion and lighting effects), this model may have positional deviations.
[0066] Target calibration points refer to one or more three-dimensional geometric feature points selected in the initial task scenario model for tactile spatial calibration. These points possess distinct tactile characteristics, facilitating accurate identification by the dexterous hand using tactile sensors. These tactile characteristics can manifest as predictable and stable macroscopic spatial geometric invariance, meaning the spatial location of the target calibration point is unique and definite in the real physical world. When the dexterous hand touches the point in different directions or postures, although the vector direction of the contact force changes, the position of the force center point sensed by the force sensor remains unchanged in three-dimensional space. Alternatively, the target calibration points can possess identifiable microscopic tactile signatures. When the dexterous hand comes into contact with the target calibration point, the tactile sensor on the dexterous hand can generate a unique and easily identifiable feedback pattern, such as a concentrated force area (the pressure distribution detected by the tactile sensor appears as a concentrated high point, rather than a dispersed surface) or a stable contact pattern (the geometry of the contact surface, such as circular, elliptical, or linear, remains stable or exhibits predictable regular changes at different touch angles).
[0067] Example 1: The target calibration point can be a corner point of a polyhedron (such as a prism, cube, pyramid model, etc.). Specifically, it can be like this: Figure 2 As shown, target calibration point A is selected at one corner of the cube. When a dexterous hand touches target calibration point A in different directions, the tactile signals fed back by the tactile sensors at its fingertips exhibit a highly concentrated point of application, with a force center. Furthermore, the location and area of this force center do not change with the direction of the finger's touch. The initial coordinates refer to the coordinates (x, y, z) of the target calibration point in the initial task scenario model.
[0068] Example 2: The target calibration point can be the center point of a regular protrusion or depression, such as the apex of a hemispherical protrusion, the center of the end face of a small cylinder (such as a pin), or the center of the bottom of a conical depression. Their central axes are geometrically well-defined; for example, the apex of a hemispherical protrusion is its highest point, and its position is unique. They all possess distinct microscopic tactile signatures. For instance, when a dexterous hand contacts a hemispherical protrusion, the tactile sensor generates a small circular pressure zone. When the finger slides over the apex, the center of force stabilizes at the apex. When a dexterous hand contacts the end face of a cylinder, the tactile sensor generates a small circular pressure zone. When the fingertip of a dexterous hand probes the bottom of a conical depression, the force converges at the center of the bottom, and the contact pattern is constrained by the geometry of the depression, exhibiting clear characteristics.
[0069] Step S20: Based on the target calibration point and the initial coordinates, determine the target force point and motion calibration strategy of the dexterous hand, and control the dexterous hand to execute the motion calibration strategy.
[0070] In this embodiment, the target force point refers to a predefined contact area on the dexterous hand (e.g., ...). Figure 3 The N point (located at the center of the fingertip) is used to contact the target calibration point. The motion calibration strategy refers to the plan for controlling the movement of the dexterous hand, including preliminary calibration actions (coarse positioning) and target calibration actions (fine adjustment) to ensure that the target force point accurately contacts the target calibration point.
[0071] In some implementations, such as Figure 4 As shown, step S20 includes:
[0072] Step S21: Based on the target calibration point and the initial coordinates, determine the target force point of the dexterous hand and the preliminary calibration action.
[0073] In this embodiment, the preliminary calibration action is a coarse motion strategy that rapidly moves the target force point to its initial coordinates.
[0074] Specifically, based on the geometric characteristics of the target calibration point (such as protruding surfaces, corners, etc.), a certain sensing point can be selected as the target force point on the dexterous hand equipped with a tactile sensor. The robot motion planner uses the initial coordinates (x, y, z) of the target calibration point as the target point and calculates the initial calibration action of the dexterous hand moving from the current position to the target point.
[0075] Step S22: Control the dexterous hand to perform the preliminary calibration action, and determine the actual force point when the dexterous hand provides a contact signal.
[0076] In this embodiment, the contact signal refers to the electrical signal generated by the tactile sensor on the dexterous hand, indicating that it has made physical contact with an external object. A pressure threshold can be preset, which is used to trigger the contact signal when the force detected by the tactile sensor exceeds the pressure threshold. The actual force point refers to the sensing point on the dexterous hand that actually makes contact with the object when the contact signal is triggered. Figure 5 This diagram illustrates the distribution of various sensor units on the dexterous hand's tactile sensor when the dexterous hand provides a contact signal. The gray dot at the center of the fingertip represents the pre-set target force point. The solid black dot represents the actual force point. It can be seen that there is an offset P between the actual force point and the target force point.
[0077] Step S23: Determine the position offset of the dexterous hand based on the target force point and the actual force point.
[0078] In this embodiment, the position offset refers to the spatial coordinates of the target force point. Spatial coordinates of the actual stress point Spatial positional deviation, i.e., positional offset .
[0079] Step S24: Based on the position offset, determine the target calibration action of the dexterous hand, and control the dexterous hand to perform the target calibration action so that the dexterous hand contacts the target calibration point through the target force point.
[0080] In this embodiment, the target calibration action refers to the fine adjustment motion command generated to compensate for the position offset. For example, the dexterous hand can be directly commanded to move a distance of |P| in the opposite direction of the position offset P, so that the actual force point coincides with the target force point.
[0081] Step S30: When the dexterous hand comes into contact with the target calibration point through the target force point, the actual coordinates of the target calibration point are determined based on the real-time movement coordinates of the dexterous hand.
[0082] In this embodiment, real-time motion coordinates refer to the real-time three-dimensional coordinates of the target force point when the dexterous hand comes into contact with the target calibration point. Since the target force point and the target calibration point are in contact, the actual coordinates of the target calibration point can be determined by the real-time three-dimensional coordinates of the target force point.
[0083] Figure 6The figure formed by the dashed lines is a cube determined by the initial task scene model, with one corner of it being the target force point A, and the initial coordinates being (x, y, z). The figure formed by the solid lines is the actual placement position of the cube determined by tactile perception, with one corner of it being the actual force point B, and the actual coordinates being (x', y', z').
[0084] Step S40: Based on the initial coordinates and the actual coordinates, calculate the error correction value, and correct the initial task scenario model based on the error correction value to obtain the real task scenario model.
[0085] In this embodiment, the error correction value refers to the difference between the initial coordinates and the actual coordinates, which can be represented as (Δx, Δy, Δz), where Δx = x' - x, Δy = y' - y, and Δz = z' - z. Applying the error correction value to the initial task scenario model corrects all coordinate points in the model, and the corrected model becomes the real task scenario model.
[0086] Specifically, each point in the initial task scenario model ( , , ) transform into ( +Δx, +Δy, +Δz) can obtain a model of the real task scenario.
[0087] The robot visual positioning calibration method based on dexterous hand tactile feedback proposed in this application combines visual and tactile perception, enabling the robot to perform calibration through tactile feedback when faced with interference factors such as changes in ambient light and camera distortion, thereby improving the robot's visual positioning accuracy.
[0088] In one embodiment, such as Figure 7 As shown, the procedure before step S30 also includes:
[0089] Step A10: Obtain the tactile feature information of the target calibration point.
[0090] In this embodiment, tactile feature information refers to the inherent physical attributes of the target calibration point that can be perceived by the tactile sensor and distinguished from other points.
[0091] For example, when the target calibration point is an ideal, sharp corner, the point of contact and the area of contact remain unchanged regardless of the direction of the hand's touch. The direction of contact refers to the spatial orientation of the central axis of the dexterity's end (such as the fingertip) relative to the normal to the surface of the target calibration point when it is at the point of contact. Changing the direction of contact means adjusting the "angle of entry" of the finger while keeping the contact point essentially unchanged.
[0092] For example, when the target calibration point is an ideal, circular raised plane, the force-bearing area is an ideal circle when the dexterous hand contacts the target calibration point at an angle parallel to the raised plane.
[0093] Step A20: When the target force point of the dexterous hand provides a feedback contact signal, determine the verification action strategy corresponding to the tactile feature information, and control the dexterous hand to execute the verification action strategy.
[0094] In this embodiment, the verification action strategy refers to a series of active, minute robot motion commands designed to stimulate and observe corresponding tactile feature information. Its purpose is not to move the robot's position, but to detect the geometric properties of the contact point by changing its contact posture.
[0095] For example, if the tactile feature information indicates that the point of force application and the area of force application do not change with the direction of the dexterity hand's touch when the dexterity hand contacts the target calibration point, the verification action strategy can be to change the direction of the dexterity hand's fingertips and control the dexterity hand to change the direction of the fingertips' touch multiple times. Specifically, this verification action strategy can be implemented by instructing the dexterity hand's fingertips to make multiple small-amplitude directional changes around the current contact point while maintaining a slight contact force, such as sliding left and right or exploring in circles.
[0096] Step A30: Obtain the tactile signal fed back by the dexterous hand when executing the verification action strategy; when the tactile signal matches the tactile feature information, determine that the dexterous hand is in contact with the target calibration point through the target force point.
[0097] In this embodiment, by acquiring and analyzing the tactile signals fed back by the dexterous hand when executing the verification action strategy, the force change state of the dexterous hand can be determined. When the force change state matches the corresponding tactile feature information, it is determined that the dexterous hand is in contact with the target calibration point through the target force point.
[0098] For example, when the tactile feature information indicates that the force-bearing point and area do not change with the touch direction of the dexterous hand when it contacts the target calibration point, the tactile signals fed back by the dexterous hand when it changes the touch direction of its fingertips multiple times can be acquired. By analyzing the tactile signals, if it is determined that the force-bearing point and area of the dexterous hand have not changed, it can be determined that the dexterous hand is in contact with the target calibration point through the target force-bearing point. Figure 6 and Figure 8 As shown, when the dexterous hand touches point B in multiple touch directions (touch directions 1, 2, and 3), the point of force application and the area of force application do not change. Therefore, point B is the target calibration point.
[0099] It should be noted that, in this embodiment, "no change" means that the fluctuation range of the detected force point position and the force area is less than a preset fluctuation threshold, so as to eliminate sensor noise and minor control errors.
[0100] The robot vision positioning calibration method based on dexterous hand tactile feedback proposed in this application introduces tactile feature information and executes corresponding verification action strategies to confirm whether the dexterous hand has accurately and stably contacted the target calibration point. This effectively prevents calibration errors caused by accidental contact (such as encountering non-target inclined planes, curved surfaces, or debris) and ensures the reliability of the calibration point.
[0101] In one embodiment, there are multiple target calibration points, which refer to two or more feature points selected in the initial task scenario model and distributed in space.
[0102] In some implementations, these multiple target calibration points should cover the robot's intended operating space as much as possible and should not be on the same straight line to provide sufficient geometric constraints. For example... Figure 2 The four corner points of the cube shown can all be selected as target calibration points.
[0103] Based on multiple target calibration points, step S40 calculates the error correction value based on the initial coordinates and the actual coordinates, including: forming a set of point pairs based on the initial coordinates and corresponding actual coordinates of each target calibration point. The set of point pairs is then used as input to a preset spatial transformation model to solve for the error correction value. The preset spatial transformation model is a mathematical function or matrix; its input is a set of {initial coordinates, actual coordinates} point pairs formed based on multiple target calibration points; and its output is a set of parameters that optimally describes the specific transformation relationship from the initial space to the real space, i.e., the error correction value.
[0104] As an optional implementation, the preset spatial transformation model can be a rigid body transformation matrix, which contains a rotation matrix and Translation vector The goal is to minimize the sum of squared errors between the transformed coordinates and the true coordinates of all points. The resulting [R|T] matrix is the final error correction value. For any point I in the model, its corrected coordinates... .
[0105] As another optional implementation, to account for potential scaling and cropping in the initial task scene model, at least four of the multiple target calibration points exist, and these multiple target calibration points are not coplanar. The preset spatial transformation model can be an affine transformation model, containing a... linear transformation matrix and Translation vector A system of linear equations is constructed based on a set of point pairs, and written as follows: The form is given. Solve using the least squares method: Then, the vector B is recombined into a matrix. sum vector .
[0106] Where X is the design matrix, which contains the coefficients of all initial coordinates, and is a ( The matrix (n is the number of calibration points). Y corresponds to the observation vector, which contains the values of all actual coordinates and is a... B is a column vector. B is a parameter vector containing the unknowns to be solved (i.e., the transformation matrix). Translation vector (all elements), is Column vectors.
[0107] In the robot visual positioning calibration method based on dexterous hand tactile feedback proposed in this application, firstly, by selecting multiple target calibration points and ensuring they cover the robot's expected operating space as much as possible, and are not on the same straight line, sufficient geometric constraints can be provided. This layout ensures that visual positioning errors are corrected from different angles and positions, avoiding the problems of error accumulation and deviation amplification that may be caused by a single calibration point.
[0108] Secondly, by forming a set of point pairs between the initial and actual coordinates of these target calibration points and using this set as input to a pre-defined spatial transformation model, the optimal error correction value can be solved. This method not only considers the error of a single point but also integrates the relative positional relationships between multiple points, thus providing a globally optimal error correction scheme. Compared to methods relying solely on single-point calibration, multi-point calibration can more accurately reflect the error distribution throughout the entire operating space, improving the comprehensiveness and accuracy of the calibration.
[0109] Thirdly, through multi-point calibration, even if some target calibration points are disturbed, other target calibration points can still provide enough information to correct the error, thereby ensuring the stability and reliability of the entire system in complex environments.
[0110] In one embodiment, the dexterous hand includes multiple fingers, each finger having a target force-bearing point. Step S30 includes:
[0111] When each finger of the dexterous hand comes into contact with the target calibration point through the corresponding target force point, the actual coordinates of the target calibration point are determined based on the real-time movement coordinates of each finger.
[0112] Specifically, each finger of the dexterous hand can be individually controlled to contact the same target calibration point through its own target force point. When the tactile sensor of a finger provides a stable contact signal and confirms that it is in contact with the target calibration point through its target force point, the real-time motion coordinates of that finger's target force point at that moment are recorded. These recorded real-time motion coordinates are the actual coordinates of the target calibration point measured based on that finger. Therefore, for N fingers, N actual coordinate measurements will be obtained.
[0113] Step S40 calculates the error correction value based on the initial coordinates and the actual coordinates, including: calculating a single correction value for each finger based on the initial coordinates and the actual coordinates calculated for each finger respectively; based on the single correction value for each finger, the error correction value can be calculated by arithmetic mean, weighted mean, outlier removal and averaging method or data fusion algorithm based on Kalman filter, etc.
[0114] In the robot visual positioning calibration method based on dexterous hand tactile feedback proposed in this application, firstly, in actual operation, each finger of the dexterous hand may be subject to different environmental disturbances, such as mechanical errors of the fingers, noise of the tactile sensor, etc. By using multi-point measurement, even if the measurement values of some fingers are disturbed, the measurement values of other fingers can still provide accurate information, thereby effectively enhancing the robustness of the system and ensuring that reliable error correction values can still be obtained even when some data is inaccurate.
[0115] In one embodiment, such as Figure 9 As shown, the procedure before step S10 also includes:
[0116] Step S01: Acquire environmental image data captured by the binocular camera, and correct the environmental image data based on the correction parameters obtained from the pre-test.
[0117] In this embodiment, a binocular camera refers to two cameras that maintain a fixed baseline distance in the horizontal direction, used to simulate the human eye, synchronously acquiring images from slightly different perspectives to obtain depth information. Environmental image data refers to the image pairs synchronously acquired by the left and right cameras. The calibration parameters obtained through pre-testing refer to the internal parameters (focal length, principal point, distortion coefficients) and external parameters (rotation and translation relationship between the two cameras) obtained by calibrating the binocular camera beforehand (usually using a calibration board such as a checkerboard).
[0118] By utilizing the correction parameters obtained from pre-testing, environmental image data is corrected to a certain extent to eliminate lens distortion and project the image pairs into an ideal state where they are coplanar and aligned.
[0119] Step S02: Perform stereo matching on the corrected environmental image data using a semi-global block matching algorithm to obtain a disparity map.
[0120] In this embodiment, the semi-global block matching algorithm is a stereo matching algorithm. It calculates pixel matching costs using mutual information or the Birchfield-Tomasi (BT) cost function, combines this with one-dimensional dynamic programming for cost aggregation, and optimizes disparity results through various constraints (such as uniqueness and left-right consistency). This approach achieves a denser and more accurate disparity map while maintaining efficiency. A disparity map is a grayscale image of the same size as the original environmental image data, where the grayscale value of each pixel represents the difference in horizontal coordinates (i.e., disparity) between the corresponding points in the left and right images. For each pixel in the disparity map, depth information = (focal length × baseline distance) / disparity.
[0121] Step S03: Obtain the depth information of each pixel in the disparity map, and back-project each pixel into three-dimensional space based on the depth information of each pixel to form a three-dimensional point cloud.
[0122] Step S04: Connect the three-dimensional point cloud into a three-dimensional mesh, and map the color information of the environmental image data onto the three-dimensional mesh to generate an initial task scene model.
[0123] In some implementations, the Delaunay triangulation algorithm can be applied to the 3D point cloud to connect adjacent points and form a 3D mesh. Based on the correspondence between the 3D mesh vertices and the pixels of the (left or right eye corrected) environmental image data, the pixel colors of the (left or right eye corrected) environmental image data are assigned to the corresponding mesh surfaces to generate an initial task scene model.
[0124] In the robot vision positioning and calibration method based on dexterous hand tactile feedback proposed in this application, the initial task scene model generated through the above preprocessing and modeling steps has high accuracy and high reliability, providing more accurate basic data for subsequent positioning calibration. This makes the real task scene model obtained by correcting the initial task scene model with error correction values closer to the real scene, significantly improving the accuracy and reliability of robot vision positioning.
[0125] In one embodiment, such as Figure 10 As shown, the method further includes:
[0126] Step S50: Upon receiving an operation task, obtain the operation scenario model corresponding to the operation task.
[0127] In this embodiment, the operation task refers to the specific work instructions that the robot needs to execute, such as... Figure 11 As shown, the gripped part is inserted into point C of the assembly. The operation scenario model refers to a three-dimensional model containing the operation target and its surrounding environment, which the robot builds in real-time or near real-time, or retrieves from memory, to perform the operation task.
[0128] Step S60: Based on the operation scenario model, determine the initial operation coordinates of the operation target in the operation task.
[0129] In this embodiment, the operation target refers to the specific object or part that needs to be operated in the operation task, such as... Figure 11 The center point C of the hole to be inserted in the assembly shown is an example. Initial operation coordinates refer to the three-dimensional coordinates of the operation target in the operation scene model, such as... Figure 11 The coordinates of point C are shown.
[0130] Step S70: Based on the error correction value and the initial operation coordinates, determine the actual operation coordinates, and execute the operation task based on the actual operation coordinates.
[0131] In this embodiment, the error correction value is obtained from the calibration process of the aforementioned embodiment. The parameters used to correct the system error of the visual model can be represented as (Δx, Δy, Δz). The error correction value is applied to the initial operating coordinates C(…). , , Transformed into actual operational coordinates C' ( +Δx, +Δy, +Δz). This enables the robot motion controller to perform path planning and inverse kinematics calculations based on the actual operating coordinates, and drive the joint motors to ultimately complete the actual operation task.
[0132] In the robot vision positioning calibration method based on dexterous hand tactile sense proposed in this application embodiment, the initial operation coordinates are corrected by introducing error correction values, which can significantly improve the true position accuracy of the operation target, thereby improving the accuracy and reliability of the operation.
[0133] like Figure 12 As shown in the figure, this application embodiment also provides a robot vision positioning calibration system based on dexterous hand tactile feedback. The system includes a robot controller 10, a motion control module 20, a dexterous hand equipped with a tactile sensor, a dexterous hand controller 30, and a tactile data processing module 40.
[0134] Robot controller 10 is used to acquire the initial task scene model;
[0135] The motion control module 20 is used to determine the target calibration point, the initial coordinates of the target calibration point, the target force point of the dexterous hand, and the motion calibration strategy based on the initial task scenario model.
[0136] A dexterous hand controller 30 is used to control the dexterous hand to execute the motion calibration strategy, the dexterous hand being equipped with a tactile sensor;
[0137] The tactile data processing module 40 is used to analyze the tactile signals fed back by the tactile sensor. When it is determined from the tactile signals that the dexterous hand is in contact with the target calibration point through the target force point, the actual coordinates of the target calibration point are determined from the real-time movement coordinates of the dexterous hand. Based on the initial coordinates and the actual coordinates, an error correction value is calculated.
[0138] The robot controller 10 is also used to correct the initial task scenario model based on the error correction value to obtain a real task scenario model.
[0139] In this embodiment, the robot controller 10 is the central processing unit of the system. It can be an industrial computer running a robot operating system (such as ROS) or a high-performance embedded controller, responsible for task scheduling, data integration, decision-making, and routing information between various functional modules. The motion control module 20 is a dedicated software module or hardware controller for robot motion planning and execution, used to convert target coordinates and postures into specific joint motion commands. The end effector of the dexterous hand robot has multiple independently actuated fingers, with three-dimensional force tactile sensors integrated at the fingertips or finger surfaces, capable of measuring three-dimensional force / torque and pressure distribution. The dexterous hand controller 30 is a dedicated controller for controlling the movement of multiple fingers of the dexterous hand, used to receive advanced gesture or coordinate commands and decode them into detailed control signals for each joint of each finger. The tactile data processing module 40 is a software module for processing, analyzing, and interpreting the signals measured by the tactile sensors.
[0140] In some embodiments, the system further includes a task module 50, a binocular camera 60, and a visual information control module 70.
[0141] Task module 50 is used to receive visual positioning calibration tasks;
[0142] A binocular camera 60 is used to collect environmental image data;
[0143] The visual information control module 70 is used to construct an initial task scene model based on the environmental image data.
[0144] In this embodiment, the task module 50 is a software module that provides a human-computer interaction or task scheduling interface, used to receive, parse, and manage high-level task instructions issued by the user or upper-level system. The binocular cameras 60 are a pair of precisely calibrated and synchronized cameras used to acquire environmental image data from two slightly different perspectives to reconstruct the 3D environment. The visual information control module 70 is a software module used to process the environmental image data and execute 3D reconstruction algorithms.
[0145] In some embodiments, the motion control module 20 is used to determine the target force point and preliminary calibration action of the dexterous hand based on the target calibration point and the initial coordinates, and send the preliminary calibration action to the dexterous hand controller 30; the dexterous hand controller 30 is used to control the dexterous hand to perform the preliminary calibration action, and when the tactile sensor provides a contact signal, determine the actual force point and send the actual point to the motion control module 20; the motion control module 20 is used to determine the position offset of the dexterous hand based on the target force point and the actual force point; the motion control module 20 is also used to determine the target calibration action of the dexterous hand based on the position offset, and send the target calibration action to the dexterous hand controller 30; the dexterous hand controller 30 is used to control the dexterous hand to perform the target calibration action so that the dexterous hand contacts the target calibration point through the target force point.
[0146] In some implementations, the tactile data processing module 40 is used to acquire tactile feature information of the target calibration point; by analyzing the tactile signals fed back by the tactile sensor, when the dexterous hand receives a contact signal at the target force point, it determines the verification action strategy corresponding to the tactile feature information; acquires the tactile signal fed back by the dexterous hand when executing the verification action strategy, and when the tactile signal matches the tactile feature information, it determines that the dexterous hand contacts the target calibration point through the target force point.
[0147] In some embodiments, the tactile feature information includes the fact that when the dexterous hand contacts the target calibration point, the force point and force area do not change with the touch direction of the dexterous hand. The tactile data processing module 40 is used to determine, by analyzing the tactile signals fed back by the tactile sensor, when the dexterous hand provides a contact signal at the target force point, to determine that the verification action strategy corresponding to the tactile feature information is to change the touch direction of the dexterous hand's fingertips; to acquire the tactile signals fed back by the dexterous hand when it changes the touch direction of its fingertips multiple times, and to determine, by analyzing the tactile signals, when the force point and force area of the dexterous hand do not change, to determine that the dexterous hand is in contact with the target calibration point through the target force point.
[0148] In some implementations, there are multiple target calibration points. The tactile data processing module 40 is used to form a set of point pairs based on the initial coordinates and corresponding actual coordinates of each target calibration point. The set of point pairs is used as input to a preset spatial transformation model to solve for the error correction value.
[0149] In some embodiments, the dexterous hand includes multiple fingers, each finger having a target force point. The tactile data processing module 40 is used to determine the actual coordinates of the target calibration point based on the real-time motion coordinates of each finger when the tactile signal determines that each finger of the dexterous hand is in contact with the target calibration point through the corresponding target force point; calculate a single correction value for each finger based on the initial coordinates and the actual coordinates calculated for each finger; and calculate an error correction value based on the single correction value for each finger.
[0150] In some implementations, the task module 50 is used to receive an operation task and send the operation task to the robot controller 10; the robot controller 10 is used to obtain the operation scene model corresponding to the operation task constructed by the visual information control module 70, and is also used to determine the initial operation coordinates of the operation target in the operation task based on the operation scene model, and determine the actual operation coordinates based on the error correction value and the initial operation coordinates, so as to execute the operation task based on the actual operation coordinates.
[0151] In the robot visual positioning and calibration system based on dexterous hand tactile feedback proposed in this application, firstly, the system integrates two perception methods: binocular vision and dexterous hand tactile feedback. Binocular vision provides a wide range of scene information and preliminary positioning, while the dexterous hand's tactile sensors provide precise positioning feedback upon contact with the target. Through the analysis of tactile signals by the tactile data processing module 40, it can accurately determine whether the dexterous hand has truly contacted the target calibration point, and correct the visual positioning error accordingly. This multimodal perception fusion method effectively overcomes the limitations of single-vision positioning in complex environments, significantly improving the accuracy and reliability of positioning calibration.
[0152] Secondly, by breaking down the robot's visual positioning and calibration process into multiple modules and clearly defining the functions and interaction methods of each module, a high degree of modularity and systematization of the system is achieved. This design not only makes the functions of each part of the system clear, easy to maintain and upgrade, but also facilitates flexible configuration and expansion in different application scenarios. For example, hardware modules such as the binocular camera 60 and tactile sensors can be quickly replaced or upgraded, or the algorithms of software modules such as visual information processing and motion control can be adjusted according to specific task requirements.
[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
[0154] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, any of the embodiments or implementations claimed above can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.
Claims
1. A robot visual positioning calibration method based on dexterous hand tactile feedback, characterized in that, The method includes: An initial task scene model is constructed based on environmental image data, and the target calibration point and its initial coordinates are determined based on the initial task scene model. Based on the target calibration point and the initial coordinates, the target force point and motion calibration strategy of the dexterous hand are determined, and the dexterous hand is controlled to execute the motion calibration strategy. When the dexterous hand comes into contact with the target calibration point through the target force point, the actual coordinates of the target calibration point are determined based on the real-time movement coordinates of the dexterous hand; Based on the initial coordinates and the actual coordinates, an error correction value is calculated, and the initial task scenario model is corrected based on the error correction value to obtain the real task scenario model; Specifically, before determining the actual coordinates of the target calibration point based on the real-time action coordinates of the dexterous hand when the dexterous hand contacts the target force point, the method further includes: acquiring tactile feature information of the target calibration point, wherein the tactile feature information includes that the force point and force area do not change with the touch direction of the dexterous hand when the dexterous hand contacts the target calibration point; when the target force point of the dexterous hand provides a contact signal, determining that the verification action strategy corresponding to the tactile feature information is to change the touch direction of the dexterous hand's fingertips, and controlling the dexterous hand to change the touch direction of the fingertips multiple times; acquiring the tactile signals fed back by the dexterous hand when it changes the touch direction of the fingertips multiple times, and determining that the dexterous hand contacts the target calibration point when the force point and force area of the dexterous hand do not change by analyzing the tactile signals.
2. The robot visual positioning calibration method based on dexterous hand tactile feedback as described in claim 1, characterized in that, The process of determining the target force points and motion calibration strategy of the dexterous hand based on the target calibration points and the initial coordinates, and controlling the dexterous hand to execute the motion calibration strategy, includes: Based on the target calibration point and the initial coordinates, determine the target force point of the dexterous hand and the initial calibration action; Control the dexterous hand to perform the preliminary calibration action, and determine the actual force point when the dexterous hand provides a contact signal; Based on the target force point and the actual force point, determine the positional offset of the dexterous hand; Based on the position offset, the target calibration action of the dexterous hand is determined, and the dexterous hand is controlled to perform the target calibration action so that the dexterous hand contacts the target calibration point through the target force point.
3. The robot visual positioning calibration method based on dexterous hand tactile feedback as described in claim 1, characterized in that, There are multiple target calibration points. The error correction value is calculated based on the initial coordinates and the actual coordinates, including: A set of point pairs is formed based on the initial coordinates and corresponding actual coordinates of each target calibration point; The set of point pairs is used as input to a preset spatial transformation model to obtain the error correction value.
4. The robot visual positioning calibration method based on dexterous hand tactile feedback as described in claim 1, characterized in that, The dexterous hand includes multiple fingers, each finger having a target force-bearing point. The step of determining the actual coordinates of the target calibration point based on the real-time motion coordinates of the dexterous hand when the dexterous hand contacts the target calibration point through the target force-bearing point includes: When each finger of the dexterous hand comes into contact with the target calibration point through the corresponding target force point, the actual coordinates of the target calibration point are determined based on the real-time movement coordinates of each finger. The calculation of the error correction value based on the initial coordinates and the actual coordinates includes: Based on the initial coordinates and the actual coordinates corresponding to each finger, a single correction value is calculated for each finger. The error correction value is calculated based on the individual correction value corresponding to each finger.
5. The robot visual positioning calibration method based on dexterous hand tactile feedback as described in claim 1, characterized in that, Before acquiring the initial task scene model constructed based on environmental image data, and determining the target calibration point and its initial coordinates based on the initial task scene model, the method further includes: The environmental image data collected by the binocular camera is acquired, and the environmental image data is corrected based on the correction parameters obtained from the pre-test. A disparity map is obtained by performing stereo matching on the corrected environmental image data using a semi-global block matching algorithm. The depth information of each pixel in the disparity map is obtained, and each pixel is back-projected into three-dimensional space based on the depth information of each pixel to form a three-dimensional point cloud. The three-dimensional point cloud is connected into a three-dimensional mesh, and the color information of the environmental image data is mapped onto the three-dimensional mesh to generate an initial task scene model.
6. The robot visual positioning calibration method based on dexterous hand tactile feedback according to any one of claims 1 to 5, characterized in that, The method further includes: Upon receiving an operation task, obtain the operation scenario model corresponding to the operation task; Based on the operation scenario model, the initial operation coordinates of the operation target in the operation task are determined; Based on the error correction value and the initial operation coordinates, the actual operation coordinates are determined, and the operation task is executed based on the actual operation coordinates.
7. A robot visual positioning calibration system based on dexterous hand tactile feedback, characterized in that, The system includes: The robot controller is used to acquire the initial task scenario model. The motion control module is used to determine the target calibration point, the initial coordinates of the target calibration point, the target force point of the dexterous hand, and the motion calibration strategy based on the initial task scenario model. A dexterous hand controller for controlling a dexterous hand to execute the motion calibration strategy, wherein the dexterous hand is equipped with a tactile sensor; A tactile data processing module is used to analyze the tactile signals fed back by the tactile sensor. When it is determined based on the tactile signals that the dexterous hand is in contact with the target calibration point through the target force point, the module determines the actual coordinates of the target calibration point based on the real-time action coordinates of the dexterous hand, and calculates an error correction value based on the initial coordinates and the actual coordinates. Specifically, it acquires tactile feature information of the target calibration point, including that the force point and force area do not change with the touch direction of the dexterous hand when it contacts the target calibration point. When the target force point of the dexterous hand feeds back a contact signal, it determines that the verification action strategy corresponding to the tactile feature information is to change the fingertip touch direction of the dexterous hand and controls the dexterous hand to change the fingertip touch direction multiple times. It acquires the tactile signals fed back by the dexterous hand when it changes the fingertip touch direction multiple times, and determines that the dexterous hand is in contact with the target calibration point through the target force point when the force point and force area of the dexterous hand do not change by analyzing the tactile signals. The robot controller is also used to correct the initial task scenario model based on the error correction value to obtain a real task scenario model.
8. The robot visual positioning calibration system based on dexterous hand tactile feedback as described in claim 7, characterized in that, The system also includes: The task module is used to receive visual positioning calibration tasks; A binocular camera is used to collect environmental image data; The visual information control module is used to construct an initial task scene model based on the environmental image data.