Calibration method and device of finger-type tactile sensor and server

Abstract

The application provides a kind of calibration method, device and server of finger type visual tactile sensor, it is related to the technical field of sensor calibration, comprising: obtaining first contact image and second contact image, and based on initial camera parameter, simulation processing is carried out to finger type calibration sleeve and first contact image, to determine target camera parameter;Based on target camera parameter, simulation processing is carried out to calibration ball and second contact image, to determine the target normal vector set corresponding to the contact area of calibration ball and elastomer;The boundary depth of elastomer model is obtained, and based on target normal vector set and boundary depth, depth reconstruction processing is carried out to contact area, to obtain the pixel depth value of each pixel point;The pixel depth value of each pixel point is used to carry out model training processing to initial mechanical model, to obtain target mechanical model, to use target mechanical model for mechanical analysis when finger type visual tactile sensor executes grabbing operation.The application can significantly improve the adaptation ability and accuracy of sensor.

Description

Technical Field

[0001] This invention relates to the technical field of sensor calibration, and in particular to a calibration method, apparatus and server for a finger-shaped visual tactile sensor. Background Technology

[0002] To enable robots to manipulate various tools and objects with the same dexterity and precision as humans, they need to be equipped with powerful perception systems, among which tactile sensing is the most important component. Currently, related technologies suggest that to enable robots to achieve high-precision positioning of small objects, the tactile sensors on their fingertips typically employ visual-tactile sensors. Existing visual-tactile technologies are mainly based on the principle of RGB three-color lamp illumination, achieving perception by establishing a mapping relationship between light intensity and gradient. However, this technology has a high barrier to entry, is difficult to manufacture, and the sensor's mechanical prediction accuracy is unstable, thus limiting the flexibility and precision of its grasping operations. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a calibration method, apparatus and server for a finger-shaped visual tactile sensor, which can significantly improve the sensor's adaptability and accuracy.

[0004] In a first aspect, embodiments of the present invention provide a calibration method for a finger-shaped visual tactile sensor. The method includes: acquiring a first contact image and a second contact image between an elastomer in the finger-shaped visual tactile sensor and a finger-shaped calibration sleeve and a calibration ball, respectively; performing simulation processing on the finger-shaped calibration sleeve and the first contact image based on initial camera parameters to determine target camera parameters; performing simulation processing on the calibration ball and the second contact image based on the target camera parameters to determine the target position of the calibration ball and the target normal vector set corresponding to the contact area between the calibration ball and the elastomer; acquiring the boundary depth of the elastomer model; performing depth reconstruction processing on the contact area of ​​the elastomer based on the target normal vector set and the boundary depth to obtain the pixel depth value corresponding to each pixel point in the contact area; and using the pixel depth value corresponding to each pixel point to perform model training processing on the initial mechanical model to obtain a target mechanical model, so as to perform mechanical analysis using the target mechanical model when the finger-shaped visual tactile sensor performs a grasping operation.

[0005] In one implementation, the step of determining the target camera parameters by simulating the finger calibration sleeve and the first contact image based on initial camera parameters includes: using the initial camera parameters, performing model rendering processing on the finger calibration sleeve in a simulation environment to generate a three-dimensional model of the finger calibration sleeve; if the three-dimensional model of the finger calibration sleeve coincides with the first contact image, then the camera parameters at this time are determined as the target camera parameters.

[0006] In one embodiment, the steps of performing simulation processing on the calibration sphere and the second contact image based on the target camera parameters to determine the target position of the calibration sphere and the target normal vector set corresponding to the contact area between the calibration sphere and the elastic body include: using the target camera parameters to perform model rendering processing on the calibration sphere in the simulation environment to generate a three-dimensional model of the calibration sphere; adjusting the coordinate position of the calibration sphere so that the three-dimensional model of the calibration sphere coincides with the second contact image, and determining the coordinates of the calibration sphere when they coincide as the target position, so as to determine the target normal vector set corresponding to the contact area between the calibration sphere and the elastic body based on the target position.

[0007] In one embodiment, the step of obtaining the boundary depth of the elastomer model includes: using target camera parameters, setting a simulation camera in a simulation environment, and acquiring and processing the elastomer model from the perspective of the simulation camera to determine the boundary depth corresponding to the outermost pixel in the contact contour of the elastomer model.

[0008] In one implementation, the step of performing depth reconstruction processing on the contact area of ​​the elastic body based on the target normal vector set and the boundary depth to obtain the pixel depth value corresponding to each pixel in the contact area includes: using the boundary depth as a constraint, using the target normal vector set to perform pixel-by-pixel Poisson integral reconstruction on each pixel in the contact area to obtain the pixel depth value corresponding to each pixel.

[0009] In one implementation, after obtaining the pixel depth value corresponding to each pixel, the method includes: mapping the pixel depth value corresponding to each pixel to a depth map based on the mapping relationship between pixel coordinates.

[0010] In one embodiment, the step of training an initial mechanical model using the pixel depth values ​​corresponding to each pixel to obtain a target mechanical model includes: sending a depth map to the initial mechanical model to obtain the pixel normal force corresponding to each pixel, and training the initial mechanical model based on the pixel normal force and the actual measured force of the finger-shaped visual tactile sensor to obtain the target mechanical model.

[0011] Secondly, embodiments of the present invention also provide a calibration device for a finger-shaped visual-tactile sensor. The device includes: a camera parameter calibration module, which acquires a first contact image and a second contact image between the elastomer in the finger-shaped visual-tactile sensor and a finger-shaped calibration sleeve and a calibration ball, respectively, and performs simulation processing on the finger-shaped calibration sleeve and the first contact image based on initial camera parameters to determine target camera parameters; a normal vector calibration module, which performs simulation processing on the calibration ball and the second contact image based on the target camera parameters to determine the target position of the calibration ball and the target normal vector set corresponding to the contact area between the calibration ball and the elastomer; a depth reconstruction module, which acquires the boundary depth of the elastomer model and performs depth reconstruction processing on the contact area of ​​the elastomer based on the target normal vector set and the boundary depth to obtain the pixel depth value corresponding to each pixel in the contact area; and a model training module, which uses the pixel depth value corresponding to each pixel to perform model training processing on the initial mechanical model to obtain a target mechanical model, so as to perform mechanical analysis using the target mechanical model when the finger-shaped visual-tactile sensor performs a grasping operation.

[0012] Thirdly, embodiments of the present invention also provide a server, including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement any of the methods provided in the first aspect.

[0013] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement any of the methods provided in the first aspect.

[0014] The embodiments of the present invention bring the following beneficial effects:

[0015] This invention provides a calibration method, apparatus, and server for a finger-shaped visual-tactile sensor. The method acquires first and second contact images between the elastomer in the finger-shaped visual-tactile sensor and a finger-shaped calibration sleeve and a calibration ball, respectively. Based on initial camera parameters, it performs simulation processing on the finger-shaped calibration sleeve and the first contact image to determine the target camera parameters. Then, based on the target camera parameters, it performs simulation processing on the calibration ball and the second contact image to determine the target position of the calibration ball and the target normal vector set corresponding to the contact area between the calibration ball and the elastomer. Next, it acquires the boundary depth of the elastomer model and performs depth reconstruction processing on the contact area of ​​the elastomer based on the target normal vector set and the boundary depth to obtain the pixel depth value corresponding to each pixel in the contact area. Finally, it uses the pixel depth values ​​corresponding to each pixel to perform model training processing on the initial mechanical model to obtain the target mechanical model. This target mechanical model is then used for mechanical analysis when the finger-shaped visual-tactile sensor performs a grasping operation. This invention can significantly improve the sensor's adaptability and accuracy.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the structure of a finger-shaped visual-tactile sensor provided in an embodiment of the present invention;

[0020] Figure 2 A schematic flowchart illustrating a calibration method for a finger-shaped visual-tactile sensor provided in an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of a calibration object provided in an embodiment of the present invention;

[0022] Figure 4 This is a flowchart illustrating a camera parameter determination method provided in an embodiment of the present invention.

[0023] Figure 5 A flowchart illustrating a method for determining the truth value of a normal vector according to an embodiment of the present invention;

[0024] Figure 6 This is a flowchart illustrating a pixel depth determination method provided in an embodiment of the present invention.

[0025] Figure 7 A schematic diagram of the structure of a calibration device for a finger-shaped visual-tactile sensor provided in an embodiment of the present invention;

[0026] Figure 8 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Currently, early industrial robots only need to perform simple repetitive tasks such as grasping and carrying, using simple grippers. However, as robots need to enter human living and working environments (such as home services, medical rehabilitation, and space exploration), they must be able to manipulate various tools and objects as flexibly and precisely as humans (such as opening doors with keys, using scissors, and performing surgery). In order to make the robot's hand dexterous, not only is a complex mechanical structure required (usually with multiple joints and fingers, mimicking 15-20 degrees of freedom of the human hand), but also a powerful sensing system is also needed. Among them, tactile sense is the most important component. Giving it tactile sense is of great significance for the robot's learning of operational skills, enabling the robot to adaptively adjust its gripping force and posture.

[0029] Many different tactile sensing and perception systems have been designed both domestically and internationally for robot perception and manipulation tasks, such as object characteristic recognition, slip detection, grasping stability assessment, and dexterous manipulation of robotic arms. Based on the sensor's principle and shape, tactile sensors used on fingertips can be divided into three categories: tactile arrays, bionic fingertips, and visual-tactile sensors. Due to limited spatial resolution, tactile arrays and bionic fingertips struggle to achieve high-precision positioning of small objects; while visual-tactile sensors can obtain high-resolution, sensitive, and high-quality tactile information at a lower manufacturing cost.

[0030] Existing visual-tactile technologies are mainly based on the principle of RGB three-color lamp illumination. They achieve perception by establishing a mapping relationship between light intensity and gradient. This approach is highly dependent on the brightness consistency of the lamp beads and the uniformity of light distribution. Therefore, it not only has extremely high requirements for the selection of lamp beads, but also requires the design of special light guide structures. These two points together increase the technical threshold for manufacturing.

[0031] Furthermore, due to the complex curved surface of finger-type sensors, it is difficult to mathematically model them, and it is even more difficult to distribute the light source evenly on the curved surface. Most mainstream visual-tactile sensors on the market are planar in shape. Although this is beneficial for simplifying the principle and manufacturing, it is in fundamental conflict with the adaptation requirements of bionic dexterous hands to complex curved surfaces, and the mechanical prediction accuracy is unstable, thus limiting the flexibility and precision of its grasping operation. Based on this, the present invention provides a calibration method for finger-type visual-tactile sensors, which can reduce manufacturing difficulty, increase yield rate, and improve mechanical prediction accuracy, thereby providing reliable technical support for the in-depth application of robotics technology.

[0032] To facilitate understanding of this embodiment, a finger-shaped visual-tactile sensor disclosed in this invention will first be described in detail. (See [link to relevant documentation]). Figure 1 The diagram shown illustrates the structure of a finger-shaped visual-tactile sensor. Figure 1 As shown, it includes: structural components, a transparent elastomer, a light source component, and a camera. The principle is as follows: when an object contacts the elastomer, it deforms. The camera captures the contact image, and a normal vector prediction model infers the mechanical normal vectors on each contact pixel. Surface integration is used to obtain the pixel depth point cloud, and a mechanical prediction model infers the mechanical information. Specifically:

[0033] 1. Transparent elastomers are usually made of highly transparent materials such as gels, and their surfaces are covered with a highly reflective coating. In order to improve the optical reflective properties of the outer surface of the elastomer, it is usually prepared by adding silver powder to the gel.

[0034] 2. The structural components are the main parts, serving as the mounting base for the light source, camera, and transparent elastomer.

[0035] 3. The elastomer and structural components are installed in close contact. In most cases, the manufacturing process involves molding the gel and structural components in one step.

[0036] 4. The camera is installed inside a cylindrical cavity, and a circular transparent plate is installed in the cavity to prevent the gel from entering the cavity.

[0037] 5. The light source is usually a red, green, or blue light source.

[0038] 6. The backplate is installed on the structural components and its function is to shield the circuit structure on the back.

[0039] based on Figure 1The schematic diagram of the finger-shaped visual-tactile sensor shown in this invention provides a detailed description of the calibration method for the finger-shaped visual-tactile sensor. (See also...) Figure 2 The diagram shows a calibration method for a finger-type visual tactile sensor, which mainly includes the following steps S202 to S208:

[0040] Step S202: Obtain the first contact image and the second contact image between the elastomer in the finger-shaped visual tactile sensor and the finger-shaped calibration sleeve and the calibration ball, respectively. Based on the initial camera parameters, perform simulation processing on the finger-shaped calibration sleeve and the first contact image to determine the target camera parameters.

[0041] In one embodiment, the finger-shaped calibration sleeve and the calibration ball are different calibration objects. The finger-shaped calibration sleeve is a rigid outer sleeve that can tightly fit the transparent elastomer, and its inner surface has a special raised shape. When it is tightly fitted with the elastomer, the image in the camera at this time is acquired, that is, the first contact image; the complete structure of the calibration ball is shown in [reference needed]. Figure 3 The schematic diagram of a calibration device shown includes: a calibration ball in contact with a transparent elastomer, a six-dimensional force sensor, and a connector.

[0042] Step S204: Based on the target camera parameters, perform simulation processing on the calibration ball and the second contact image to determine the target position of the calibration ball and the target normal vector set corresponding to the contact area between the calibration ball and the elastic body.

[0043] In other words, based on the target camera parameters, a simulated camera model that is completely consistent with the actual shooting perspective can be established in the simulation environment. The three-dimensional geometric model of the calibration sphere is then imported into this environment. By iteratively adjusting the spatial pose of the calibration sphere, the simulated rendered image and the second contact image are made to coincide at the pixel level, thereby determining the target position of the calibration sphere. Subsequently, using the calibrated camera intrinsic and extrinsic parameters and the known surface equation of the calibration sphere, the local normal direction of each pixel in the contact area between the calibration sphere and the elastic body under the coincident state is calculated. These are then summarized to form a target normal vector set, providing the necessary gradient information for subsequent depth reconstruction.

[0044] Step S206: Obtain the boundary depth of the elastic body model, and perform depth reconstruction processing on the contact area of ​​the elastic body based on the target normal vector set and the boundary depth to obtain the pixel depth value corresponding to each pixel in the contact area.

[0045] In other words, firstly, a 3D model of a transparent elastomer with the same dimensions as the actual object is loaded into the simulation environment. Then, using the calibrated target camera parameters, the virtual camera is positioned in the exact same pose as the actual shot. Next, the model is rendered and sampled to directly read the depth values ​​corresponding to the outermost pixels of the contact area contour, forming a boundary depth prior. Then, the contact area of ​​the elastomer is reconstructed based on the target normal vector set and the boundary depth: using the boundary depth as a constraint, the pixel-by-pixel gradient information provided by the target normal vector set is substituted into the discrete Poisson equation. By solving this equation, the entire contact area is reconstructed, thereby calculating the height value of each pixel in the area. Finally, a pixel depth value matrix corresponding to the real shape is obtained, providing input features for subsequent mechanical model training.

[0046] Step S208: Using the pixel depth values ​​corresponding to each pixel point, the initial mechanical model is trained to obtain the target mechanical model, so as to perform mechanical analysis using the target mechanical model when the finger-shaped visual tactile sensor performs the grasping operation.

[0047] In one implementation, the calibration in the above scheme refers to acquiring parameter information of each component within the sensor, as well as acquiring paired data between multiple sets of images, normal vectors, and forces, to provide modeling data for training the normal vector prediction model and the force prediction model. The calibration process includes: 1. Acquiring the camera's intrinsic parameters, extrinsic parameters, and pixel-to-millimeter conversion coefficients, which are necessary in subsequent calibration and prediction. 2. Acquiring multiple sets of image, normal vector, and force data. 3. Using the data sets to train the normal vector prediction model and the force prediction model.

[0048] The calibration method for the finger-shaped visual-tactile sensor provided in this embodiment of the invention enables the finger-shaped visual-tactile sensor to accurately obtain the parameters required for mechanical prediction on complex curved surfaces. Therefore, the finger-shaped visual-tactile sensor can be made into any shape. At the same time, the calibration and prediction method based on neural networks can adaptively fit the mechanical information of different points on uneven light guides and complex curved surfaces. Therefore, there is no need for complex light guide structures, and the error accumulation caused by complex mathematical modeling can be avoided, thereby achieving high-precision mechanical prediction.

[0049] This invention also provides an implementation method for calibrating a finger-shaped visual-tactile sensor, as detailed in (1) to (5) below:

[0050] (1) Obtain the intrinsic and extrinsic parameters of the camera and calculate the pixel-to-millimeter conversion coefficient: Using the initial camera parameters, perform model rendering on the finger calibration sleeve in the simulation environment to generate a three-dimensional model of the finger calibration sleeve. If the three-dimensional model of the finger calibration sleeve coincides with the first contact image, then the camera parameters at this time are determined as the target camera parameters.

[0051] For details, see Figure 4 The flowchart shown illustrates a method for determining camera parameters. In a simulation environment, a contact image is imported and its transparency is set to 60%. In the simulation environment, using the initial camera parameters, a camera and calibration sleeve are imported. The calibration sleeve pattern in the camera's field of view is simulated in the simulation environment and is called the simulation pattern. Then, it is observed whether the simulation pattern coincides with the contact image. If they do not coincide, the camera parameters need to be adjusted until they coincide. At this point, the camera parameters are the correct parameters, i.e., the target camera parameters.

[0052] In one implementation, the pixel-to-millimeter conversion factor (pixmm) is calculated:

[0053]

[0054]

[0055]

[0056] Where (x,y) are pixel coordinates, and 2 represents the distance between two pixels. Since the unit of depth z is mm, the pixel distance needs to be multiplied by pixmm to convert it to mm.

[0057] (2) Obtain image, normal vector and force data set: Using the target camera parameters, the calibration ball is rendered in the simulation environment to generate a three-dimensional model of the calibration ball. Then, by adjusting the coordinate position of the calibration ball, the three-dimensional model of the calibration ball coincides with the second contact image. The coordinates of the calibration ball when they coincide are determined as the target position. Based on the target position, the target normal vector set corresponding to the contact area between the calibration ball and the elastic body is determined.

[0058] For details, see Figure 5 The flowchart illustrates a method for determining the true value of normal vectors. First, the sensor is brought into contact with and pressed against a transparent elastomer, and the contact image and mechanical information obtained by the camera are acquired at this time. Then, the contact image is imported into 3D software, the transparency is set to 60%, and a 3D model of the calibration sphere at the initial position is imported. The camera model is set using the previously determined intrinsic and extrinsic parameters to simulate the calibration sphere in the camera's view. The position of the calibration sphere is adjusted until it coincides with the contact image. At this point, the true values ​​of all normal vectors within the contact range between the calibration sphere and the elastomer are determined. Finally, the contact image, true values ​​of normal vectors, and mechanical information obtained in the above steps are combined to form a calibration data set consisting of an image, normal vectors, and mechanical information.

[0059] (3) Obtaining pixel depth: Using the target camera parameters, set up the simulation camera in the simulation environment, and use the perspective of the simulation camera to collect and process the elastic body model to determine the boundary depth corresponding to the outermost pixel in the contact contour of the elastic body model. Then, using the boundary depth as a constraint, use the target normal vector set to reconstruct each pixel in the contact area pixel by pixel Poisson integral to obtain the pixel depth value corresponding to each pixel.

[0060] For details, see Figure 6 The flowchart shown illustrates a pixel depth determination method. After obtaining the normal vector corresponding to each pixel, the camera and the corresponding transparent elastic body 3D model need to be imported into the simulation environment according to the intrinsic and extrinsic parameters to obtain the sensor boundary depth. Then, starting from the boundary, the normal vector is integrated pixel by pixel to obtain the depth of each pixel.

[0061] In one implementation, the surface normal is known. ,depth The gradient can be calculated as follows:

[0062]

[0063]

[0064] Furthermore, the 3D reconstruction problem can be modeled as solving the following Poisson equation:

[0065]

[0066] Discretizing the above equation using the central difference, we have the following for pixel (i,j):

[0067]

[0068]

[0069]

[0070]

[0071] Discrete linear systems:

[0072] b

[0073] Where A is the sparse matrix of coding depth coefficients, z is the vector containing the depth of all pixels, and b is the divergence term of the Poisson equation.

[0074] Using the actual depth of the image boundary as a priori By weight Integrating the above equation yields the augmented coefficient linear system:

[0075]

[0076] in, , , This is a diagonal matrix, where each diagonal element is 1 at the position corresponding to a priori pixels with effective depth. The depth can be obtained by solving the above overdetermined equations using the least squares method.

[0077] In addition, due to The unit is Therefore The unit also needs to be ,and , Since it is dimensionless, it is necessary to convert it before integrating the normal vector. , Multiply by pixels to convert the unit to mm / pixel, thus making The unit is .

[0078] (4) Training images and normal vectors: Based on the mapping relationship between pixel coordinates, the pixel depth value corresponding to each pixel is mapped to a depth map. Specifically: According to the photometric stereo method, for a Lambertian surface, the brightness observed by the camera can be expressed as:

[0079]

[0080] Where I represents pixel brightness. Let l be the albedo, l be the light source direction vector, and n be the surface normal vector. l can be obtained through a camera. Since l is a constant, it is unknown but definite for different points on the surface. Therefore, it can be assumed that there is a unique and definite mapping relationship between the surface normal and the pixel intensity and pixel coordinates. This mapping relationship F can be learned from a large amount of data through a neural network.

[0081]

[0082] in, For the normal of a point on the surface, These are the pixel coordinates of that point. This represents the pixel brightness captured by the camera under RGB three-color light sources.

[0083] (5) Training depth and mechanical model: The depth map is sent to the initial mechanical model to obtain the pixel normal force corresponding to each pixel point. Based on the pixel normal force and the actual measured force of the finger-shaped visual tactile sensor, the initial mechanical model is trained to obtain the target mechanical model. In one embodiment, according to the principle of elasticity, the force on an elastic body is related to its deformation. The neural network is used to simulate this relationship. The pixel depth is used as the input and the pixel normal force is used as the output. The sum of the forces is the downward pressure of the six-dimensional force sensor, and the depth and mechanical model can be trained.

[0084] In summary, compared with common planar visual tactile sensors, this invention provides a finger-shaped sensor structure that can adapt to various dexterous hand configurations, enabling them to perform precise operations. Furthermore, through the configuration of an innovative calibration method, it can adapt to uneven light source distribution, thus eliminating the need for complex light guide structures like other products while still achieving high accuracy. It can also adapt to various complex curved surfaces, significantly increasing the product's adaptability and customizability, while greatly increasing the product yield and reducing manufacturing costs.

[0085] Compared with traditional calibration methods, the calibration method provided by this invention is simple and effective, with a straightforward procedure. By using simulation software and a 3D model, it obtains camera parameters, images, normal vectors, and force data sets, which can avoid the accumulation of errors caused by complex mathematical modeling, thereby achieving high-precision mechanical prediction.

[0086] Regarding the calibration method for the finger-shaped visual-tactile sensor provided in the foregoing embodiments, this embodiment of the invention provides a calibration device for the finger-shaped visual-tactile sensor, see [link to documentation]. Figure 7 The diagram shows a calibration device for a finger-type visual-tactile sensor, which includes the following components:

[0087] The camera parameter calibration module 702 acquires the first contact image and the second contact image between the elastomer in the finger-shaped visual tactile sensor and the finger-shaped calibration sleeve and the calibration ball, respectively, and performs simulation processing on the finger-shaped calibration sleeve and the first contact image based on the initial camera parameters to determine the target camera parameters.

[0088] The normal vector calibration module 704 performs simulation processing on the calibration ball and the second contact image based on the target camera parameters to determine the target position of the calibration ball and the target normal vector set corresponding to the contact area between the calibration ball and the elastic body.

[0089] The depth reconstruction module 706 obtains the boundary depth of the elastic body model, and performs depth reconstruction processing on the contact area of ​​the elastic body based on the target normal vector set and the boundary depth to obtain the pixel depth value corresponding to each pixel in the contact area.

[0090] The model training module 708 uses the pixel depth value corresponding to each pixel to train the initial mechanical model and obtain the target mechanical model, so as to perform mechanical analysis using the target mechanical model when the finger-shaped visual tactile sensor performs a grasping operation.

[0091] The calibration device for the finger-shaped visual-tactile sensor provided in this application embodiment can significantly improve the sensor's adaptability and accuracy.

[0092] In one embodiment, when performing the step of simulating the finger calibration sleeve and the first contact image based on initial camera parameters to determine the target camera parameters, the camera parameter calibration module 702 is further configured to: use the initial camera parameters to perform model rendering processing on the finger calibration sleeve in the simulation environment to generate a three-dimensional model of the finger calibration sleeve; if the three-dimensional model of the finger calibration sleeve coincides with the first contact image, then the camera parameters at this time are determined as the target camera parameters.

[0093] In one embodiment, when performing the steps of simulating the calibration sphere and the second contact image based on the target camera parameters to determine the target position of the calibration sphere and the target normal vector set corresponding to the contact area between the calibration sphere and the elastic body, the normal vector calibration module 704 is further configured to: use the target camera parameters to perform model rendering processing on the calibration sphere in the simulation environment to generate a three-dimensional model of the calibration sphere; adjust the coordinate position of the calibration sphere so that the three-dimensional model of the calibration sphere coincides with the second contact image, and determine the coordinates of the calibration sphere when they coincide as the target position, so as to determine the target normal vector set corresponding to the contact area between the calibration sphere and the elastic body based on the target position.

[0094] In one embodiment, when performing the step of obtaining the boundary depth of the elastomer model, the depth reconstruction module 706 is further configured to: use the target camera parameters to set up a simulation camera in the simulation environment, and use the perspective of the simulation camera to collect and process the elastomer model to determine the boundary depth corresponding to the outermost pixel in the contact contour of the elastomer model.

[0095] In one embodiment, when performing depth reconstruction processing on the contact area of ​​the elastic body based on the target normal vector set and the boundary depth to obtain the pixel depth value corresponding to each pixel in the contact area, the depth reconstruction module 706 is further configured to: use the boundary depth as a constraint condition and the target normal vector set to perform pixel-by-pixel Poisson integral reconstruction on each pixel in the contact area to obtain the pixel depth value corresponding to each pixel.

[0096] In one embodiment, after obtaining the pixel depth value corresponding to each pixel, the depth reconstruction module 706 is further configured to: map the pixel depth value corresponding to each pixel to a depth map based on the mapping relationship between pixel coordinates.

[0097] In one embodiment, when performing the step of training the initial mechanical model using the pixel depth values ​​corresponding to each pixel to obtain the target mechanical model, the model training module 708 is further configured to: send the depth map to the initial mechanical model, obtain the pixel normal force corresponding to each pixel, and perform model training on the initial mechanical model based on the pixel normal force and the actual measured force of the finger-shaped visual tactile sensor to obtain the target mechanical model.

[0098] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0099] This invention provides a server, specifically, the server includes a processor and a storage device; the storage device stores a computer program, which, when run by the processor, executes the method described in any of the above embodiments.

[0100] Figure 8 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. The server 100 includes: a processor 80, a memory 81, a bus 82, and a communication interface 83. The processor 80, the communication interface 83, and the memory 81 are connected through the bus 82. The processor 80 is used to execute executable modules, such as computer programs, stored in the memory 81.

[0101] The memory 81 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 83 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0102] Bus 82 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0103] The memory 81 is used to store programs. After receiving an execution instruction, the processor 80 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 80 or implemented by the processor 80.

[0104] The processor 80 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 80 or by instructions in software form. The processor 80 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 81. The processor 80 reads the information in memory 81 and, in conjunction with its hardware, completes the steps of the above method.

[0105] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0106] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A calibration method for a finger-shaped visual-tactile sensor, characterized in that, The method includes: Acquire the first contact image and the second contact image between the elastomer in the finger-shaped visual tactile sensor and the finger-shaped calibration sleeve and the calibration ball, respectively. Based on the initial camera parameters, perform simulation processing on the finger-shaped calibration sleeve and the first contact image to determine the target camera parameters. Based on the target camera parameters, the calibration ball and the second contact image are simulated to determine the target position of the calibration ball and the target normal vector set corresponding to the contact area between the calibration ball and the elastic body; Specifically, based on the target camera parameters, a simulation camera model consistent with the actual shooting perspective is established in the simulation environment, and the three-dimensional geometric model of the calibration ball is imported. By iteratively adjusting the spatial pose of the calibration ball, the simulation rendering image and the second contact image are made to coincide at the pixel level to determine the target position of the calibration ball. Using the calibrated camera intrinsic parameters, camera extrinsic parameters and the surface equation of the calibration ball, the local normal direction of each pixel in the contact area between the calibration ball and the elastic body under the coincident state is calculated, so as to summarize the local normal directions to form the target normal vector set. Obtain the boundary depth of the elastomer model, and perform depth reconstruction processing on the contact area of ​​the elastomer based on the target normal vector set and the boundary depth to obtain the pixel depth value corresponding to each pixel in the contact area; The initial mechanical model is trained using the pixel depth value corresponding to each pixel to obtain the target mechanical model, which is then used for mechanical analysis when the finger-shaped visual tactile sensor performs a grasping operation. The step of simulating the calibration sphere and the second contact image based on the target camera parameters to determine the target position of the calibration sphere and the target normal vector set corresponding to the contact area between the calibration sphere and the elastic body includes: using the target camera parameters to perform model rendering processing on the calibration sphere in a simulation environment to generate a three-dimensional model of the calibration sphere; adjusting the coordinate position of the calibration sphere so that the three-dimensional model of the calibration sphere coincides with the second contact image, and determining the coordinates of the calibration sphere at the time of coincidence as the target position, so as to determine the target normal vector set corresponding to the contact area between the calibration sphere and the elastic body based on the target position; The step of using the pixel depth values ​​corresponding to each pixel to train the initial mechanical model and obtain the target mechanical model includes: sending the depth map to the initial mechanical model, obtaining the pixel normal force corresponding to each pixel, and training the initial mechanical model based on the pixel normal force and the actual measured force of the finger-shaped visual tactile sensor to obtain the target mechanical model.

2. The calibration method for the finger-shaped visual-tactile sensor according to claim 1, characterized in that, The step of simulating the finger-shaped calibration sleeve and the first contact image based on initial camera parameters to determine the target camera parameters includes: Using the initial camera parameters, the finger calibration sleeve is rendered in a simulation environment to generate a three-dimensional model of the finger calibration sleeve. If the three-dimensional model of the finger calibration kit coincides with the first contact image, then the camera parameters at this time are determined as the target camera parameters.

3. The calibration method for the finger-shaped visual-tactile sensor according to claim 1, characterized in that, The step of obtaining the boundary depth of the elastic body model includes: Using the target camera parameters, a simulation camera is set up in the simulation environment, and the elastomer model is acquired and processed from the perspective of the simulation camera to determine the boundary depth corresponding to the outermost pixel in the contact contour of the elastomer model.

4. The calibration method for the finger-shaped visual-tactile sensor according to claim 1, characterized in that, The step of performing depth reconstruction processing on the contact region of the elastic body based on the target normal vector set and the boundary depth to obtain the pixel depth value corresponding to each pixel in the contact region includes: Using the boundary depth as a constraint, and employing the target normal vector set, a pixel-by-pixel Poisson integral reconstruction is performed on each pixel in the contact area to obtain the pixel depth value corresponding to each pixel.

5. The calibration method for the finger-shaped visual-tactile sensor according to claim 4, characterized in that, After obtaining the pixel depth value corresponding to each pixel, the process includes: Based on the mapping relationship between pixel coordinates, the pixel depth value corresponding to each pixel point is mapped to a depth map.

6. A calibration device for a finger-shaped visual-tactile sensor, characterized in that, The device includes: The camera parameter calibration module acquires the first contact image and the second contact image between the elastomer in the finger-shaped visual tactile sensor and the finger-shaped calibration sleeve and the calibration ball, respectively. Based on the initial camera parameters, it performs simulation processing on the finger-shaped calibration sleeve and the first contact image to determine the target camera parameters. The normal vector calibration module performs simulation processing on the calibration ball and the second contact image based on the target camera parameters to determine the target position of the calibration ball and the target normal vector set corresponding to the contact area between the calibration ball and the elastic body. Specifically, based on the target camera parameters, a simulation camera model consistent with the actual shooting perspective is established in the simulation environment, and the three-dimensional geometric model of the calibration ball is imported. By iteratively adjusting the spatial pose of the calibration ball, the simulation rendering image and the second contact image are made to coincide at the pixel level to determine the target position of the calibration ball. Using the calibrated camera intrinsic parameters, camera extrinsic parameters and the surface equation of the calibration ball, the local normal direction of each pixel in the contact area between the calibration ball and the elastic body under the coincident state is calculated, so as to summarize the local normal directions to form the target normal vector set. The depth reconstruction module obtains the boundary depth of the elastic body model, and performs depth reconstruction processing on the contact area of ​​the elastic body based on the target normal vector set and the boundary depth to obtain the pixel depth value corresponding to each pixel in the contact area. The model training module uses the pixel depth value corresponding to each pixel to train the initial mechanical model to obtain the target mechanical model, so as to perform mechanical analysis using the target mechanical model when the finger-shaped visual tactile sensor performs the grasping operation. The step of simulating the calibration sphere and the second contact image based on the target camera parameters to determine the target position of the calibration sphere and the target normal vector set corresponding to the contact area between the calibration sphere and the elastic body includes: using the target camera parameters to perform model rendering processing on the calibration sphere in a simulation environment to generate a three-dimensional model of the calibration sphere; adjusting the coordinate position of the calibration sphere so that the three-dimensional model of the calibration sphere coincides with the second contact image, and determining the coordinates of the calibration sphere at the time of coincidence as the target position, so as to determine the target normal vector set corresponding to the contact area between the calibration sphere and the elastic body based on the target position; The step of using the pixel depth values ​​corresponding to each pixel to train the initial mechanical model and obtain the target mechanical model includes: sending the depth map to the initial mechanical model, obtaining the pixel normal force corresponding to each pixel, and training the initial mechanical model based on the pixel normal force and the actual measured force of the finger-shaped visual tactile sensor to obtain the target mechanical model.

7. A server, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method described in any one of claims 1 to 5.

Images