An operating object force estimation method based on a visual-tactile sensor

Abstract

The application discloses an operating object force estimation method based on a visual-tactile sensor, and comprises the following steps: decomposing the displacement of a marked point in the contact area of an operating object and a sensor silica gel into a tangential displacement component and a rotational displacement component; constructing a constraint equation group according to the rotational displacement component of each marked point; solving the constraint equation group to obtain the tangential displacement component and the rotational displacement component of each marked point; and inputting the tangential displacement component and the rotational displacement component into a fitting function to estimate the force and the torque borne by the object, wherein the fitting function is a function fitted according to the displacement of the marked point collected in a calibration process and the force and the torque borne by the object. The application decomposes the external force borne by the silica gel into a tangential force and a normal force torque by setting a reasonable mechanical model, decomposes the displacement of the silica gel when deformed into a tangential displacement component and a rotational displacement component, builds multiple constraint equations for solving, can simultaneously estimate the tangential force and the normal force torque, and thus improves the accuracy of the force estimation of the robot on the operating object.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot force detection, in particular to an operating object force estimation method based on a visual-tactile sensor. BACKGROUND

[0002] With the continuous progress of science and technology, robots are increasingly widely used in real life, but the resulting safety problems have also received increasing attention. When performing different operation tasks, robots need to operate different objects, and if the operating object collides with other objects or is blocked by force during operation, it is not only easy to cause damage to the robot or the object, but also can even cause harm to personal safety. Therefore, in human-robot collaborative operation, robot operation in unstructured environments, precision installation and many other tasks, how to enable robots to have force perception ability for the operating objects is an unavoidable problem.

[0003] Methods for enabling robots to have force perception ability mainly include two categories: traditional force-torque sensors and tactile sensors. Traditional force and torque sensors can mostly detect single-point or single-joint force and torque information in detail, have a large volume, are usually installed at the joints or end flanges of the robot arm, and can directly obtain force and torque signals. However, since they cannot be installed at the end of the gripper, if the force of the operating object needs to be obtained, the specific physical parameters of the robot arm need to be known and then a dynamics method is used to calculate the force and torque signals. If the physical parameter model is not accurate enough, the calculated force and torque signals can easily contain large errors. SUMMARY

[0004] The present application aims to solve the problem of how to improve the accuracy of robot operating object force estimation, and proposes an operating object force estimation method based on a visual-tactile sensor.

[0005] The technical problem of the present application is solved by the following technical solution:

[0006] An operating object force estimation method based on a visual-tactile sensor, comprising the following steps:

[0007] S1, disassembling the displacement of the marked points in the contact area of the operating object and the sensor silica gel into a tangential displacement component generated under the action of tangential force and a rotational displacement component generated under the action of normal torque;

[0008] S2, constructing a constraint equation set according to the rotational displacement components of the marked points;

[0009] S3, solving the constraint equation set to obtain the tangential displacement component and the rotational displacement component of each marked point;

[0010] S4, inputting the tangential displacement component and the rotational displacement component into a fitting function to estimate the force and the torque on the object, wherein the fitting function is a function fitted according to the displacement of the marker points and the force and the torque on the object collected in the calibration process.

[0011] In some embodiments, the function relationship in step S4 is fitted by a method of collecting calibration data, comprising the following steps:

[0012] A1, controlling the robot arm to make the visual-tactile sensor contact the sample, and keeping the visual-tactile sensor in constant force contact with the sample in the vertical direction;

[0013] A2, controlling the robot arm to move horizontally or rotate to make the silicone membrane deform, solving a constraint equation group constructed according to the rotational displacement component of each marker point, and extracting the horizontal displacement of the marker point displacement and the tangent value tanθ of the rotation angle, while recording the tangential force and the normal force torque T read by the rotation matrix at this time;

[0014] A3, fitting the functions of x to T and tanθ to T by using the collected multiple groups of data.

[0015] In some embodiments, the step S2 of constructing the constraint equation group comprises:

[0016] constructing a first constraint equation group with the rotational displacement component of each marker point being perpendicular to the line connecting it to the rotation center;

[0017] constructing a second constraint equation group with the distance from each marker point to the rotation center being kept unchanged before and after rotation.

[0018] In some embodiments, the step of constructing the first constraint equation group comprises:

[0019] obtaining the following expressions with any two marker points in the contact area:

[0020] (x mi -x mj )x f +(x Di -x Dj )x c +(y mi -y mj )y f +(y Di -y Dj )y c =x mi *x Di +y mi *y Di +x mj ​​​​x Dj y mj x Dj (i≠j)

[0021] where P mi = (x mi , y mi ) represents the camera coordinate system coordinates of the current marker point, P c = (x c , y c ) represents the camera coordinate system coordinates of the current rotation center, and the coordinate form of the displacement vector D i of the marker point is (x Di , y Di ); the coordinate form of the horizontal displacement component D i in D f is (x f , y f ), and the suffixes i and j represent the relevant variables of the corresponding marker points.

[0022] In some embodiments, the constructing the first constraint equation set further includes:

[0023] Suppose there are n marker points, and n expressions can be obtained, which are expressed in matrix form as follows:

[0024]

[0025] where P mi = (x mi , y mi ) represents the camera coordinate system coordinates of the current marker point, P c = (x c , y c ) represents the camera coordinate system coordinates of the current rotation center, and the coordinate form of the displacement vector D i of the marker point is (x Di , y Di ); the coordinate form of the horizontal displacement component D i in D f is (x f , y f ), and the suffixes i and j represent the relevant variables of the corresponding marker points.

[0026] In some embodiments, the constructing the second constraint equation set includes:

[0027] Any two points in all contact areas are marked to obtain the following expression:

[0028] 2(x′ mi -x′ mj )x f -2(xDi -x Dj )x c +2(y′ mi -y′ mj )y f -2(y Di -y Dj )y c =x Di (x mi +x′ mi )+y Di (y mi +y′ mi )+x Dj (x mj +x′ mj )+y Dj (y mj +y′ mj (i≠j)

[0029] Where P' mi =(x' mi ,y' mi ) and P mi =(x mi ,y mi P represents the camera coordinates before and after the marker point moves. c =(x c ,y c () represents the camera coordinates of the current rotation center, and the displacement vector D of the marked point. i The coordinate form is (x Di ,y Di ,);D i The horizontal displacement component D in f The coordinate form is (x f ,y f The suffixes i and j represent the relevant variables of the corresponding marker points.

[0030] In some embodiments, constructing the second set of constraint equations further includes:

[0031] Assuming there are n marker points in total, it is possible to obtain The expression, in matrix form, is as follows:

[0032]

[0033] Where P mi =(x mi ,y mi P represents the camera coordinates of the current marker point. c =(x c ,y c() represents the camera coordinates of the current rotation center, and the displacement vector D of the marked point. i The coordinate form is (x Di ,y Di ,);D i The horizontal displacement component D in f The coordinate form is (x f ,y f ), where j is any marked point.

[0034] In some embodiments, solving the constraint equation system in step S3 is performed using the least squares method, including: solving Y = AX using the least squares method to obtain the following expression:

[0035] X = (A T A) -1 A T Y

[0036] in

[0037] Finally, the tangential displacement D caused by the tangential force can be obtained. f =(x f ,y f The coordinates P of the rotation center in the image. c =(x c ,y c ).

[0038] In some embodiments, the magnitude of the normal torque is positively correlated with the amplitude of the rotational motion, and the expression for the amplitude of the rotational motion is:

[0039] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the above-described method for estimating the force on an object based on a visual-tactile sensor.

[0040] The present invention has the following beneficial effects:

[0041] This invention proposes a method for estimating the force on a manipulated object based on a visual-tactile sensor. By setting a reasonable mechanical model, the external force on the silicone is decomposed into tangential force and normal torque. The displacement of the silicone during deformation is decomposed into tangential displacement component generated under tangential force and rotational displacement component generated under normal torque. Multiple sets of constraint equations are built and solved, which can realize the simultaneous estimation of tangential force and normal torque, thereby improving the accuracy of the robot's estimation of the force on the manipulated object.

[0042] In some embodiments, the present invention also has the following effects:

[0043] The application constructs a distance constraint equation by means of the characteristic that the contact area is relatively static, avoids the problem that the direction of the displacement vector fluctuates too much due to noise, and further improves the accuracy of the estimation result.

[0044] The application discloses a calibration method, which can realize rapid calibration of a sensor and make the sensor have better universality.

[0045] Other advantages of the application will be further described in the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a force estimation method flow chart of an operating object based on a visual tactile sensor in the embodiment of the application;

[0047] Figure 2 is a force estimation method flow chart of an operating object in the prior art one;

[0048] Figure 3 is a specific force estimation method flow chart of an operating object based on a visual tactile sensor in the embodiment of the application;

[0049] Figure 4 is a displacement D decomposition schematic diagram of a marker point in the embodiment of the application;

[0050] Figure 5 is a force analysis diagram of each marker point in a contact surface in the embodiment of the application;

[0051] Figure 6 is an exploded view of a sensor structure in the embodiment of the application;

[0052] Figure 7 is a structure and position schematic diagram of a calibration device in the embodiment of the application;

[0053] Figure 8 is a tangential displacement and dt and tangential force fitting relationship diagram in the embodiment of the application;

[0054] Figure 9 is a torque τ and rotation angle tangent value tanθ relationship diagram in the embodiment of the application;

[0055] Figure 10 is an x-axis tangential force estimation error comparison diagram between the prior art two and the embodiment of the application;

[0056] Figure 11 is a y-axis tangential force estimation error comparison diagram between the prior art two and the embodiment of the application;

[0057] Figure 12 is a normal force torque estimation error comparison diagram between the prior art two and the embodiment of the application;

[0058] In the figure, 1-silicone membrane, 2-transparent acrylic plate, 3-lamp strip, 4-sensor shell, 5-camera, 6-mechanical arm end, 7-force-torque sensor, 8-sensor connecting piece, 9-vision-tactile sensor, 10-operating object, 11-operating object fixing piece. DETAILED DESCRIPTION

[0059] The application will be further described below with reference to the drawings and in conjunction with the preferred embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0060] It should be noted that the left, right, up, down, top, bottom and other orientation terms in the embodiments are only relative concepts or are referenced to the normal use state of the product, and should not be considered as limiting.

[0061] The tactile sensor pattern can be based on electricity, magnetism, optics, vision and the like according to principles. The electrical tactile sensor mainly relies on measuring the resistance or capacitance to sense the contact force. Such a sensor has a large range and good linearity, but is susceptible to noise interference and generally has low integration. The magnetic tactile sensor acquires the contact force by sensing the change of the magnetic field distribution inside the element caused by pressure. Such a sensor has good durability but is difficult to process and difficult to maintain good consistency. The optical tactile sensor relies on optical fibers to sense the deformation at the contact to sense the force signal. The sensitivity is excellent but requires a relatively complex acquisition device and has low signal linearity.

[0062] The vision-based tactile sensor uses a camera to collect the deformation image of the silicone membrane after contacting with the outside world to sense the contact force. The deformation signal of the silicone can be further highlighted by adding marker points on the silicone mold. A large number of dense force signals can be obtained by using images, and the pose of the operating object in the hand can also be judged by deformation details, which can greatly improve the operating ability of the robot.

[0063] As shown in Figure 1 The operating object force estimation method based on the vision-tactile sensor according to the embodiments of the present application includes an operating object force analysis algorithm based on the vision-tactile sensor. The relatively disordered and nonlinear force and torque information in the image can be converted into relatively linear tangential displacement and rotation angle, thereby decomposing the mixed force on each marker point into tangential force parallel to the contact surface and torque perpendicular to the contact surface. The embodiments also design a calibration tool according to the principle. With the help of the tool, the sensor can be calibrated and used only with a small amount of data, greatly improving the universality of the algorithm.

[0064] Before describing the embodiments of the present application, the reasons for the defects of the prior art will be analyzed:

[0065] Prior art one:

[0066] As shown in Figure 2 , the collected image is taken as input, first the displacement of the marker points on the silicone mold is identified by means of optical flow tracking algorithm, and then the force information of each point in the image is learned by means of neural network method.

[0067] In order to better reduce the error in the image coordinate system and the physical coordinate system, the method first learns the 6D affine transformation of the marker points by means of STN network (synchronous transmission network). Then a u-net (semantic segmentation network) is used to learn the three-dimensional force situation of each point by taking the displacement of each point as input.

[0068] In order to obtain the three-dimensional force data label of each point, the method uses finite element simulation software to maximize the approximation of the accurate force information of each point by building a similar environment in the simulation environment as in reality.

[0069] Prior art one has the following shortcomings:

[0070] 1. The neural network training needs to collect a large amount of accurate data set, so the acquisition of such a large data set is difficult, and when encountering a contact force type with large difference from the data set, it is difficult to have good robustness.

[0071] 2. The simulation method to obtain accurate force signal label needs to establish an accurate simulation model for the silicone film, and the model parameters of hyperelastic objects are difficult to measure, and the contact model is not only affected by the contact force, but also affected by other external physical environment, so the simulation data label is easy to deviate from the reality.

[0072] 3. Although this method can obtain dense force information, in most cases in practical application, such dense force signal is not needed, and more attention is paid to the overall force information of the operating object, and dense calculation will occupy too much computing amount.

[0073] Prior art two:

[0074] Raj Kolamuri et al. of Carnegie Mellon University proposed a method based on vector orthogonal fitting moment, which mainly includes three steps: object contact detection, rotation start detection and rotation angle measurement. The specific content is as follows:

[0075] (1). Object contact detection: first collect the image of the initial non-contact as the initial image, and take the difference between the real-time image and the initial image, if the difference is greater than a certain threshold, it means that the region is in contact with the operating object, and after the contact is stable, an image is taken as the contact frame.

[0076] (2). Detection of rotation start: whether rotation occurs is determined by calculating the relative motion change and angular change of the contact post-mark vector w.r.t the contact mark of each frame. If it is greater than a set threshold, it indicates that rotation starts.

[0077] (3). Measurement of rotation angle: to calculate the rotation angle, the coordinates of the rotation center need to be determined first. This method lists an equation for each marker point by means of the vector from the rotation center to the marker point being perpendicular to the marker point displacement, and uses the least square method to fit the rotation center. Finally, the rotation angle can be obtained by calculating the included angle between the center of the circle and the initial point and the current point.

[0078] The prior art two has the following disadvantages:

[0079] 1. When fitting, only the influence of the moment on the displacement of the marker point is considered, and the existence of the tangential force is ignored, which can easily cause a large error when there is a large tangential force.

[0080] 2. When the displacement of the marker point caused by the moment is small, slight noise of the current marker point can easily cause a large angle change in the direction of the displacement vector, and in this case, the fitting can easily have a large estimation error.

[0081] The problems to be solved by the embodiments of the present application are three problems in the process of force analysis of the operated object using the visual-tactile sensor:

[0082] First: in view of the first disadvantage in the prior art two, the tangential force is added in the analysis process, the displacement of the marker point is modeled as the tangential displacement caused by the tangential force and the rotational displacement caused by the normal moment, and a multi-equation system is established based on the model for analysis, so as to ensure that the normal moment of the operated object can be estimated while the tangential force can be estimated.

[0083] Second: in view of the second disadvantage in the prior art two, in the analysis process, not only the vector from the center of the circle to the current marker point being perpendicular to the rotational displacement vector of the marker point is used as the only constraint term, but also the characteristic that the length of the rotational displacement of the marker point is on the same circle before and after rotation is introduced to construct a multi-equation for optimization, so as to reduce the problem of large vector direction jumping when facing small normal moment.

[0084] Third: the present embodiment proposes a sensor calibration method for the above-mentioned operated object force analysis algorithm, which can quickly realize sensor calibration and is easy to migrate, greatly improving the universality of the above-mentioned sensor. Compared with the deep learning method in the technical 1, only a small amount of data set is needed to fit an accurate force signal.

[0085] Reference Figure 3 The specific process of the operated object force estimation method based on the visual-tactile sensor in the embodiments of the present application includes the following steps:

[0086] Operation object contact detection

[0087] When the surface of the silica gel is in contact with the object, the surface is flat and the texture is relatively simple. When the silica gel is in contact with the operation object, the surface will be deformed due to the contact with the operation object, and a certain texture will appear. In this embodiment, the image gradient is first extracted by means of the Sobel operator, and then the region with complex texture in the image is extracted by means of mean filtering and median filtering. For the misjudgment region in the image caused by noise, this embodiment adopts the method of erosion and expansion of the binary image for screening, and the remaining region after screening is regarded as the contact region of the sensor and the operation object. At the same time, the pose of the operation object relative to the sensor can be preliminarily determined by using the contact texture, which is beneficial to the subsequent more delicate operation of the operation object.

[0088] Operation object force model establishment

[0089] In this embodiment, the establishment of the operation object force model needs to meet two conditions: first, only consider the operation object in the plane parallel to the gripper, and most of the operation scenes in reality meet this condition. Second, the contact area between the silica gel and the operation object remains relatively static when the operation object force moves, that is, there is no sliding between them, and the tangential force is only provided by static friction. This assumption can be easily realized in the real environment as long as the operation object force is not too large (within the range).

[0090] When the operation object force moves, the part of the silica gel mold in contact with the operation object remains relatively static, so the silica gel mold will produce corresponding deformation, and the reaction force generated by the silica gel mold on the operation object is opposite to the deformation direction. In this embodiment, the deformation of the silica gel mold is characterized by marking points on the surface of the silica gel and tracking. Since the object is only operated in the plane parallel to the gripper, that is, the operation object is only subjected to the pressure perpendicular to the contact surface generated by the gripper and the tangential force f parallel to the contact surface and the normal moment τ perpendicular to the contact surface (hereinafter referred to as normal moment), which can be expressed by equation (1):

[0091] F = f + τ / d (1) Wherein, F represents the resultant force of the tangential force and the normal moment of the two at a certain point on the surface of the silica gel, d represents the Euclidean distance between the point and the center of rotation, and the action of F on the sensor is manifested as the movement of the marking point in the image. Referring to the decomposition of force, this embodiment decomposes the displacement D of the marking point into the horizontal displacement D f caused by the tangential force and the rotational displacement D τ , as shown in equation (2): Figure 4

[0092]

[0093] Operation object force analysis ​

[0094] According to the characteristics of the tangential force, the tangential force directions of each marker point in the contact surface are consistent, and the sizes are the same, so the tangential displacement of each marker point is the same. That is (1, 2,..., i represents the number of the marker point). The force generated by the normal moment on the contact surface presents a concentric circle distribution, and the force is tangent to the circle with the rotation center as the center, so D τ As shown in Figure 5 .

[0095] Where P mi = (x mi , y mi ) represents the camera coordinate system coordinates of the current marker point, and P c = (x c , y c ) represents the camera coordinate system coordinates of the current rotation center. According to the fact that the straight line passing through a point on a circle and the center of the circle is perpendicular to the tangent of the point, it can be obtained that P c P mi ⊥D τi , and by combining formula (2), it can be obtained that Transformed into a camera coordinate system expression:

[0096] (x mi -x c )(x Di -x f )+(y mi -y c )(y Di -y f ) = 0 (3)

[0097] Where the displacement vector D i of the marker point can be written as a coordinate form (x Di , y Di ); The horizontal displacement component D i in D f can be written as a coordinate form (x f , y f )

[0098] Simplifying can obtain:

[0099] x mi x f +x Di x c +y mi y f +y Di y c -x mi *x Di -y mi *y Di = -xc x f -y c y f (4)

[0100] since -x c x f -y c y f is a constant term, which is irrelevant to the marker points, the displacement of any marker point satisfies the above formula. Then we can take any marker point j, j≠i, which still satisfies formula (4), and subtract the two to eliminate the constant term:

[0101] (x mi -x mj )x f +(x Di -x Dj )x c +(y mi -y mj )y f +(y Di -y Dj )y c =x mi *x Di +y mi *y Di -x mj *x Dj -y mj *y Dj (i≠j) (5)

[0102] The above expression can be obtained by marking any two points in all contact areas, and if there are n marker points, we can obtain expressions, which can be written in matrix form as:

[0103]

[0104] When D i is small, it is easily affected by the end tracking noise, which causes the direction of the vector to fluctuate to a large extent, easily leading to a large error in the above expression. In order to reduce the potential impact of this factor, we add the relative rest of each point in the contact part as a constraint term. From equation (2), we know that in the camera coordinate system, the motion of the marker point can be analyzed as horizontal motion D f and rotational motion D τ , where we use the rotation matrix R τ to represent the rotational motion, then the motion of each point in the contact area in the camera coordinate system can be represented as:

[0105]

[0106] where P'c = (x' c , y' c ) and P' mi = (x' mi , y' mi ) represent the coordinates of the marker points before and after the rotation, respectively. According to assumption two, we know that the contact area of the silicone mold and the object to be manipulated remain relatively static, i.e., the marker points in the contact area remain relatively static when the force is applied, and the rotation matrix |R τ | = 1, so we have:

[0107]

[0108] Because the center of rotation P c is in the contact area, the displacement caused by the rotation during the movement is 0, i.e., D τ = 0, so we have:

[0109]

[0110] O represents the origin of the camera coordinate system, and by combining equations (7)-(9), we have:

[0111]

[0112] Converting equation (10) to the camera coordinate system, we have:

[0113] (x mi -x c ) 2 +(y mi -y c ) 2 = (x' mi -x c -x f ) 2 +(y' mi -y c -y f ) 2 (11)

[0114] Simplifying, we have:

[0115] 2x' mi x f -2(x mi -x' mi )x c +2y' mi y f -2(y mi -y' mi )y c +x mi 2 -x' mi2 +y mi 2 -y' mi 2 =(x c +x f ) 2 +x c 2 +(y c +y f ) 2 +y c 2 (12)

[0116] From the definition of P mi and P' mi , we have:

[0117]

[0118] Since (x c +x f ) 2 +x c 2 +(y c +y f ) 2 +y c 2 is a constant term, which is independent of the marker point, the above formula is satisfied for the displacement of any marker point. Then we can take any marker point j, j≠i, which still satisfies formula (12). Subtracting the two and eliminating the constant term, we can get:

[0119] 2(x′ mi -x′ mj )x f -2(x Di -x Dj )x c +2(y′ mi -y′ mj )y f -2(y Di -y Dj )y c =x Di (x mi +x′ mi )+y Di (y mi +y′ mi )+x Dj (x mj +x′ mj )+y Dj (y mj +y′ mj )(i≠j) (14)

[0120] Two points in all contact areas are marked as points, and the above expression can be obtained. If there are n points, the expression can be obtained, which can be written in matrix form as:

[0121] Combined with formulas (6) and (15), n(n-1) equations can be obtained, and:

[0122]

[0123]

[0124] Then Y = AX can be obtained, and the solution can be obtained by using the least square method:

[0125] X = (A T A) -1 A T Y (16)

[0126] Finally, the tangential displacement D caused by the tangential force can be obtained f = (x f ,y f ), and the coordinates of the rotation center in the image P c = (x c ,y c ).

[0127] Since the size of the normal force moment is positively correlated with the amplitude of the rotational motion, the embodiment uses to represent the amplitude of the rotational motion.

[0128] The above formulas (3)-(6) are the first constraint equation set constructed by the perpendicular of the rotational displacement component of each marker point to the line connecting it to the rotation center, and the formulas (7)-(15) are the second constraint equation set constructed by the positive correlation between the length of the rotational displacement of each marker point and the distance from the rotation center. Formula (16) can be used to obtain the rotational displacement component by using the known total displacement and tangential displacement. The angle of the rotational motion can be obtained by using the rotational displacement component and the coordinates of the rotation center. The angle of the rotational motion has a certain corresponding relationship with the normal force moment. The function fitting in the calibration includes fitting the mapping function of the rotational angle and the normal force moment.

[0129] Calibration method

[0130] As known from the above subsection, the tangential displacement and the rotational amplitude are respectively positively correlated with the tangential moment and the normal moment. In order to obtain accurate tangential force and normal force moment information, the two need to be calibrated.

[0131] The sensor structure designed in the embodiment is as follows: Figure 6 ​As shown, including silica gel film 1, transparent acrylic plate 2, lamp strip 3, sensor shell 4, camera 5; reference Figure 7 , which is fixed relative to the 6-dimensional force-torque sensor, mechanical arm, and the calibration device includes a force-torque sensor 5 on the end of the mechanical arm 4, sensor connector 6, visual tactile sensor 7, two sensors can be symmetrically installed on the end of the common two-finger gripper for grabbing the object to be operated, and two symmetric sensors are used to clamp the object during operation.

[0132] Calibration steps:

[0133] 1. Control the mechanical arm to move the visual tactile sensor to the top of the operation sample, i.e., the operation object 8, which is fixed by the operation object fixing part 9, and then move downward to make the visual tactile sensor contact the operation sample, and keep the visual tactile sensor in constant force contact with the operation object in the vertical direction.

[0134] 2. On the basis of ensuring the constant force of the visual tactile sensor in the vertical direction of the operation sample, control the mechanical arm to move horizontally or rotate to make the silica gel film deform, and use the method in 3.2.3 to extract the horizontal displacement of the marker point displacement and the tangent value of the rotation angle tanθ, while recording the tangential force and normal torque T read by the rotation matrix at this time.

[0135] 3. Use the method proposed in 2 to collect multiple sets of data, and input these data into matlab to fit to tanθ to T function. Then and tanθ can be used to estimate and T.

[0136] Principle of calibration process

[0137] Calibration only by pressing the calibration single sensor by the mechanical arm, for single-sided sensor and pressing effect similar to operation, the sensing principle is the same as the sensing principle in operation. When the object is in a balanced state after being subjected to external force, the object is subjected to external contact force and friction force generated by the contraction of silica gel due to deformation. Since it is in a balanced state, the force generated by the silica gel on the object should be equal in size and opposite in direction to the external contact force. Because this force is generated by the deformation of silica gel, it has a certain correlation with the degree of deformation of silica gel, which can be reflected by the displacement of the marker point, i.e. the external contact force on the object can be detected by detecting the displacement of the marker point. However, the function relationship between the two is affected by a series of factors such as material and silica gel thickness, so the function relationship between the two is fitted by collecting calibration data.

[0138] Data collection results are as follows Figure 8 , Figure 9 ​​, Figure 8 is f and D f relationship diagram, Figure 9 is f and D f shows a linear relationship, and tan and tan show a logarithmic relationship.

[0139] Comparison of tangential force fitting results: Compared with the prior art two, since it directly sets the tangential force to 0 when setting the constraint equation set, in order to facilitate comparison, we use the average displacement of all marker points to fit the tangential force, and the tangential force estimation error comparison of the two

[0140] The tangential force has two degrees of freedom, and the embodiment of the application uses x-axis and y-axis to represent the two degrees of freedom of the tangential force Figure 10 is the x-axis tangential force estimation error, Figure 11 is the y-axis tangential force estimation error, wherein the dashed line represents the fitting result using the average displacement, and the solid line is the fitting result using the method of the embodiment. It can be seen that compared with using the average value, our method can better reduce the fitting error when the tangential force and the normal force moment exist at the same time, especially when the rotation center is far away from the center of the contact area.

[0141] Comparison of normal force moment fitting results: We calculate the tan and the actual force moment for curve fitting, and then calculate the difference between the fitting result and the true force moment using another group of data, and the error result is as shown in Figure 12 : wherein the dashed line is the estimation error of the prior art two, and the solid line is the estimation error of the embodiment. From the figure, we can find that our method is more stable than method two, and the error fluctuation is smaller.

[0142] The operating object force analysis algorithm proposed in the embodiment can be optimized and solved by using different constraint terms to construct equations or fitted by using a neural network method.

[0143] The embodiment has the following advantages:

[0144] 1. Simultaneously extracting the tangential force and the normal force moment from the relatively disordered marker point displacement signal: The embodiment sets up a reasonable mechanical model, builds multiple constraint equations, and uses the least square method to solve, so as to simultaneously estimate the tangential force and the normal force moment.

[0145] 2. Relieving the situation that the error is large when the displacement of the marker point is too small: The embodiment constructs a distance constraint equation by virtue of the relative stillness of the contact area, avoiding the problem of too large fluctuation of the displacement vector direction caused by noise, and improving the accuracy of the estimation result.

[0146] 3. A calibration method corresponding to the algorithm is proposed: the embodiment proposes a calibration method corresponding to the algorithm, by means of which the rapid calibration of the sensor can be realized, and the sensor has better universality.

[0147] The embodiment has innovativeness in building the contact force model of the operating object in the application of the visual tactile sensor, and proposes a method of constructing a constraint equation group containing tangential force and normal force moment; in the calibration process of the visual tactile sensor, the calibration method and the matching equipment thereof proposed by the embodiment have replicability.

[0148] The above is a further detailed description of the present application in combination with the specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. An operation force estimation method based on a vision-haptic sensor, characterized by, The method comprises the following steps: S1, disassembling the displacement of the marked points in the contact area of the object to be operated and the sensor silica gel into a tangential displacement component generated under the action of a tangential force and a rotational displacement component generated under the action of a normal force moment; S2, constructing a constraint equation set according to the rotational displacement components of the marked points; The construction of the constraint equation set comprises: constructing a first constraint equation set by perpendicularly connecting the rotational displacement components of the marked points to the connecting lines of the rotational center; and constructing a second constraint equation set by keeping the distances of the marked points to the rotational center unchanged before and after rotation; The construction of the first constraint equation set comprises: The following expression is obtained for any two point marked points in all contact areas: (x mi -x mj )x f +(x Di -x Dj )x c +(y mi -y mj )y f +(y Di -y Dj )y c = x mi * x Di + y mi * y Di - x mj * x Dj - y mj * y Dj (i≠j) where P mi = (x mi , y mi ) represents the camera coordinate system coordinates of the current marker point, P c = (x c , y c ) represents the camera coordinate system coordinates of the current rotation center, and the coordinate form of the displacement vector D i of the marker point is (x Di , y Di ); the coordinate form of the horizontal displacement component D i in D f is (x f , y f ), and the suffixes i and j represent the relevant variables of the corresponding marker point; S3, simultaneously solving the first constraint equation set and the second constraint equation set to obtain the tangential displacement component and the rotational displacement component of each marked point; S4, inputting the tangential displacement component and the rotational displacement component into a fitting function to estimate the force and moment acting on the object, wherein the fitting function is a function fitted according to the displacement of the marked points and the force and moment acting on the object collected in the calibration process.

2. The method of claim 1, wherein the force applied to the object is estimated based on the visual-tactile sensor. The function relationship in step S4 is fitted by a method of collecting calibration data, comprising the following steps: A1, controlling the mechanical arm to first make the visual tactile sensor contact with the operation object sample, and keeping the constant force contact of the visual tactile sensor and the operation object in the vertical direction; A2, control the horizontal motion or rotation of the mechanical arm, make the silicone membrane deformation, solve the constraint equation set constructed according to the rotation displacement component of each mark point, and extract the horizontal displacement of the mark point displacement And the tangent value of the rotation angle tanθ, while recording the tangential force read by the rotation matrix at this time And the normal force moment T; A3, by using the collected multiple sets of data, fitting to tan theta to a function of T.

3. The method of claim 1, wherein the force applied to the object is estimated based on the visual-tactile sensor. The construction of the first constraint equation set further comprises: Suppose there are n markers in total, and the following expressions can be obtained, which are expressed in matrix form as follows: where P mi = (x mi , y mi ) represents the camera coordinate system coordinates of the current marker point, P c = (x c , y c ) represents the camera coordinate system coordinates of the current rotation center, and the coordinate form of the displacement vector D i of the marker point is (x Di , y Di ); the coordinate form of the horizontal displacement component D i in D f is (x f , y f ), and the suffixes i and j represent the relevant variables of the corresponding marker point.

4. The method of claim 1, wherein the force applied to the object is estimated based on the visual-tactile sensor. The construction of the second constraint equation set comprises: The following expression is obtained for any two point marked points in all contact areas: 2(x′ mi -x′ mj )x f -2(x Di -x Dj )x c +2(y′ mi -y′ mj )y f -2(y Di -y Dj )y c = x Di (x mi +x′ mi )+ y Di (y mi +y′ mi )+ x Dj (x mj +x′ mj )+ y Dj (y mj +y′ mj )(i≠j) where P mi = (x mi ,y mi ) and P mi = (x mi ,y mi ) represent the camera coordinate system coordinates of the marker points before and after movement, P c = (x c ,y c ) represents the camera coordinate system coordinates of the current center of rotation, and the coordinate form of the displacement vector D i of the marker points is (x Di ,y Di ); the coordinate form of the horizontal displacement component D i in D f is (x f ,y f ), and the suffixes i and j represent the relevant variables of the corresponding marker points.

5. The method of claim 4, wherein the force applied to the object is estimated based on the change in the capacitance of the tactile sensor. The construction of the second constraint equation set further comprises: Suppose there are n markers in total, and the following expressions can be obtained, which are expressed in matrix form as follows: where P mi = (x mi ,y mi ) and P mi = (x mi ,y mi ) represent the camera coordinate system coordinates of the marker points before and after movement, P c = (x c ,y c ) represents the camera coordinate system coordinates of the current center of rotation, and the coordinate form of the displacement vector D i of the marker points is (x Di ,y Di ); the coordinate form of the horizontal displacement component D i in D f is (x f ,y f ), and the suffixes i and j represent the relevant variables of the corresponding marker points.

6. The method of claim 1, wherein the force applied to the object is estimated based on the visual-tactile sensor. The solving in step S3 is solved by the least square method, comprising: solving Y = AX by the least square method to obtain the following expression: X = (A T A) -1 A T Y wherein The tangential displacement D caused by the tangential force can finally be found f = (x f , y f ), the center of rotation in the image has the coordinates P c = (x c , y c ).

7. The method of claim 1, wherein the force of the object is estimated based on the visual-tactile sensor. The magnitude of the normal force moment is positively correlated with the amplitude of the rotational motion, and the amplitude of the rotational motion is expressed as 8. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to realize the force estimation method of the operation object based on the visual tactile sensor in any one of claims 1-7.

Images