An omnidirectional adaptive tactile finger based on visual-tactile fusion and its use method

Abstract

The present invention discloses an omnidirectional adaptive tactile finger based on visual-touch fusion and a method for using the same, comprising: a soft finger body, an image sensor and a bracket; the image sensor is installed between the brackets, the bracket is installed at the bottom of the soft finger body and the image sensor is placed directly below the soft finger body; in the process of the omnidirectional adaptive tactile finger contacting an object, based on the intrinsic force perception principle, non-embedded form of tactile sensation is realized through visual-touch fusion, the coupling solution problem of internal and external perception in soft tactile sensation is overcome, the two-dimensional contact force position and size perception of a large deformation part is realized, and the preparation process is simplified at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of tactile sensing of soft manipulators, and in particular to an omnidirectional adaptive tactile finger based on visual-tactile fusion and a method of using the finger. Background Art

[0002] The sense of touch of a soft hand is different from that of a rigid manipulator. The flexibility, nonlinearity, viscoelasticity, and hysteresis characteristics of the soft hand's own materials and structures pose great obstacles to the realization of its tactile sensing. Currently, the realization of soft touch is basically achieved by embedded means, using conductive liquid, stretchable electrodes, gratings, etc. as tactile media. Under the premise of not affecting the performance of the soft hand itself, by detecting changes in impedance, capacitance, and spectral signals, a certain contact force position and size perception function can be achieved.

[0003] In 2017, Liang Guanhao of Zhejiang University developed a high-resolution, high-sensitivity capacitive tactile sensor array, which realized the three-dimensional contact force perception of the contact unit. The touch unit consists of three parts: an upper contact boss, a middle sensor unit, and a flexible base. The main principle of the sensor unit is to place an elastic medium between the two plates to convert the deformation signal into the capacitance between the plates. Liang established a model between the capacitance value and the force magnitude through theoretical modeling, simulation, and experimental calibration to achieve multi-dimensional contact force measurement. The volume of a single sensor unit is in the millimeter level, so it is very convenient to achieve high-density arrangement and perception. However, the preparation process of the sensor is complex and the range is small, and it needs to be processed by high-priced equipment such as photolithography machines. In 2018, Han et al. of the Korea Advanced Institute of Science and Technology designed a 1×3 distributed impedance sensor array that can identify contact position and positive pressure. They proposed a recursive neural network model - heuristic recursive neural network, and conducted verification experiments on two piezoresistive sensor unit arrays. The results showed that the model trained by this network can well distinguish the position of force and predict the size of positive pressure. In 2019, Massari et al. from the Italian Institute of Biorobotics embedded an optical fiber with four segments of Bragg gratings (FBG) in the middle of a 6mm thick elastic strip to form a 1×4 sensor unit array, and proposed a learning-based sensor system calibration method. When there is an external load, the grating will produce local bending and the wavelength of the transmitted light will shift. The wavelength shift can be obtained by detecting the spectral information at the receiving end through the receiver. In the data acquisition process, Massari et al. used a hybrid form of finite element simulation and physical loading experiments to obtain data pairs of grating reflection spectrum signals and loading information (loading position and loading force), which effectively reduced the amount of experimental data collection and improved data acquisition efficiency. Afterwards, a mapping model between the corresponding four grating wavelength offset data sets and the loading information set was trained through a neural network, achieving continuous prediction of the loading position and loading force in a single dimension. The predicted root mean square errors were 2.45 mm (0-86 mm) and 0.164 N (0-2.5 N), and the determination coefficients were 0.99 and 0.97, respectively. The measurement range is in brackets.

[0004] From the above analysis, it can be seen that embedded sensing technology has been widely studied and applied in the field of soft touch. However, on the one hand, the form of embedded sensing makes the overall preparation process of touch fingers complicated, and it is difficult to achieve mass production. On the other hand, the above research basically assumes that the substrate is rigid, so its sensing part is often limited to the tip of the finger (basically no deformation during work). For parts that have large bends during work, the above sensing methods are difficult to achieve the expected effect. Summary of the invention

[0005] In view of the above-mentioned problems existing in the prior art, the present invention provides an omnidirectional adaptive tactile finger based on vision-touch fusion and a method of using the same, introduces image sensors into the field of soft hand touch, and realizes non-embedded touch through vision-touch fusion based on the principle of intrinsic force perception, thereby overcoming the coupling solution problem of internal and external perception in soft touch, realizing the two-dimensional contact force position and size perception of large deformation parts, and simplifying the preparation process.

[0006] The present invention discloses an omnidirectional adaptive tactile finger based on visual-tactile fusion, comprising: a soft finger body, an image sensor and a bracket;

[0007] The image sensor is installed between the brackets, the brackets are installed at the bottom of the soft finger body and the image sensor is placed just below the soft finger body.

[0008] As a further improvement of the present invention, the soft finger body is a spatial three-dimensional network structure, and the spatial three-dimensional network structure is based on the positions of nodes and uses connecting rods to carry out orderly combination in space.

[0009] As a further improvement of the present invention, the image sensor includes a USB camera.

[0010] As a further improvement of the present invention, the bracket is composed of two half brackets with the same structure, and a receiving cavity for receiving the image sensor is provided in the middle of the bracket; the soft finger body is mounted and fixed to the bracket by bolts.

[0011] The present invention also discloses a method for using an omnidirectional adaptive tactile finger, comprising:

[0012] During the process of the omnidirectional adaptive tactile finger contacting an object, the two-dimensional contact force position and magnitude are predicted based on visual-tactile fusion.

[0013] As a further improvement of the present invention, the prediction of the two-dimensional contact force position and magnitude based on visual-tactile fusion specifically includes:

[0014] Model building:

[0015] Obtain multiple sets of images I, loading height and horizontal displacement (Δx, h) captured by the image sensor during the contact deformation process, and the components of the contact force at the contact position along the horizontal and vertical directions (Fx, Fy), and store them in the sensor image set A, the loading position set B, and the contact force set C respectively;

[0016] Through the calibration method of mechanical loading experiment, a potential-force mapping model f is established from the discrete single mapping of the loading position set B to the contact force set C. BC ;

[0017] Through deep learning methods, a bitmap mapping model f is established from the sensor image set A to the loading position set B. AB ;

[0018] Experimental prediction:

[0019] Based on the image sensor, the sensor image I of the soft finger body is obtained in real time t ;

[0020] Based on the bitmap model f AB and sensor image I t , predict the current loading position (Δx t ,h t );

[0021] Based on the potential mapping model f BC and the current loading position (Δx t ,h t ), predict the current two-dimensional contact force (Fx t ,Fy t ).

[0022] As a further improvement of the present invention, the model establishment includes:

[0023] The omnidirectional adaptive touch finger is mounted on a multi-axis mechanical measurement platform, and a push rod is applied to the soft finger body;

[0024] By measuring the corresponding support reaction force at a given loading height and horizontal displacement (Δx, h), the contact force components (Fx, Fy) of the corresponding contact position along the horizontal and vertical directions are obtained;

[0025] Acquire a sensor image I of the soft finger body through an image sensor;

[0026] Repeat the above steps to obtain the sensor image set A, the loading position set B and the contact force set C.

[0027] As a further improvement of the present invention, in the process of establishing the model,

[0028] Based on the corresponding (Δx, h) and (Fx, Fy) in the loading position set B and the contact force set C, the potential-force mapping model f of the discrete single mapping from the loading position set B to the contact force set C is established by linear interpolation method. BC .

[0029] As a further improvement of the present invention, when establishing the potential mapping model f BC When the deformation area of ​​the soft finger body is divided into N rectangular areas, the average value of the support reaction force of all test points in the rectangular area where (Δx, h) is located is used as the representation of the support reaction force in this area.

[0030] As a further improvement of the present invention, when establishing the bitmap mapping model f AB Before processing, the original image I0 is filtered, denoised, expanded and scaled to obtain image I.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. The present invention adopts a non-embedded structure, with a simple overall configuration and convenient preparation. The entire touch finger only includes four main parts: a soft finger body, a camera, and two identical brackets. The overall assembly can be completed in a few minutes, which is conducive to mass production and promotion of industrial applications.

[0033] 2. The present invention can simultaneously sense the position and size of the two-dimensional contact force at the large deformation part. The above information can be used for shape recognition, slip detection, and enhanced grasping stability during the grasping process. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the explosion structure of an omnidirectional adaptive tactile finger based on visual-tactile fusion disclosed in an embodiment of the present invention;

[0035] Figure 2 A three-dimensional structure diagram of a soft finger body of an omnidirectional adaptive tactile finger based on visual-touch fusion disclosed in an embodiment of the present invention;

[0036] Figure 3 A three-dimensional structural diagram of a bracket of an omnidirectional adaptive tactile finger based on visual-tactile fusion disclosed in an embodiment of the present invention;

[0037] Figure 4 A bracket mechanical measurement platform for omnidirectional adaptive tactile fingers based on visual-tactile fusion disclosed in an embodiment of the present invention;

[0038] Figure 5 A schematic diagram of the principle of an omnidirectional adaptive tactile finger based on visual-tactile fusion disclosed in an embodiment of the present invention;

[0039] Figure 6 A design flow chart of an omnidirectional adaptive tactile finger based on visual-tactile fusion disclosed in an embodiment of the present invention;

[0040] Figure 7 A schematic diagram of the bracket loading area classification of an omnidirectional adaptive tactile finger based on visual-tactile fusion disclosed in an embodiment of the present invention;

[0041] Figure 8 This is a picture preprocessing effect diagram of a bracket of an omnidirectional adaptive tactile finger based on visual-tactile fusion disclosed in an embodiment of the present invention;

[0042] Fig. 9 This is a diagram of the real-time operation system interface of an omnidirectional adaptive tactile finger bracket based on vision-touch fusion disclosed in an embodiment of the present invention.

[0043] In the figure:

[0044] 1. Soft finger body; 2. Bracket; 3. Image sensor; 4, 5. Mounting holes; 6. Countersunk hole. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0047] The present invention provides an omnidirectional adaptive tactile finger based on visual-tactile fusion. Based on this structure, the robot can be helped to complete safe and efficient adaptive grasping based on tactile perception.

[0048] like Figures 1 to 3 As shown, the hardware part of the omnidirectional adaptive tactile finger of the present invention consists of two parts, the upper part is a passive adaptive soft finger body 1, which is also an elastic element for tactile perception; the lower part is mainly composed of a bracket 2 and an image sensor 3.

[0049] The soft finger body 1 of the present invention is a spatial three-dimensional network structure, which is based on the position of the node and uses a connecting rod to perform orderly combination in space. Specifically, the spatial three-dimensional network structure is a first basic unit, a second basic unit, a first basic unit and a plurality of second basic units superimposed, or a plurality of second basic units superimposed; wherein the first basic unit includes a first upper structure and a first lower structure, the first upper structure includes a first node, the first lower structure includes at least three second nodes, and at least three second nodes are not collinear; the first node and all second nodes form a three-dimensional network structure through a connecting rod, and the connecting rod is connected between two second nodes or between the first node and the second node; the second basic unit includes a second upper structure and a second lower structure, the second upper structure includes at least two third nodes, the second lower structure includes at least two fourth nodes, and at least two fourth nodes are not coplanar with at least two third nodes; all third nodes and all fourth nodes form a three-dimensional network structure through a connecting rod, and the connecting rod is connected between two third nodes, between two fourth nodes, or between the third node and the fourth node. Further, for the spatial three-dimensional network structure, please refer to the relevant records of the applicant's existing patents 201910461095X, 2019104610911, 2019104610907, 2019104609933, 2019104609967, 2019104610428, and 201910460542X. Figure 1 , 2 As shown, the soft finger body 1 of the present invention adopts the superposition of multiple second basic units, which can be a grid finger structure of any shape, such as cylindrical, rectangular, etc., which has omnidirectional adaptability and coverage of the target object during the grasping process. Furthermore, the soft finger body 1 is preferably formed by coating with TPU soft material.

[0050] The bracket 2 of the present invention wraps the image sensor 3 and is installed at the bottom of the soft finger body 1, and the image sensor 3 is placed directly below the soft finger body 1. The image sensor 3 can be a commonly used network USB camera; Figure 1 , 3 As shown, the bracket 2 is composed of two half brackets with the same structure, and a receiving cavity for receiving the image sensor 3 is provided in the middle of the bracket 2; the bracket 2 is formed by PLA material using FDM technology, and the soft finger body 1 is installed and fixed to the bracket 2 by bolts.

[0051] The hardware structure of the omnidirectional adaptive touch finger of the present invention is simple, and the preparation and assembly are simple. There are only three steps from processing to assembly, including:

[0052] Step 1: Processing and molding of the soft finger body 1:

[0053] The material of the soft finger body 1 is TPU95, and the processing technology is lamination molding; wherein, four mounting holes 5 with a diameter of 3 mm are provided at the bottom of the finger for connection with the bracket through bolts, and there are 14 mounting holes 4 with a diameter of 2 mm on the surface and fingertips of the finger.

[0054] Step 2: Processing and forming of bracket 2:

[0055] The structure of the bracket 2 is relatively complex, so it is processed and formed using FDM technology, and the material is PLA. After the bracket body is printed, the M3 copper nuts are heated and embedded in the countersunk holes 6 on the upper and lower end surfaces of the bracket.

[0056] Step 3: Component assembly:

[0057] First, install the purchased USB camera into the bracket 2, and fix the left and right brackets 2 with long screws; then fix the soft finger body 1 to the upper end surface of the bracket with bolts to complete the assembly.

[0058] like Figure 5 , 6 As shown, the process of the omnidirectional adaptive touch finger contacting an object of the present invention predicts the position and magnitude of the two-dimensional contact force based on visual-tactile fusion; specifically includes:

[0059] Step 1: Build the hardware environment:

[0060] Design the required omnidirectional adaptive tactile finger structure, shape and materials.

[0061] Step 2: Establishment of mapping model:

[0062] Step 21, obtaining multiple sets of images I, loading height and horizontal displacement (Δx, h) captured by the image sensor during the contact deformation process, and the components of the contact force at the contact position along the horizontal and vertical directions (Fx, Fy), and storing them in the sensor image set A, the loading position set B, and the contact force set C respectively; wherein the loading position and the contact force together constitute the complete external tactile information (Δx, h, Fx, Fy)

[0063] Step 22: Through the calibration method of mechanical loading experiment, a potential force mapping model f is established from the discrete single mapping of the loading position set B to the contact force set C under specific friction and contact curvature radius conditions. BC ;

[0064] Step 23: Use the image sensor to capture the geometric features of the finger itself as a touch signal, and establish an image bit mapping model f of the abstract single mapping from the sensor image set A to the loading position set B through the deep learning method. AB ;

[0065] Step 3: Real-time system establishment

[0066] The real-time system is established by embedding the two mapping models into the cyclically running system to achieve continuous prediction of tactile information and establish a real-time tactile system;

[0067] For example, when contact deformation occurs at a certain time t, the sensor image I of the soft finger body is obtained in real time based on the image sensor. t ; Based on the bitmap model f AB and sensor image I t , predict the current loading position (Δx t ,h t ); Based on the potential mapping model f BC and the current loading position (Δx t ,h t ), predict the current two-dimensional contact force (Fx t ,Fy t ).

[0068] Example:

[0069] Step 1: Build the hardware environment:

[0070] The structure and assembly method of the omnidirectional adaptive touch finger are described above;

[0071] Step 2: Establishment of mapping model:

[0072] 1. Potential Mapping Model

[0073] Through mechanical loading experiments, Figure 4 The multi-axis mechanical measurement platform shown in the figure can obtain a scatter plot of the spatial distribution of (Fx, Fy) in (Δx, h) by measuring the corresponding support reaction force under a given loading condition (Δx, h); through the linear interpolation method, a potential-force mapping model f can be established from the discrete single mapping of the loading position set B to the contact force set C. BC ; With the help of the potential mapping model f BC , by inputting the loading position (Δx, h), the corresponding (Fx, Fy) can be solved;

[0074] 2. Image Mapping Model

[0075] The image bitmap model is constructed by the transfer learning method. Taking the Alexnet network as an example, the construction process of the transfer learning model is explained. The input of the model is an RGB image of size 227×227×3. The entire model contains 8 weight layers, including 5 convolutional layers and 3 fully connected layers, with a total of more than 60 million parameters, and has completed heavy pre-training work on the ImageNet dataset. On the basis of the existing model, the structure and parameters of the first 7 weight layers are retained, and only the last fully connected layer is adjusted. The classification (regression) prediction of the loading position interval is performed to establish the learning model Alexnet-M.

[0076] like Figure 7 As shown in the figure, by dividing the set deformation area into 50 rectangular blocks, 50 sets of image data are evenly collected in each area, and then the image is preprocessed in the HSV domain. The data preprocessing effect is shown in the figure. Figure 8 As shown, the steps are:

[0077] 1) By converting the color image I0 with a pixel size of 1280×720×3 from RGB space to HSV space, we obtain image I h By setting the hue threshold, most of the heterochromatic background can be filtered out; after filtering, the remaining noise is mainly the contact part between the push rod and the finger, which is difficult to filter out due to the reflection of the white push rod. Considering that white objects have good reflectivity, their intensity in the image is generally higher than that of objects of other colors. Therefore, based on the first filtering, we filter by setting the brightness threshold and binarize the image to obtain image I b .

[0078] 2) I b There are still some scattered noise points in the image. By identifying the connected domain in the binary image, the largest connected domain in the image is extracted to filter the scattered noise points in the image. After a small amount of smoothing on the edge of the filtered image, the image I is obtained. b1 .

[0079] 3) For picture I b1 Binary image I b2 Expand the image from a single channel to three channels, each channel is consistent with the original image, and the image I with a size of 1280×720×3 is obtained. b3 .

[0080] 4) In order to meet the requirements of neural networks such as Alexnet-M for the input size of the image, the original image needs to be scaled.

[0081] The preprocessed data were labeled with corresponding category labels according to the loading conditions and organized into data sets. After random classification according to 0.7:0.2:0.1, the training set, validation set and test set were obtained. The prediction accuracy of the trained model in the validation set was 88.20%, and the prediction accuracy in the test set was 90.00%.

[0082] Step 3: Real-time system establishment

[0083] The real-time operation of the system is completed in the Matlab environment of a laptop computer. The basic operation interface is as follows: Fig. 9 The control panel of the whole program is completed by the GUI interface of Matlab. The left half displays the real-time touch image and the predicted information of the contact position, the lower right part displays the predicted force under the current loading state and the frequency of the system operation, and the upper right part is the operation area, which controls the start, end and data acquisition of the system.

[0084] Furthermore, since the position mapping model currently uses a classification algorithm, the predicted loading position is not an accurate point, but a rectangular area distributed in the (Δx, h) space. The specific coordinates of the loading point cannot be determined, so we are currently unable to directly solve the support reaction force of the corresponding point through the potential mapping model. In order to verify the feasibility of this visual-tactile fusion solution, we proposed a temporary and rough support reaction force evaluation method, using the average value of the support reaction force of all test points in the rectangular area as the representation of the support reaction force in the area. We randomly selected 250 position points in the overall loading space for testing. The prediction effect of the two-dimensional contact force is 0.820N for the horizontal (normal) support reaction force Fx, and the determination coefficient is 0.998. For the vertical (tangential) support reaction force, the root mean square error of the prediction Fy is 0.355N, and the determination coefficient is 0.946, which has preliminary use value.

[0085] application:

[0086] The touch finger of the present invention can be fixed on different mounting plates through the mounting hole at the bottom of the bracket to form a soft gripper with different configurations, which can be used to grasp various types of objects. Its perception of the position and size of the contact force helps to realize functions such as sliding monitoring, object shape recognition, and object hardness recognition during the grasping process.

[0087] Slip detection: The tactile finger can sense the position of the contact force. During vertical grasping, the contact height position will change when slip occurs. Therefore, slip detection can be achieved by detecting the contact height h during the grasping process.

[0088] Object shape recognition: When grasping objects of different shapes, the deformation characteristics of the fingers will have different trends. We use the camera to record the deformation of the fingers in the picture, and then use the machine learning algorithm and the picture data for training to achieve the distinction between objects of different shapes.

[0089] Object hardness identification: Objects of different hardness will produce different deformation signals when subjected to the same loading force. Harder objects deform less while softer objects deform more. Based on this information, we can identify the hardness of an object based on the contact depth Δx in the contact position information of the touch finger under a specific loading displacement.

[0090] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An omnidirectional adaptive tactile finger based on visual-tactile fusion, characterized in that: The touch-sensing finger comprises: a soft finger body, an image sensor and a bracket; The soft finger body is made of soft material, the image sensor is installed between the brackets, the brackets are installed at the bottom of the soft finger body and the image sensor is placed directly below the soft finger body; In the process of the omnidirectional adaptive tactile finger contacting the object, the two-dimensional contact force position and magnitude are predicted based on visual-tactile fusion; The method for using the touch-sensitive finger comprises: a) Model establishment: Obtain multiple sets of images I, loading height and horizontal displacement (Δx, h) captured by the image sensor during the contact deformation process, and the components of the contact force at the contact position along the horizontal and vertical directions (Fx, Fy), and store them in the sensor image set A, the loading position set B, and the contact force set C respectively; Through the calibration method of mechanical loading experiment, a potential-force mapping model f is established from the discrete single mapping of the loading position set B to the contact force set C. BC ; Through deep learning methods, a bitmap mapping model f is established from the sensor image set A to the loading position set B. AB ; b) Establishment of real-time touch system: Based on the image sensor, the sensor image I of the soft finger body is obtained in real time t ; Based on the bitmap model f AB and sensor image I t , predict the current loading position (Δxt,ht); Based on the potential mapping model f BC and the current loading position (Δx t ,h t ), predict the current two-dimensional contact force (Fx t ,Fy t ).

2. The omnidirectional adaptive tactile finger according to claim 1, characterized in that: The soft finger body is a spatial three-dimensional network structure, and the spatial three-dimensional network structure is based on the positions of nodes and is orderly combined in space using connecting rods.

3. The omnidirectional adaptive tactile finger according to claim 1, characterized in that: The image sensor includes a USB camera.

4. The omnidirectional adaptive tactile finger according to any one of claims 1 to 3, characterized in that: The bracket is composed of two half brackets with the same structure. A receiving cavity for receiving the image sensor is provided in the middle of the bracket. The soft finger body is fixed to the bracket by bolts.

5. The omnidirectional adaptive tactile finger according to claim 1, characterized in that: The model establishment includes: The omnidirectional adaptive touch finger is mounted on a multi-axis mechanical measurement platform, and a push rod is applied to the soft finger body; By measuring the corresponding support reaction force at a given loading height and horizontal displacement (Δx, h), the contact force components (Fx, Fy) of the corresponding contact position along the horizontal and vertical directions are obtained; The sensor image I of the soft finger body is obtained through the image sensor.

6. The omnidirectional adaptive tactile finger according to claim 1, characterized in that: In the process of model building, Based on the corresponding (Δx, h) and (Fx, Fy) in the loading position set B and the contact force set C, the potential-force mapping model f of the discrete single mapping from the loading position set B to the contact force set C is established by linear interpolation method. BC .

7. The omnidirectional adaptive tactile finger according to claim 1, characterized in that: In establishing the potential mapping model f BC When the deformation area of ​​the soft finger body is divided into N rectangular areas, the average value of the support reaction force of all test points in the rectangular area where (Δx, h) is located is used as the representation of the support reaction force in this area.

8. The omnidirectional adaptive tactile finger according to claim 1, characterized in that: In establishing the bitmap model AB Before filtering, denoising, expanding and scaling the original image I0, an image is obtained.

Images