A Tactile Image Super-Resolution Reconstruction Method and Acquisition System

Abstract

The present invention relates to a method and acquisition system for super-resolution reconstruction of tactile images, belonging to the field of tactile perception. First, a high-resolution tactile image sample is obtained by using a low-resolution tactile sensor, then a deep learning-based tactile super-resolution model is trained using a tactile super-resolution dataset, and finally, the tactile data of the contact surface to be measured is reconstructed into a super-resolution tactile image by using the tactile super-resolution model. The present invention only uses the existing low-resolution tactile sensors based on tactile sensing units, adopts the deep learning-based tactile super-resolution reconstruction technology, can effectively restore the shape of the contact surface, and while improving the resolution of the tactile sensor, still maintains the characteristics of the sensor being lightweight, flexible, and easy to integrate into devices such as robots.

Description

Technical Field

[0001] The present invention relates to the field of tactile perception, and in particular to a method and an acquisition system for super-resolution reconstruction of tactile images. Background Art

[0002] A tactile sensor is a sensor that can collect tactile information such as the distribution of forces on the contact surface and surface texture. Currently, the mainstream tactile sensors can be divided into two types. The first type is the vision-based tactile sensor, and the second type is the taxel-based tactile sensor. The vision-based tactile sensor captures the deformation of the contact surface as tactile information through a camera. Typical products include the Gelsight tactile sensor of the Massachusetts Institute of Technology, the TacTip tactile sensor of the University of Bristol, and the DIGIT tactile sensor of Meta. Since the camera requires a certain amount of space, the volumes of these sensors are relatively bulky, making it difficult to integrate them into devices such as robots and not conforming to human tactile perception behavior. The taxel-based tactile sensor combines multiple tactile sensing units (taxels) together, and each sensing unit represents the tactile information within a certain area. The distance between two adjacent tactile sensing units is defined as the tactile resolution of the sensor. Typical products include the uSkin triaxial force tactile sensor of Xela Robotic and the Contactile tactile sensor of Contactile. This type of tactile sensor can directly measure the distribution of force and displacement information during the contact process. However, due to current manufacturing process limitations, the resolution of the sensor (the distance between two tactile sensing units) is much lower than that of the vision-based tactile sensor. Moreover, as the resolution of the sensor increases, a series of problems will also arise, such as more connecting wires, lower response frequencies, and amplified crosstalk between devices.

[0003] Super Resolution (SR) refers to a technology that can restore low-resolution (LR) data to high-resolution (HR) data. It has been studied and applied in the fields of computer vision (image super-resolution, video super-resolution) and audio (audio super-resolution), but there is little research in the tactile field. The tactile super-resolution technology can obtain high-resolution tactile information only using existing low-resolution tactile sensors, enabling tactile sensors based on tactile sensor units to obtain high-resolution tactile information while still ensuring the characteristics of being lightweight, flexible, and easy to integrate into robotic devices.

[0004] Therefore, how to use super-resolution technology to improve the resolution of tactile information obtained by tactile sensors is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The object of the present invention is to provide a tactile image super-resolution reconstruction method and acquisition system to improve the tactile information resolution of a tactile sensor.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A tactile image super-resolution reconstruction method, comprising:

[0008] Using a tactile sensor to sample each contact surface multiple times, obtaining tactile data of each contact surface at each sampling and the coordinates of the center point of the tactile sensor;

[0009] According to the coordinates of the center point of the tactile sensor at each sampling, all sampled tactile data of each contact surface are aligned to obtain a high-resolution tactile image sample of the corresponding contact surface;

[0010] The tactile data sampled each time in the central area of ​​the contact surface and the high-resolution tactile image samples of the corresponding contact surface are selected to form a data pair to construct a tactile super-resolution dataset;

[0011] According to the tactile super-resolution dataset, a supervised machine learning method is used to train a deep learning network model to obtain a tactile super-resolution model;

[0012] The tactile data of the contact surface to be measured collected by the tactile sensor is input into the tactile super-resolution model, and a high-resolution tactile image of the contact surface to be measured is output.

[0013] Optionally, the method of using a tactile sensor to sample each contact surface multiple times to obtain tactile data of each contact surface at each sampling and the coordinates of the center point of the tactile sensor specifically includes:

[0014] Set the number of acquisitions I along the X-axis and the number of acquisitions J along the Y-axis of the robot arm;

[0015] According to the resolution of the tactile sensor, the number of acquisitions I and the number of acquisitions J, a preset distance Δx for one movement along the X-axis and a preset distance Δy for one movement along the Y-axis are determined respectively;

[0016] Mount the tactile sensor on the end of the robotic arm;

[0017] Preset sampling position (x, y, h init )=(0,0,h init ), where h init is the vertical distance from the tactile sensor to the contact surface;

[0018] Move the robotic arm with the tactile sensor to the sampling position (x, y, h) above the contact surface init ), and at the same time adjust the Z-axis at the end of the robotic arm to be perpendicular to the horizontal plane, and record the central point coordinates of the tactile sensor;

[0019] Control the robotic arm to move downward until the tactile sensor contacts the contact surface, and collect multiple groups of tactile data;

[0020] The robotic arm moves upward and returns to the sampling position (x, y, h init ), completing the th sampling;

[0021] Let x increase by Δx, and replace (x + Δx, y, h init ) with (x, y, h init ), and return to the step "Move the robotic arm with the tactile sensor to the sampling position (x, y, h init ) above the contact surface, and at the same time adjust the Z-axis at the end of the robotic arm to be perpendicular to the horizontal plane", until Control the robotic arm to return to the sampling position (x, y, h init );

[0022] Let y increase by Δy, and replace (x, y + Δy, h init ) with (x, y, h init ), and return to the step "Move the robotic arm with the tactile sensor to the sampling position (x, y, h init ) above the contact surface, and at the same time adjust the Z-axis at the end of the robotic arm to be perpendicular to the horizontal plane", until Stop sampling, and obtain the tactile data at each sampling of the contact surface and the central point coordinates of the tactile sensor.

[0023] Optionally, according to the resolution of the tactile sensor, the number of acquisitions I, and the number of acquisitions J, respectively determine the preset distance Δx for moving once along the X-axis and the preset distance Δy for moving once along the Y-axis, specifically including:

[0024] According to the resolution of the tactile sensor and the number of acquisitions I, use the formula d = Δx × I to determine the preset distance Δx for moving once along the X-axis; where d represents the distance between adjacent tactile sensing units on the tactile sensor;

[0025] According to the resolution of the tactile sensor and the number of acquisitions J, use the formula d = Δy × J to determine the preset distance Δy for moving once along the Y-axis.

[0026] Optionally, according to the central point coordinates of the tactile sensor at each sampling, register the tactile data of all samplings of each contact surface to obtain a high-resolution tactile image sample corresponding to the contact surface, specifically including:

[0027] Average the multiple groups of tactile data collected in each sampling after high-pass filtering, and use the average value as the tactile data for each sampling;

[0028] According to the tactile data of each sampling on each contact surface and the central point coordinates of the tactile sensor, use the formula T HR [I×m+i,J×n+j]=T i,j [m,n] for registration to obtain an initial high-resolution tactile image sample; where, [m,n] represents the [m,n]th tactile sensing unit of the tactile sensor, and T HR [I×m+i,J×n+j] represents the x, y, and z-axis data corresponding to the [I×m+i, J×n+j]th pixel of the high-resolution tactile image, and T i，j [m, n] represents the x, y, and z-axis tactile data corresponding to the [m, n]th tactile sensing unit when the tactile sensor collects the i-th time along the X-axis and the j-th time along the Y-axis;

[0029] Perform smoothing processing on the initial high-resolution tactile image sample to obtain a high-resolution tactile image sample of the contact surface.

[0030] Optionally, select the tactile data of each sampling in the central area of the contact surface and the high-resolution tactile image sample corresponding to the contact surface to form a data pair, and construct a tactile super-resolution data set, specifically including:

[0031] Select the tactile data collected by each contact surface within and range as input, and use the high-resolution tactile image sample of each contact surface as a label to construct a tactile super-resolution data set;

[0032] where, represents rounding down, K represents the coefficient at which the low-resolution and high-resolution tactile data can ignore errors within the preset range; the larger K is, the higher the tolerance for errors, and generally 4 or 6 is taken in the experiment.

[0033] Optionally, the deep learning network model is a deep learning model based on a convolutional neural network or a deep learning model based on a generative adversarial network;

[0034] Both the deep learning model based on a convolutional neural network and the deep learning model based on a generative adversarial network include an upsampling layer, a feature extraction layer, and an output layer connected in sequence;

[0035] The loss function of the deep learning model based on a convolutional neural network is where, (M, N) represents the resolution of the tactile sensor, Represents the loss of a deep learning model based on a convolutional neural network. Represents the mean squared error. Represents the high-resolution tactile image predicted by the model, T HR Represents the true high-resolution tactile image;

[0036] The loss function of the deep learning model based on the generative adversarial network is Where Represents the loss of a deep learning model based on the generative adversarial network. Represents the adversarial loss term. Represents the original low-resolution tactile data of the x, y, and z axes collected by the tactile sensor. Represents the high-resolution tactile image generated (predicted) by the generative network. Represents the gap between the high-resolution tactile image predicted by the discriminative network and the true high-resolution tactile image.

[0037] A tactile image super-resolution acquisition system, the acquisition system includes: a robotic arm, a tactile sensor, and a host computer;

[0038] The tactile sensor is installed at the end of the robotic arm;

[0039] The control end and the signal output end of the robotic arm are both connected to the host computer; the host computer is used to receive the center point coordinates of the tactile sensor and control the robotic arm to move along a preset acquisition path according to the center point coordinates of the tactile sensor;

[0040] The signal output end of the tactile sensor is connected to the host computer; the host computer is used to receive the tactile data of the contact surface collected by the tactile sensor after the tactile sensor contacts the contact surface, and reconstruct the high-resolution tactile image of the contact surface according to the tactile data and the center point coordinates of the tactile sensor.

[0041] Optionally, the host computer includes:

[0042] A sampling module, used to obtain the tactile data and the center point coordinates of the tactile sensor during each sampling of each contact surface;

[0043] A label obtaining module, used to register the tactile data of all samplings of each contact surface according to the center point coordinates of the tactile sensor during each sampling, and obtain a high-resolution tactile image sample corresponding to the contact surface;

[0044] A dataset construction module, used to select the tactile data during each sampling within the central area of the contact surface and the high-resolution tactile image sample corresponding to the contact surface to form a data pair, and construct a tactile super-resolution dataset;

[0045] A training module, configured to train a deep learning network model by using a supervised machine learning method according to the tactile super-resolution data set, so as to obtain a tactile super-resolution model;

[0046] An application module, configured to input the tactile data of the contact surface to be measured collected by a tactile sensor into the tactile super-resolution model, and output a high-resolution tactile image of the contact surface to be measured.

[0047] According to the specific embodiments provided by the present invention, the following technical effects are disclosed:

[0048] The present invention discloses a method and a collection system for super-resolution reconstruction of tactile images. First, a high-resolution tactile image sample is obtained by using a low-resolution tactile sensor, then a tactile super-resolution model based on deep learning is trained by using a tactile super-resolution data set, and finally, the tactile data of the contact surface to be measured is reconstructed into a super-resolution tactile image by using the tactile super-resolution model. The present invention only uses an existing low-resolution tactile sensor based on a tactile sensing unit, and adopts a tactile super-resolution reconstruction technology based on deep learning, which can effectively restore the shape of the contact surface. While improving the resolution of the tactile sensor, it still maintains the characteristics of being light, flexible, and easy to be integrated into devices such as robots. Description of the Drawings

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0050] Figure 1 It is a flowchart of the method for super-resolution reconstruction of tactile images provided in Embodiment 1 of the present invention;

[0051] Figure 2 It is a schematic diagram of the principle of the method for super-resolution reconstruction of tactile images provided in Embodiment 1 of the present invention;

[0052] Figure 3 It is a schematic diagram of the single-touch data sampling process provided in Embodiment 1 of the present invention; Figure 3 (a) is a schematic diagram when the end of the robotic arm is at the collection position, Figure 3 (b) is a schematic diagram when the tactile sensor is in contact with the contact surface, Figure 3 (c) is a schematic diagram of the movement of the robotic arm after collection, Figure 3 (d) is a schematic diagram of the robotic arm moving a preset distance Δx along the X axis;

[0053] Figure 4It is the multi - sampling robotic arm motion trajectory diagram provided by Embodiment 1 of the present invention;

[0054] Figure 5 It is the schematic diagram for obtaining high - resolution tactile image samples provided by Embodiment 1 of the present invention;

[0055] Figure 6 It is the framework diagram of the deep - learning model based on convolutional neural network provided by Embodiment 1 of the present invention;

[0056] Figure 7 It is the discriminant network framework diagram of the deep - learning model based on generative adversarial network provided by Embodiment 1 of the present invention;

[0057] Figure 8 It is the schematic diagram of the image registration process provided by Embodiment 1 of the present invention;

[0058] Figure 9 It is the structural diagram of the tactile image super - resolution acquisition system provided by Embodiment 2 of the present invention. Detailed implementation manners

[0059] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0060] The purpose of the present invention is to provide a tactile image super - resolution reconstruction method and an acquisition system to improve the tactile information resolution of tactile sensors.

[0061] To make the above - mentioned objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0062] Embodiment 1

[0063] The embodiment of the present invention provides a tactile image super - resolution reconstruction method, as Figure 1-2 shown, including the following steps:

[0064] Step S1, use a tactile sensor to perform multiple samplings on each contact surface to obtain the tactile data at each sampling of each contact surface and the central point coordinates of the tactile sensor.

[0065] Use a tactile sensor to perform multiple samplings. Each time of sampling, move a small distance along the X - axis (Y - axis), and record the tactile data and the central point coordinates of the sensor throughout the process.

[0066] Collect tactile data by moving the robotic arm and obtain a tactile sequence through multiple samplings. The entire collection process is as follows:

[0067] Step 1: Set the number of samplings I along the X-axis and the number of samplings J along the Y-axis for the robotic arm.

[0068] Step 2: Determine the preset distance Δx for one movement along the X-axis and the preset distance Δy for one movement along the Y-axis respectively according to the resolution of the tactile sensor, the number of samplings I, and the number of samplings J.

[0069] Assume the original resolution of the tactile sensor is (M, N), that is, the tactile sensor has M×N independent tactile sensing units.

[0070] According to the resolution of the tactile sensor and the number of samplings I, use the formula d = Δx×I to determine the preset distance Δx for one movement along the X-axis; where d represents the distance between adjacent tactile sensing units on the tactile sensor.

[0071] According to the resolution of the tactile sensor and the number of samplings J, use the formula d = Δy×J to determine the preset distance Δy for one movement along the Y-axis.

[0072] Step 3: Install the tactile sensor at the end of the robotic arm.

[0073] Step 4: Preset the sampling position (x, y, h init ) = (0, 0, h init ). Where h init is the vertical distance from the tactile sensor to the contact surface.

[0074] Step 5: Move the robotic arm to drive the tactile sensor to the sampling position (x, y, h init ) above the contact surface. At the same time, adjust the Z-axis at the end of the robotic arm to be perpendicular to the horizontal plane and record the coordinates of the center point of the tactile sensor, as shown in Figure 3 (a).

[0075] Step 6: Control the robotic arm to move downward until the tactile sensor contacts the contact surface and collect multiple groups of tactile data. When the sensor contacts the contact surface and reaches the set threshold, the sensor stops moving for a period of time to ensure that enough data during contact is collected, as shown in Figure 3 (b).

[0076] Step 7: The robotic arm moves upward and returns to the sampling position (x, y, h init ), completing the th sampling, and record the tactile sequence of the entire collection process, as shown in Figure 3 (c).

[0077] Step 8: Let x increase by Δx and set (x + Δx, y, hinit ) Replace (x, y, h init ), and return to the step "Make the robotic arm drive the tactile sensor to move to the sampling position (x, y, h init ) above the contact surface, and at the same time adjust the Z-axis at the end of the robotic arm to be perpendicular to the horizontal plane", until Control the robotic arm to return to the sampling position (x, y, h init ), as Figure 3 shown in (d).

[0078] The ninth step: Let y increase by Δy, and replace x, y + Δy, h init ) with (x, y, h init ), and return to the step "Make the robotic arm drive the tactile sensor to move to the sampling position (x, y, h init ) above the contact surface, and at the same time adjust the Z-axis at the end of the robotic arm to be perpendicular to the horizontal plane", until Stop sampling and obtain the tactile data and the central point coordinates of the tactile sensor each time the contact surface is sampled.

[0079] That is, the robotic arm moves to the initial position, and the Z-axis at the end of the robotic arm is perpendicular to the horizontal plane; the robotic arm moves slowly downward, and the sensor contacts the contact surface. When the set threshold is reached, the sensor stops moving for a period of time to ensure that enough data during contact is collected; the robotic arm moves upward to the initial position. Thus, the sampling process for the [0, 0]th time is completed, and the tactile sequence of the entire collection process is recorded; the robotic arm moves a small distance Δx (Δy) along the X-axis (Y-axis) for the next collection. At this time, the position information of the central point of the sensor (x1, y1, h init ) is recorded, and the above-mentioned steps are repeated until the sampling times are reached and stopped. The entire sampling movement trajectory is as Figure 4 shown, sampling along a Z-shaped trajectory.

[0080] Step S2: According to the central point coordinates of the tactile sensor each time sampling is performed, register the tactile data sampled for each contact surface to obtain a high-resolution tactile image sample corresponding to the contact surface.

[0081] According to the collected data and the sensor position, splice the low-resolution tactile images into a high-resolution tactile image. Assume that the original sensor resolution is (M, N), that is, the sensor has M × N independent tactile sensing units. The sensor collects data I times along the X-axis and J times along the Y-axis. By processing and registering the tactile data during this process, a tactile image with a resolution of (I × M, J × N) can be obtained.

[0082] The specific process is as follows:

[0083] Step 1: Preprocess the tactile sequence: After high-pass filtering, average multiple groups of tactile data collected in each sampling, and use the average value as the tactile data for each sampling.

[0084] Considering a single sampling process, we can utilize to determine how many times the sensor has collected data along the X-axis and Y-axis respectively at this time; use high-pass filtering to obtain the tactile information corresponding to the contact moment and take the average to eliminate the influence of the self-noise of the tactile sensor. Finally, obtain the low-resolution tactile data T i，j ∈R M×N . Repeating the above operations can obtain the low-resolution tactile data T i，j for the entire multiple-sampling process, where i = {0, 1,..., I} and j = {0, 1,..., J}. Here, i and j represent the relative offset distances of the sensor center from the initial acquisition position.

[0085] Step 2: Perform registration using the position information of the low-resolution tactile images. Figure 8 The registration process is given. According to the tactile data of each contact surface in each sampling and the center point coordinates of the tactile sensor, use the formula T HR [I×m + i, J×n + j] = T i，j [m, n] for registration to obtain the initial high-resolution tactile image sample; where [m, n] represents the [m, n]th tactile sensing unit of the tactile sensor, and T HR [I×m + i, J×n + j] represents the x, y, and z-axis data corresponding to the pixel point [I×m + i, J×n + j] of the high-resolution tactile image, and T i，j [m, n] represents the x, y, and z-axis tactile data corresponding to the [m, n]th tactile sensing unit when the tactile sensor collects data for the i-th time along the X-axis and the j-th time along the Y-axis;

[0086] Step 3: Smooth the initial high-resolution tactile image sample to obtain the high-resolution tactile image sample of each contact surface. Since each sensing unit of the tactile sensor has its own characteristic curve, the registered image has an obvious block effect, and this phenomenon can be significantly eliminated through the smoothed image.

[0087] Refer to Figure 5 , left: the original tactile sequence; middle: the tactile data (low resolution) of each contact surface; right: the registered high-resolution data, only one. That is, I×J low-resolution images LR are registered into one high-resolution image HR.

[0088] In step S3, select the original tactile data (i.e., low-resolution data) near the center of the contact surface and the corresponding high-resolution tactile image to form a data pair, and construct a tactile super-resolution dataset;

[0089] A high - resolution tactile data is registered from I×J original low - resolution tactile data, so one high - resolution image corresponds to I×J low - resolution images. Since there is a certain gap in the contact area represented by the registered high - resolution tactile image and the low - resolution tactile image, they cannot be directly used as a data pair. During the registration process, the low - resolution images at the edge positions have a large difference from the high - resolution images (that is, i is close to 0 or I, and j is close to 0 or J), while the images in the middle positions have a smaller difference (when , the error is 0). We consider that this error can be ignored in the middle positions and select A total of K×K low - resolution tactile images and high - resolution images are used as data pairs. Among them, represents rounding down, and K represents the coefficient of the error that can be ignored for low - resolution and high - resolution tactile data within a preset range; the larger K is, the higher the tolerance for error. Generally, 4 or 6 is taken in the experiment.

[0090] This method can use a tactile sensor with an original resolution of (M, N) to obtain a tactile image with a resolution of (I×M, J×N), that is, the more sampling times, the higher the obtained resolution, but at the same time, the higher the accuracy requirement for the robotic arm.

[0091] Step S4, according to the tactile super - resolution data set, use a supervised machine - learning method to train a deep - learning network model to obtain a tactile super - resolution model.

[0092] This patent presents two super - resolution models based on deep learning to illustrate the feasibility of this method. Any method based on supervised machine learning can be applied to this model to improve the tactile super - resolution performance.

[0093] (1) Tactile super - resolution model based on convolutional neural network

[0094] As Figure 6 the generation network part of, the model includes three parts: an up - sampling layer, a feature extraction layer, and an output layer. First, the input low - resolution data is up - sampled to make its dimension consistent with the high - resolution data, then the image features are extracted, and finally the output layer is used to adjust the data dimension. The mean squared error (MSE) is used as the loss function of the model. The specific loss function is as follows:

[0095]

[0096] Among them represents the high - resolution tactile image predicted by the model. This loss function compares the differences between each pixel point of the true value and the predicted value.

[0097] (2) Tactile super - resolution model based on generative adversarial network

[0098] This model is consistent with the prediction model based on the convolutional model network. The difference lies in adding an adversarial loss term to the loss function. The specific loss is as follows:

[0099]

[0100] Where represents the adversarial loss term, such as Figure 7 the discriminative network part of. That is, the high-resolution tactile data predicted by the generation network and the real high-resolution tactile data are input into the discriminative network, allowing the network to distinguish which is the real data, so as to achieve the purpose of optimization.

[0101] Step S5: Input the tactile data of the contact surface to be measured collected by the tactile sensor into the tactile super-resolution model, and output the high-resolution tactile image of the contact surface to be measured.

[0102] The original low-resolution tactile image collected by the tactile sensor at the current moment is input into the super-resolution model. The model outputs the corresponding high-resolution tactile image according to the mapping relationship learned before, realizing real-time tactile image super-resolution. Figure 2 The entire process of tactile super-resolution is given.

[0103] To solve the problem of too low resolution of the tactile sensor based on the tactile sensing unit, the present invention provides a tactile super-resolution reconstruction technology based on deep learning, which can effectively restore the shape of the contact surface and improve the resolution of the tactile sensor. This technology specifically includes a method for obtaining high-resolution tactile data using a low-resolution tactile sensor and tactile image super-resolution based on deep learning, which are composed of two parts.

[0104] The present invention can obtain high-resolution tactile data only by using the existing low-resolution tactile sensor based on the tactile sensing unit. While improving the sensor resolution, it still maintains the characteristics of the sensor being lightweight, flexible, and easy to integrate into devices such as robots.

[0105] Embodiment 2

[0106] Embodiment 2 of the present invention provides a tactile image super-resolution acquisition system, as Figure 9 shown. The acquisition system includes: a robotic arm, a tactile sensor, and a host computer.

[0107] The tactile sensor is installed at the end of the robotic arm. The control end and the signal output end of the robotic arm are both connected to the host computer; the host computer is used to receive the central point coordinates of the tactile sensor and control the robotic arm to move along a preset acquisition path according to the central point coordinates of the tactile sensor. The signal output end of the tactile sensor is connected to the host computer; the host computer is used to receive the tactile data of the contact surface collected by the tactile sensor after the tactile sensor contacts the contact surface, and reconstruct a high-resolution tactile image of the contact surface according to the tactile data and the central point coordinates of the tactile sensor.

[0108] Exemplarily, the host computer includes: a sampling module, a label obtaining module, a dataset construction module, a training module, and an application module.

[0109] The sampling module is used to obtain the tactile data and the central point coordinates of the tactile sensor during each sampling of each contact surface. The label obtaining module is used to register the tactile data of all samplings of each contact surface according to the central point coordinates to obtain a high-resolution tactile image sample of the corresponding contact surface. The dataset construction module is used to select the tactile data of each sampling within the central area of the contact surface and the high-resolution tactile image sample of the corresponding contact surface to form a data pair and construct a tactile super-resolution dataset. The training module is used to train a deep learning network model by using a supervised machine learning method according to the tactile super-resolution dataset to obtain a tactile super-resolution model. The application module is used to input the tactile data of the contact surface to be measured collected by the tactile sensor into the tactile super-resolution model and output a high-resolution tactile image of the contact surface to be measured.

[0110] The tactile image super-resolution acquisition system of the present invention is used to sample the contact surface multiple times and record data. The tactile sensor is installed at the end of the robotic arm, and the tactile data is collected by the movement of the robotic arm.

[0111] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other.

[0112] Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for super-resolution reconstruction of tactile images, characterized in that, Including: Using a tactile sensor to sample each contact surface multiple times to obtain the tactile data and the central point coordinates of the tactile sensor at each sampling of each contact surface; Registering the tactile data of all samplings of each contact surface according to the central point coordinates of the tactile sensor at each sampling to obtain a high-resolution tactile image sample corresponding to the contact surface; Selecting the tactile data at each sampling within the central area of the contact surface and the high-resolution tactile image sample corresponding to the contact surface to form a data pair, and constructing a tactile super-resolution data set; According to the tactile super-resolution data set, training a deep learning network model using a supervised machine learning method to obtain a tactile super-resolution model; Inputting the tactile data of the contact surface to be measured collected by the tactile sensor into the tactile super-resolution model, and outputting a high-resolution tactile image of the contact surface to be measured.

2. The tactile image super-resolution reconstruction method according to claim 1, wherein The step of using a tactile sensor to sample each contact surface multiple times to obtain the tactile data and the central point coordinates of the tactile sensor at each sampling of each contact surface specifically includes: Setting the number of samplings I along the X-axis and the number of samplings J along the Y-axis of the robotic arm; According to the resolution of the tactile sensor, the number of samplings I, and the number of samplings J, respectively determining the preset distance Δx for moving once along the X-axis and the preset distance Δy for moving once along the Y-axis; Installing the tactile sensor at the end of the robotic arm; Preset sampling position (x, y, h init ) = (0, 0, h init ); where h init is the vertical distance from the tactile sensor to the contact surface; Move the robotic arm with the tactile sensor to the sampling position (x, y, h init ) above the contact surface, and at the same time adjust the Z-axis at the end of the robotic arm to be perpendicular to the horizontal plane, and record the central point coordinates of the tactile sensor; Controlling the robotic arm to move downward until the tactile sensor contacts the contact surface, and collecting multiple groups of tactile data; The robotic arm moves upward and returns to the sampling position (x, y, h init ), completing the th sampling; Let x increase by Δx, and replace (x + Δx, y, h init ) with (x, y, h init ), and return to the step "Move the robotic arm with the tactile sensor to the sampling position (x, y, h init ) above the contact surface, and at the same time adjust the Z-axis at the end of the robotic arm to be perpendicular to the horizontal plane", until Control the robotic arm to return to the sampling position (x, y, h init ); Let y increase by Δy, and replace (x, y + Δy, h init ) with (x, y, h init ). Return to the step "Make the robotic arm drive the tactile sensor to move to the sampling position (x, y, h init ) above the contact surface, and at the same time adjust the Z-axis at the end of the robotic arm to be perpendicular to the horizontal plane", until Sampling stops, and the tactile data at each sampling of the contact surface and the center point coordinates of the tactile sensor are obtained.

3. The tactile image super-resolution reconstruction method according to claim 2, characterized in that The step of respectively determining the preset distance Δx for moving once along the X-axis and the preset distance Δy for moving once along the Y-axis according to the resolution of the tactile sensor, the number of samplings I, and the number of samplings J specifically includes: According to the resolution of the tactile sensor and the number of samplings I, using the formula d = Δx×I to determine the preset distance Δx for moving once along the X-axis; where d represents the distance between adjacent tactile sensing units on the tactile sensor; According to the resolution of the tactile sensor and the number of samplings J, using the formula d = Δy×J to determine the preset distance Δy for moving once along the Y-axis.

4. The tactile image super-resolution reconstruction method according to claim 2, characterized in that The step of registering the tactile data of all samplings of each contact surface according to the central point coordinates of the tactile sensor at each sampling to obtain a high-resolution tactile image sample corresponding to the contact surface specifically includes: Taking the average of the multiple groups of tactile data collected at each sampling after high-pass filtering, and using the average value as the tactile data at each sampling; According to the tactile data sampled each time for each contact surface and the center point coordinates of the tactile sensor, using the formula T HR [I×m+i,J×n+j] = T i,j [m,n] for registration to obtain an initial high-resolution tactile image sample; where [m,n] represents the [m,n]th tactile sensing unit of the tactile sensor, and T HR [I×m+i,J×n+j] represents the x, y, and z axis data corresponding to the [I×m+i,J×n+j]th pixel point of the high-resolution tactile image, and T i，j [m,n] represents the x, y, and z axis tactile data corresponding to the [m,n]th tactile sensing unit when the tactile sensor collects the i-th time along the X axis and the j-th time along the Y axis; Performing smoothing processing on the initial high-resolution tactile image sample to obtain a high-resolution tactile image sample of the contact surface.

5. The tactile image super-resolution reconstruction method according to claim 4, characterized in that, The The step of selecting the tactile data at each sampling within the central area of the contact surface and the high-resolution tactile image sample corresponding to the contact surface to form a data pair, and constructing a tactile super-resolution data set specifically includes: Select the tactile data collected for each contact surface within and as the input, and use the high-resolution tactile image samples of each contact surface as labels to construct a tactile super-resolution dataset; Among them, represents rounding down, and K represents the error coefficient.

6. The tactile image super-resolution reconstruction method according to claim 5, characterized in that, The deep learning network model is a deep learning model based on a convolutional neural network or a deep learning model based on a generative adversarial network; Both the deep learning model based on a convolutional neural network and the deep learning model based on a generative adversarial network include an upsampling layer, a feature extraction layer, and an output layer connected in sequence; The loss function of the deep learning model based on a convolutional neural network is Among them, (M, N) represents the resolution of the tactile sensor, represents the loss of the deep learning model based on the convolutional neural network, represents the mean square error, represents the high-resolution tactile image predicted by the model, T HR represents the true high-resolution tactile image; The loss function of the deep learning model based on a generative adversarial network is Among them, represents the loss of the deep learning model based on the generative adversarial network, represents the adversarial loss term, represents the original low-resolution tactile data of the x, y, and z axes collected by the tactile sensor, represents the high-resolution tactile image generated by the generation network, represents the gap between the high-resolution tactile image predicted by the discriminator network and the real high-resolution tactile image.

7. A tactile image super-resolution acquisition system, characterized in that, The acquisition system includes: a robotic arm, a tactile sensor, and a host computer; The tactile sensor is installed at the end of the robotic arm; The control end and the signal output end of the robotic arm are both connected to the host computer; the host computer is configured to receive the central point coordinates of the tactile sensor and control the robotic arm to move along a preset acquisition path according to the central point coordinates of the tactile sensor; The signal output end of the tactile sensor is connected to the host computer; the host computer is configured to receive the tactile data of the contact surface collected by the tactile sensor after the tactile sensor contacts the contact surface, and reconstruct a high-resolution tactile image of the contact surface according to the tactile data and the central point coordinates of the tactile sensor; The host computer includes: A sampling module, configured to obtain the tactile data and the central point coordinates of the tactile sensor during each sampling of each contact surface; A label acquisition module, configured to register the tactile data of all samplings of each contact surface according to the central point coordinates of the tactile sensor during each sampling, and obtain a high-resolution tactile image sample corresponding to the contact surface; A dataset construction module, configured to select the tactile data during each sampling within the central region of the contact surface and the high-resolution tactile image sample corresponding to the contact surface to form a data pair, and construct a tactile super-resolution dataset; A training module, configured to train a deep learning network model by using a supervised machine learning method according to the tactile super-resolution dataset, and obtain a tactile super-resolution model; An application module, configured to input the tactile data of the contact surface to be measured collected by the tactile sensor into the tactile super-resolution model, and output a high-resolution tactile image of the contact surface to be measured.

Images