Super-resolution tactile perception method and system

Abstract

The invention relates to a super-resolution tactile perception method and system. The super-resolution tactile perception method comprises the following steps: acquiring photoelectric array data for training and high-resolution tactile data for training corresponding to the photoelectric array data; training a tactile sensing data generation model by using the photoelectric array data for training and the high-resolution tactile data for training; obtaining photoelectric array data; and processing the photoelectric array data by using the tactile sensor data generation model to obtain corresponding high-resolution tactile data. According to the super-resolution tactile sensing method, a tactile sensing data generation model is designed based on the idea of a diffusion model, high-resolution tactile data matched with input low-resolution photoelectric array data can be generated through the input low-resolution photoelectric array data, and the possibility that a low-density sensor obtains the high-resolution tactile data is achieved; the touch information acquisition capability with lower power consumption, less hardware, lower cost, higher resolution and lower system complexity is realized.

Description

Technical Field

[0001] The present invention relates to the field of robot tactile interaction and intelligent perception, and particularly to a super-resolution tactile perception method and system. Background Art

[0002] In modern production, robots play an increasingly important role and need to perform precise operations in an environment of close cooperation with humans. Tactile information, as a key information carrier, plays a crucial role in enabling robots to accurately and safely complete tasks. An efficient tactile sensing system not only allows robots to "perceive" objects, but also helps them understand the properties of objects, thereby enabling more delicate operations. In the field of robotics, in order to simulate this highly complex high-resolution and multi-modal human tactile experience, it is extremely urgent to enable robots to obtain high-resolution tactile information and express it efficiently, and ultimately achieve the recognition and property determination of objects.

[0003] To enhance the tactile capabilities of robots, various tactile sensors have been continuously developed and applied to robotic operations. These sensors are based on different physical principles and transduction designs, including resistive-based, capacitive-based, optical-based, magnetic-based, and piezoelectric-effect-based tactile sensors, and have achieved preliminary applications in robotic operations. However, traditional tactile sensors still do not have a spatial resolution at the micro-nano scale. Most traditional tactile sensors can only provide pressure estimates at a single point or within a specified measurement area. Even when multiple sensors are arranged together, the spatial resolution they provide is relatively low. Therefore, it is difficult for traditional tactile sensors to measure the detailed distribution of contacts. Currently, there have been quite a few reports pointing out the effectiveness of array tactile sensors and the importance of the spatial resolution of robotic tactile sensors. For example, piezoelectric-based array tactile sensors, capacitive-based array tactile sensors, and fiber-optic-based array tactile sensors have also been widely studied, showing the potential to improve resolution. However, the number of tactile sensing units in these sensors is usually only a few dozen at most. Further, tactile sensing arrays based on microelectromechanical, microelectrode, and other technologies can achieve smaller pitches and more tactile sensing units. However, when the number of tactile sensing units increases, the large amount of wiring for reading data from the sensing units usually becomes a tricky challenge. This makes it difficult for array sensors to adapt to complex multi-curved surface scenarios. In addition, the number of wires and packaging limitations also hinder their connection to the post-processing system. At the same time, signal crosstalk and lower response frequencies caused by closely distributed and large numbers of sensing units and wires cannot be ignored, which may in turn reduce the spatial resolution of the tactile sensors. In addition, although vision-based tactile sensors (visuotactile sensors) achieve a tactile perception resolution at the order of hundreds of micrometers through optical imaging of the contact surface, limited by their internal imaging systems (imaging distance, field of view size, etc.) and specific light source designs, it is difficult to reduce their volume and spatial thickness, making it difficult to meet the complex, variable, small-volume, and flexible tactile perception requirements in the context of embodied intelligence scenarios.

[0004] In this case, when facing the high-density tactile perception requirements (such as compliant grasping or micro-texture discrimination) in the context of embodied intelligence scenarios, the complexity of the system and the difficulty of miniaturized and flexible integration will increase exponentially as the required performance increases. At the same time, higher requirements are placed on the timing read-write capabilities of the system, that is, more channels and higher power consumption are required for reading. In addition, different from vision and audition, tactile sensors have interactivity and need to continuously interact with the environment, so they are more likely to be damaged due to wire or sensing element breakage. Therefore, from a hardware perspective, in the face of the high-resolution and flexible tactile sensing required for high-density tactile perception scenarios, there is currently a lack of an effective tactile high-density coding mechanism that can reduce tactile units and thus improve tactile coding density.

[0005] In summary, current high-resolution tactile perception systems generally rely on large-scale tactile perception unit arrays or vision-based tactile perception designs. With the increase in the density of sensing units, the system needs to configure more acquisition channels and data transmission lines, resulting in problems such as complex system wiring, increased power consumption, difficult packaging, difficulty in reducing volume, difficulty in flexibility, and increased vulnerability. Especially in application scenarios such as flexible or irregular curved surfaces, these problems are more prominent, severely restricting the actual deployment and application promotion of high-resolution tactile systems. In this context, how to achieve high-resolution tactile perception effects while reducing the number of sensing units has become a key technical challenge in the current field of tactile perception. This not only relates to the resolution ability of the perception system but also directly affects the integration, energy efficiency ratio, and structural reliability of the system. Therefore, developing a super-resolution multi-modal tactile compressive sensing scheme to enhance the information density of tactile expression and achieve finer resolution and higher-modal tactile stimulus perception with fewer mechanoreceptors is of great significance. Summary of the Invention

[0006] Based on this, the purpose of the present invention is to achieve the ability to obtain tactile information with lower power consumption, fewer hardware components, lower cost, higher resolution, and lower system complexity, and to provide a super-resolution tactile perception method and system.

[0007] A super-resolution tactile perception method, characterized by comprising the following steps

[0008] S30. Obtain optoelectronic array data;

[0009] S40. Process the optoelectronic array data using a tactile sensor data generation model to obtain corresponding high-resolution tactile data, including the following sub-steps:

[0010] S41. Perform preprocessing on the time step t, optoelectronic array data y, and noisy data x t to obtain time step length encoding, optoelectronic array features, and noisy data features;

[0011] S42. Stack the time step length encoding, optoelectronic array features, and noisy data features to obtain stacked features;

[0012] S43. Perform multiple downsamplings on the stacked features to obtain encoded features with multiple different tensor sizes respectively;

[0013] S44. Perform global information modeling on the encoded features of the final output tensor in step S43 using multiple residual blocks to obtain pre-decoded features;

[0014] S45. Perform multiple upsamplings on the pre-decoded features to restore the tensor size of the pre-decoded features to the original data size to obtain decoded features;

[0015] S46. Process the decoded features using a convolutional layer to obtain denoised data x for the corresponding time step t-1 ;

[0016] S47. Repeat the above steps S41 - S46 until high - resolution tactile data x0 is generated.

[0017] The super - resolution tactile perception method of the present invention designs a tactile sensing data generation model based on the idea of a diffusion model, which can generate high - resolution tactile data that matches the input low - resolution optoelectronic array data, thereby breaking through the dependence of traditional tactile perception systems on high - density tactile sensors and realizing the possibility of obtaining high - precision tactile data with low - density sensors.

[0018] Furthermore, it further includes the following steps:

[0019] S10. Obtain training optoelectronic array data and corresponding training high - resolution tactile data;

[0020] S20. Train the tactile sensing data generation model using the training optoelectronic array data and the training high - resolution tactile data.

[0021] Furthermore, the step S20 includes the following sub - steps:

[0022] S21. Add noise to the training high - resolution tactile data to obtain noisy data x t , and this process can be expressed as: the data at time step t is obtained by combining the output data of the previous time step with standard Gaussian noise adjusted by a parameter controlling the amount of noise;

[0023] S22. Use the training optoelectronic array data as conditional input and the corresponding noisy data x t to perform noise prediction in the tactile sensing data generation model, and continuously optimize the model so that it can accurately recover the corresponding original high - resolution tactile data.

[0024] Meanwhile, the present invention also provides a super - resolution tactile perception system, including a tactile sensor, a controller, and a processing device. The tactile sensor is used to collect tactile data; the controller is electrically connected to the tactile sensor and is used to control the tactile sensor; the processing device is electrically connected to the controller and is used to process the collected tactile data, including

[0025] a conditional input unit, which is used to pre - process the time step t, the optoelectronic array data y, and the noise data x with the same size as the high - resolution tactile data t to obtain time step encoding, optoelectronic array features, and noisy data features;

[0026] Stacking unit, which is used to stack the time step encoding, optoelectronic array features and noisy data features to obtain stacked features;

[0027] Encoder unit, which is used to perform multiple downsamplings on the stacked features to obtain encoded features with multiple different tensor sizes respectively;

[0028] Bottleneck layer unit, which is used to perform global information modeling on the encoded features of the output tensor of the encoder unit by multiple residual blocks to obtain pre-decoded features;

[0029] Decoder unit, which is used to perform multiple upsamplings on the pre-decoded features to restore the tensor size of the pre-decoded features to the original data size to obtain decoded features;

[0030] Generation unit, which is used to process the decoded features by using a convolutional layer to obtain the denoised data x at the corresponding time step t-1 ;

[0031] Iterative unit, which is used to repeatedly call the above units until high-resolution tactile data x0 is generated.

[0032] Further, the tactile sensor includes an optoelectronic array layer, an optical transmission layer, a light source and a contact transducer layer. The optoelectronic array layer includes a plurality of light sensing devices arranged in an array; the optical transmission layer is arranged above the optoelectronic array layer, and the light source is used to input light into the optical transmission layer; the contact transducer layer is arranged on the force contact side of the optical transmission layer.

[0033] Further, the controller includes a conversion circuit and a collection circuit. The conversion circuit is electrically connected to the tactile sensor, and the collection circuit is electrically connected to the conversion circuit.

[0034] Further, it further includes a paired data acquisition device, and the processing device further includes a training unit;

[0035] The paired data acquisition device is used to acquire training optoelectronic array data and corresponding training high-resolution tactile data;

[0036] The training unit is used to train the tactile sensing data generation model by using the training optoelectronic array data and the training high-resolution tactile data.

[0037] Further, the paired data acquisition device includes a bracket, a detachable tactile sensor and a perception imaging system. The detachable tactile sensor is installed on the top of the bracket, and the perception imaging system is installed on the bottom of the bracket.

[0038] Further, the detachable tactile sensor includes a light transmission layer, a contact transducer layer, a photoelectric array layer, and a light source. The contact transducer layer is disposed on one side of the light transmission layer. The photoelectric array layer is detachably mounted on the other side of the light transmission layer. The light source is configured to input light into the light transmission layer.

[0039] Further, the training unit includes

[0040] a noise addition module, configured to perform noise addition processing on the training high-resolution tactile data to obtain noisy data x t , which is realized by adding standard Gaussian noise regulated by a parameter controlling the noise amount to the original data;

[0041] an optimization module, configured to use the training photoelectric array data as a condition and the corresponding noisy data x t to input into the tactile sensing data generation model for noise prediction, and continuously optimize the model so that it can accurately recover the corresponding original high-resolution tactile data.

[0042] For better understanding and implementation, the present invention will be described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 FIG. is a flowchart of a super-resolution tactile perception method;

[0044] Figure 2 FIG. is a schematic diagram of a super-resolution tactile perception method;

[0045] Figure 3 FIG. is a schematic diagram of a tactile sensing data generation model;

[0046] Figure 4 FIG. is a schematic diagram of a super-resolution tactile perception system;

[0047] Figure 5 FIG. is a schematic diagram of a tactile sensor;

[0048] Figure 6 FIG. is a schematic diagram of a conversion circuit;

[0049] Figure 7 FIG. is a schematic diagram of a paired data acquisition device. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] Please refer to Figures 1-3 , a super-resolution tactile perception method, which utilizes a tactile sensing data generation model designed based on the idea of a diffusion model and a deep neural network, includes the following steps:

[0051] S30. Obtain photoelectric array data.

[0052] S40. Process the optoelectronic array data using the tactile sensor data generation model to obtain corresponding high-resolution tactile data (in this example, visual-tactile images, hereinafter referred to as high-resolution tactile images).

[0053] The tactile sensor data generation model is designed based on the idea of diffusion models and deep neural networks. By denoising the low-resolution optoelectronic array data, it can generate corresponding high-resolution tactile images. This denoising process can be expressed as:

[0054]

[0055] where x t represents the image to be denoised; x t-1 represents the denoised image generated after one iteration; s θ (x t , t, y) represents the denoising direction predicted by the U-Net neural network; γ represents the step coefficient, which is used to control the amplitude of denoising; y represents the 8×8 optoelectronic array data, which is used as a conditional guidance for denoising; z represents the noise used for denoising sampling.

[0056] The above denoising process is an iterative process, and the image needs to be denoised multiple times to finally generate the corresponding high-resolution tactile image. Specifically, this iterative process includes the following sub-steps:

[0057] S41. Preprocess the time step t, the optoelectronic array data y, and the noisy image x t to obtain the time step encoding, the optoelectronic array feature, and the noisy image feature.

[0058] Specifically, it includes the following sub-steps:

[0059] S41a. Perform time encoding on the time step t using an encoding layer including but not limited to a multi-layer perceptron to obtain the time step encoding;

[0060] S41b. Transform the optoelectronic array data y collected by the tactile sensor into a high-dimensional feature vector using an encoding layer including but not limited to a multi-layer perceptron to obtain the optoelectronic array feature;

[0061] S41c. Perform preliminary feature extraction on the noisy image x t using a 3×3 convolutional layer to obtain the noisy image feature.

[0062] S42. Stack the time step encoding, the optoelectronic array feature, and the noisy image feature to obtain the stacked feature.

[0063] S43. Downsample the stacked features multiple times to successively obtain encoded features with tensor sizes of 128×128, 64×64, 32×32, 16×16, and 8×8 respectively.

[0064] Specifically, the encoder includes five residual blocks, each residual block contains two 3×3 convolutional layers, and uses a convolution with a stride of 2×2 to downsample the stacked features step by step, successively obtaining encoded features with tensor sizes of 128×128, 64×64, 32×32, 16×16, and 8×8 respectively.

[0065] S44. Use multiple residual blocks to perform global information modeling on the encoded features of the 8×8 tensor to obtain pre-decoded features.

[0066] Specifically, use multiple residual blocks to form a bottleneck layer, and the bottleneck layer processes the encoded features of the 8×8 tensor, keeping the tensor size unchanged during this process.

[0067] S45. Upsample the pre-decoded features multiple times to restore the tensor size of the pre-decoded features to 256×256 to obtain decoded features.

[0068] Specifically, the decoder includes a residual block structure symmetric to the encoder, uses transposed convolution to upsample the pre-decoded features, and there are skip connections set between the corresponding layers in the decoder and the encoder. The encoded features with the same tensor size are spliced into the corresponding layers of the decoder, thereby retaining more detailed information.

[0069] S46. Process the decoded features using a 3×3 convolutional layer to obtain the denoised image x at the corresponding time step t-1 。

[0070] S47. Repeat the above steps S41 - S46 until the high-resolution tactile image x0 is generated.

[0071] Furthermore, to improve the accuracy of the model for generating high-resolution tactile images, the super-resolution tactile perception method further includes the following steps:

[0072] S10. Obtain training optoelectronic array data and the corresponding training high-resolution tactile images.

[0073] Specifically, use an optoelectronic array tactile sensor to collect tactile information to obtain 8×8 optoelectronic array data; and use an optical tactile sensor to collect the corresponding training high-resolution tactile images for the above training optoelectronic array data.

[0074] S20. Use the training optoelectronic array data and the training high-resolution tactile images to train the tactile sensing data generation model.

[0075] Specifically, it includes the following sub-steps:

[0076] S21. Add noise to the high-resolution tactile image for training to obtain a noisy image x t , and this process can be expressed as:

[0077]

[0078] where x t represents the noisy image at time step t; α t represents a hyperparameter (set as a decreasing sequence) that controls the amount of noise, such that the image is gradually destroyed as the time step t increases; ∈ represents standard Gaussian noise. When the time step t is large enough, the image will degenerate into pure Gaussian noise x T .

[0079] S22. Use the training optoelectronic array data as the condition and the corresponding noisy image x t to input into the tactile sensing data generation model for noise prediction, and continuously optimize the model so that it can accurately recover the corresponding original high-resolution tactile image.

[0080] By collecting high-resolution tactile images corresponding to the optoelectronic array data and using this paired data to train the model, the prediction accuracy of the model for noise can be effectively improved, and thus the accuracy of the high-resolution tactile images generated by the model can be improved.

[0081] The super-resolution tactile perception method described in the present invention uses a neural network to process tactile data, can generate high-resolution tactile images using low-resolution optoelectronic array tactile data, realizes the possibility of obtaining high-resolution tactile images using low-resolution tactile sensors, effectively reduces the dependence on high-precision physical sensors, and greatly reduces the hardware cost and system complexity of the device.

[0082] Please refer to Figure 4 , based on the above super-resolution tactile perception method, the present invention also provides a super-resolution tactile perception system, including a tactile sensor 10, a controller 20, and a processing device 30. The tactile sensor 10 is used to collect tactile data, the controller 20 is used to control the operation of the tactile sensor 10, and the processing device 30 is used to process the collected tactile data.

[0083] Please refer to Figure 5, the tactile sensor 10 includes a photoelectric array layer 12, a light transmission layer 13, a light source 14 and a contact transducer layer 15. The photoelectric array layer 12 includes a total of 64 photoelectric detection units including but not limited to photodiodes, photoresistors, etc. arranged in an 8×8 array. The light transmission layer 13 is arranged above the photoelectric array layer 12. The light transmission layer 13 is a flexible elastomer with transparent properties, which can be made of soft elastic materials such as EcoFlex 30, polydimethylsiloxane (Polydimethylsiloxane, referred to as PDMS), Clear Flex 30, etc. In this embodiment, PDMS is mixed with a corresponding curing agent in a ratio of 10:1 and then fully stirred to obtain a PDMS mixed solution, and then dripped on the surface of the photoelectric array layer 12, thereby preparing a layer of light transmission layer 13 with a thickness of about 2 mm. Preferably, during the preparation process, the light transmission layer 13 is placed in a vacuum chamber to remove bubbles therein. The light source 14 is installed on both sides of the light transmission layer 13 so that the generated light can be evenly transmitted in the light transmission layer 13. In this embodiment, the light source 14 includes two groups of LEDs, which are respectively installed on opposite sides of the light transmission layer 13. The contact transducer layer 15 is a thin film covering the surface of the light transmission layer 13, having good light reflection properties, and used to convert mechanical stimulation into light signals. It can be a metal film, an optical metasurface, a film with a micro-nano structure, etc. In this embodiment, the contact transducer layer 15 is a metal aluminum film with a thickness of about 120nm. When there is no external contact, the light emitted by the light source only propagates in the light transmission layer 13, and the photoelectric array layer 12 does not receive the light signal; when an external object contacts the contact transducer layer 15, the contact transducer layer 15 is deformed and sinks into the light transmission layer 13, so that the light path of the light emitted by the light source 14 is changed and incident on the photoelectric array layer 12. The photoelectric array layer 12 receives the light signal, converts it into an electrical signal, and transmits it to the controller 20.

[0084] The controller 20 includes a conversion circuit 21 and a collection circuit 22. The conversion circuit 21 is connected to the tactile sensor 10 and is used to receive the electrical signal output by the photoelectric array layer 12 and convert it into an analog signal; the collection circuit 22 is connected to the conversion circuit 21 and is used to receive the analog signal and convert it into a digital signal for storage, so as to facilitate subsequent further processing. Figure 6, in this embodiment, the conversion circuit 21 includes two four-channel operational amplifiers and an eight-channel analog switch multiplexer. Electrodes of each row of the optoelectronic array layer 12 are respectively connected to the inverting input terminals of the four-channel operational amplifiers, and the negative feedback terminals of the four-channel operational amplifiers are grounded, thereby forming a "virtual ground" to achieve a zero potential under ideal conditions and avoid signal interference between other rows during operation. The multiplexer is connected to electrodes of each column of the optoelectronic array layer 12 and is used to apply an excitation voltage to the electrodes of each column one by one, so that the selected column electrodes have a driving voltage while the other column electrodes have an equal potential (zero potential). The acquisition circuit 22 is a data acquisition card, and a data acquisition program is developed in cooperation with programming and development environments including but not limited to NI LabVIEW, thereby achieving real-time recording of the optoelectronic data of the optoelectronic array.

[0085] During operation, the data acquisition card outputs a binary code through a three-digit digital port to select the corresponding physical channel, ensuring correct conduction of the target channel, thereby isolating or gating different sensor signals and avoiding signal interference. For each row of resistors of the selected column and the operational amplifier of the corresponding row, they can be equivalent to an inverting amplifier. For this, the unit resistor R1 can be (R f is the reference resistor connected to the operational amplifier):

[0086]

[0087] At this time, only the voltage output of the current row needs to be recorded to deduce the change in its resistance. Using the multi-channel synchronous acquisition function of the data acquisition card, eight-channel analog voltage signals are read simultaneously. The eight-channel data collected are organized into a two-dimensional array (matrix) in chronological order, where the rows represent the time series and the columns represent different channels. A loop is used to continuously read the data and fill it into the current row of the two-dimensional array. Each row corresponds to the eight-channel data of one sampling period. At the same time, the current state of the data matrix is displayed in real time through an oscilloscope control or a table control, which is convenient for debugging and monitoring. Finally, the data matrix is saved as a suitable format file, such as a CSV, binary file, or MAT file, etc., for subsequent data processing.

[0088] Please refer to Figure 7 , further, in order to facilitate obtaining low-resolution optoelectronic array data and the matching high-resolution tactile image, this embodiment also provides a paired data acquisition device 40. The paired data acquisition device 40 includes a bracket 41, a detachable tactile sensor 42, and an imaging system 43. The detachable tactile sensor 42 is installed on the top of the bracket 41 and is used to obtain low-resolution optoelectronic array data; the imaging system 43 is installed on the bottom of the bracket 41 and is used to obtain high-resolution tactile images.

[0089] Specifically, the detachable tactile sensor 42 is generally the same in structure as the above-mentioned tactile sensor 10, except that the optoelectronic array layer of the detachable tactile sensor 42 is a detachable structure, and a support layer is provided between the optoelectronic array layer and the light transmission layer for supporting the light transmission layer. The support layer is made of a material with high light transmittance, which is an acrylic plate in this embodiment. The imaging system 43 is aligned with the detachable tactile sensor 42 for recording the change of the light field on the transducer contact layer. When the optoelectronic array layer is removed, the entire paired data acquisition device is equivalent to a traditional vision-based tactile sensing system (visual tactile sensor).

[0090] When it is necessary to obtain paired low-resolution optoelectronic array data and high-resolution tactile images, contact pressure is applied to the contact transducer layer. For example, a spherical probe is used to press and fix the contact transducer layer, and the optoelectronic array layer is used to record the light field information at this time to obtain low-resolution optoelectronic array data. Subsequently, while keeping the contact unchanged, the optoelectronic array layer is removed, and the imaging system 43 is used to record the light field information to obtain high-resolution tactile images. Conversely, it is also possible to first use the imaging system 43 to obtain high-resolution tactile images, and then install the optoelectronic array layer to obtain low-resolution optoelectronic array data. Since the contact applied to the contact transducer layer remains unchanged throughout the process, the obtained low-resolution optoelectronic array data and high-resolution tactile images are highly matched, avoiding the problem that contact cannot be ensured to be exactly the same when contact is applied in two sets of devices in the general method.

[0091] The processing device 30 is a tactile sensing data generation model designed based on a deep neural network, which can take 8×8 optoelectronic array data as conditional input and output the corresponding high-resolution tactile image. The processing device 30 includes a conditional input unit, a stacking unit, an encoder unit, a bottleneck layer unit, a decoder unit, a generation unit, and an iterative unit.

[0092] The conditional input unit is used for preprocessing the time step t, the optoelectronic array data y, and the noisy image x t including:

[0093] A time encoding module for time encoding the time step t using an encoding layer including but not limited to a multi-layer perceptron to obtain a time step length encoding;

[0094] An optoelectronic feature module for transforming the optoelectronic array data y collected by the tactile sensor into a high-dimensional feature vector using an encoding layer including but not limited to a multi-layer perceptron to obtain optoelectronic array features;

[0095] A noise feature module for t performing preliminary feature extraction on the noisy image x using a 3×3 convolutional layer to obtain noisy image features.

[0096] The stacking unit is used to stack the time step encoding, the photoelectric array feature and the noisy image feature to obtain the stacking feature.

[0097] The encoder unit is used to downsample the stacked features multiple times to obtain encoding features with tensor sizes of 128×128, 64×64, 32×32, 16×16 and 8×8. The encoder unit includes five residual blocks, each of which contains two 3×3 convolutional layers, and uses 2×2 step convolution to downsample the stacked features step by step to obtain encoding features with tensor sizes of 128×128, 64×64, 32×32, 16×16 and 8×8.

[0098] The bottleneck layer unit is used to perform global information modeling on the coding features of the 8×8 tensor by multiple residual blocks to obtain pre-decoding features. The bottleneck layer includes multiple residual blocks, and the bottleneck layer processes the coding features of the 8×8 tensor, and keeps the size of the tensor unchanged in the process.

[0099] The decoder unit is used to perform multiple upsampling on the pre-decoding features so that the tensor size of the pre-decoding features is restored to 256×256 to obtain the decoding features.

[0100] The decoder unit includes a residual block structure symmetrical to the encoder, and uses transposed convolution to upsample the pre-decoded features. At the same time, the decoder and the corresponding layers in the encoder are provided with jump connections, and the encoded features of the same tensor size are spliced to the corresponding layers of the decoder, thereby retaining more detailed information.

[0101] The generation unit is used to process the decoded features using a 3×3 convolutional layer to obtain a denoised image x at the corresponding time step t-1 .

[0102] The iteration unit is used to repeatedly call the above units until a high-resolution tactile image x0 is generated.

[0103] Furthermore, it also includes a training unit, which is used to train the tactile sensing data generation model using the training data, thereby improving the accuracy of the model for noise prediction, and further improving the accuracy of the generated high-resolution tactile image. Specifically, the training unit includes:

[0104] The denoising module is used to perform denoising on the high-resolution tactile image for training to obtain the denoised image x t , the process can be expressed as:

[0105]

[0106] Among them, x t represents the noisy image at time step t; α tdenotes a hyperparameter for controlling the amount of noise (set as a decreasing sequence) such that the image is gradually corrupted as the time step t increases; ∈ represents standard Gaussian noise; when the time step t is large enough, the image will degenerate into pure Gaussian noise x T .

[0107] An optimization module for using the training optoelectronic array data as a condition and the corresponding noisy image x t Input the tactile sensing data into the model to generate noise prediction, and continuously optimize the model so that it can accurately recover the corresponding original high-resolution tactile image. Through the design optimization of the tactile system hardware and the deep learning-driven tactile perception technology, the present invention breaks through the design of the traditional tactile perception system that relies on a dense and complex sensor layout: by using the means of optical transduction and sparse optoelectronic tactile units to replace the traditional multi-point high-density layout, the wiring complexity and power consumption are significantly reduced. At the same time, through the neural network-driven tactile image generation technology, a sub-millimeter high-resolution tactile image is reconstructed from sparse data, and the contact position, posture, contact force, torque, object softness and hardness, etc. can be identified according to the output high-resolution tactile data, reducing the dependence on dense physical sensors, and achieving the comprehensive advantages of small volume, ultra-thin, low cost, high precision, high reliability and low power consumption. Furthermore, the overall tactile sensor of the present invention can be made of flexible materials, for example, using flexible PCB to make the optoelectronic array layer and using flexible materials to prepare the housing, etc., thereby realizing the overall flexibility of the tactile sensor and enabling it to be applied to scenarios such as flexible electronic skin. Generally speaking, the super-resolution tactile perception method provided by the present invention can effectively enhance the resolution ability of the perception system, and enhance the integration, energy efficiency ratio and structural reliability of the system. Finally, in tactile interaction scenarios such as tactile rendering, robot dexterous operation, and interactive interfaces, it empowers tactile perception capabilities with lower power consumption, lower cost, high resolution, and small volume.

[0108] The above embodiments only represent several implementation manners of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and the present invention also intends to include these changes and modifications.

Claims

1. A super-resolution tactile perception method, characterized in that: including the following steps S30. Obtain optoelectronic array data; S40. Process the optoelectronic array data by using the model generated from tactile sensing data to obtain corresponding high-resolution tactile data, including the following sub-steps: S41. Preprocess the time step \(t\), the optoelectronic array data \(y\), and the noisy data \(x\) t to obtain the time step encoding, the optoelectronic array features, and the noisy data features; S42. Stack the time-step encoding, optoelectronic array features, and noisy data features to obtain stacked features; S43. Perform multiple downsamplings on the stacked features to respectively obtain encoded features with multiple different tensor sizes; S44. Perform global information modeling on the encoded features of the final output tensor in step S43 by using multiple residual blocks to obtain pre-decoded features; S45. Perform multiple upsamplings on the pre-decoded features to restore the tensor size of the pre-decoded features to the original data size to obtain decoded features; S46. Process the decoded features using a convolutional layer to obtain the denoised data \(x\) at the corresponding time step t-1 ; S47. Repeat the above steps S41 - S46 until high-resolution tactile data x0 is generated.

2. The super-resolution tactile perception method according to claim 1, wherein: It further includes the following steps: S10. Obtain training optoelectronic array data and corresponding training high-resolution tactile data; S20. Train the tactile sensing data generation model by using the training optoelectronic array data and training high-resolution tactile data.

3. The super-resolution tactile perception method according to claim 2, wherein: The step S20 includes the following sub-steps: S21. Add noise to the high-resolution tactile data for training to obtain the noisy data x t , and this process can be expressed as: the data at time step t is obtained by combining the output data of the previous time step with standard Gaussian noise adjusted by a parameter that controls the amount of noise; S22. Use the training optoelectronic array data as a condition and the corresponding noisy data x t Input the tactile sensing data into the noise prediction model and continuously optimize the model so that it can accurately recover the corresponding original high-resolution tactile data.

4. A super-resolution tactile perception system, characterized in that: including a tactile sensor, a controller, and a processing device. The tactile sensor is used to collect tactile data; the controller is electrically connected to the tactile sensor and is used to control the tactile sensor; the processing device is electrically connected to the controller and is used to process the collected tactile data, including A conditional input unit for preprocessing the time step t, the optoelectronic array data y, and the noise data x of the same size as the high-resolution tactile data t to obtain a time step encoding, optoelectronic array features, and noisy data features; a stacking unit for stacking the time-step encoding, optoelectronic array features, and noisy data features to obtain stacked features; an encoder unit for performing multiple downsamplings on the stacked features to respectively obtain encoded features with multiple different tensor sizes; a bottleneck layer unit for performing global information modeling on the encoded features of the output tensor of the encoder unit by using multiple residual blocks to obtain pre-decoded features; a decoder unit for performing multiple upsamplings on the pre-decoded features to restore the tensor size of the pre-decoded features to the original data size to obtain decoded features; A generation unit, configured to process the decoded features by using a convolutional layer to obtain denoised data x at a corresponding time step t-1 ; an iteration unit for repeatedly calling the above units until high-resolution tactile data x0 is generated.

5. The super-resolution tactile perception system according to claim 4, characterized in that: The tactile sensor includes an optoelectronic array layer, a light transmission layer, a light source, and a contact transducer layer. The optoelectronic array layer includes a plurality of light sensing devices arranged in an array; the light transmission layer is arranged above the optoelectronic array layer, and the light source is used to input light into the light transmission layer; the contact transducer layer is arranged on the force contact side of the light transmission layer.

6. The super-resolution tactile perception system according to claim 5, wherein: The controller includes a conversion circuit and a collection circuit. The conversion circuit is electrically connected to the tactile sensor, and the collection circuit is electrically connected to the conversion circuit.

7. The super-resolution tactile perception system according to any one of claims 4-6, characterized in that: It further includes a paired data collection device, and the processing device further includes a training unit; The paired data collection device is used to obtain training optoelectronic array data and corresponding training high-resolution tactile data; The training unit is used to train the tactile sensing data generation model by using the training optoelectronic array data and training high-resolution tactile data.

8. The super-resolution tactile perception system according to claim 7, wherein: The paired data acquisition device includes a bracket, a detachable tactile sensor, and a perception imaging system. The detachable tactile sensor is installed on the top of the bracket, and the perception imaging system is installed on the bottom of the bracket.

9. The super-resolution tactile perception system according to claim 8, wherein: The detachable tactile sensor includes an optical transmission layer, a contact transducer layer, a photoelectric array layer, and a light source. The contact transducer layer is disposed on one side of the optical transmission layer, the photoelectric array layer is detachably installed on the other side of the optical transmission layer, and the light source is used to input light into the optical transmission layer.

10. The super-resolution tactile perception system according to claim 7, wherein: The training unit includes a noise-adding module for adding noise to the high-resolution tactile data for training to obtain noisy data x t , which is achieved by adding standard Gaussian noise regulated by a parameter controlling the noise amount to the original data; Optimization module, which uses the optoelectronic array data for training as a condition and the corresponding noisy data x t Input the tactile sensing data into the model for noise prediction, and continuously optimize the model so that it can accurately recover the corresponding original high-resolution tactile data.

Images