Haptic deformation reconstruction device based on lateral light guide and space perception neural network and reconstruction method thereof

By using a visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network, a ring-shaped discrete dot array light source is formed by a miniature camera and light guide plate. Combined with spatial perception neural network to process tactile images, the problem of three-dimensional reconstruction distortion caused by non-uniformity of light field in miniaturized visual-tactile sensors is solved, and high-precision three-dimensional reconstruction is achieved.

CN122049248BActive Publication Date: 2026-07-03SUN YAT SEN UNIV

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
SUN YAT SEN UNIV
Filing Date
2026-04-15
Publication Date
2026-07-03

AI Technical Summary

Technical Problem

During the miniaturization process of existing visual-tactile sensors, the nonlinear and non-uniform discrete light field causes distortion in three-dimensional reconstruction, making it difficult to achieve high-precision three-dimensional reconstruction in a small space.

Method used

A visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network is adopted. It utilizes a miniature camera, light guide plate, optical fiber and elastic components, combined with spatial perception neural network, to form a ring discrete dot matrix light source through axial light emitted by optical fiber, thereby improving the uniformity of light, and then processes tactile images through spatial perception neural network to obtain three-dimensional information.

Benefits of technology

It improves the accuracy of 3D deformation reconstruction, solves the reconstruction distortion problem caused by the non-uniformity of the light field in miniaturized devices, and realizes high-precision 3D reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122049248B_ABST
    Figure CN122049248B_ABST
Patent Text Reader

Abstract

The application discloses a kind of based on lateral light guide and spatial perception neural network's visual haptics deformation reconstruction device and its reconstruction method, reconstruction device includes: micro camera;Sleeve is configured to be covered outside micro camera, multiple mounting holes are formed on the barrel wall of sleeve;Light guide plate has reflecting surface;Elastic component has reflective layer;Multiple optical fibers are located in corresponding mounting hole;The light emitted by optical fiber is reflected to elastic component by reflecting surface, and irradiates reflective layer;When elastic component contacts target, reflective layer is deformed, micro camera is configured to take pictures to reflective layer and obtain haptic image, so as to determine the haptic information of target according to haptic image.Multiple optical fibers emit axial light under the action of light guide plate to form multiple lateral light, that is, annular discrete dot array light source is formed, the uniformity of light in elastic component is improved, so as to facilitate to improve the accuracy of three-dimensional deformation reconstruction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of deformation reconstruction technology, and more particularly to a visual-tactile deformation reconstruction device and reconstruction method based on lateral light guiding and spatial perception neural networks. Background Technology

[0002] Vision-Based Tactile Sensors (VBTS) capture images of the deformation of an elastomer when it comes into contact with an object using an internal camera. They then reconstruct the tactile information of the contact surface using computer vision algorithms or deep learning. Advances in fields such as dexterous robotics and minimally invasive medicine have placed higher demands on the miniaturization of VBTS.

[0003] In existing technologies, visual-tactile sensors typically employ large-volume lateral light source arrays or light guide plate structures to ensure uniform illumination within the elastomer. This traditional structure is generally difficult to integrate into small spaces. Due to space constraints, the light source degenerates into discrete point-like emission, forming complex asymmetric light spots within the elastomer. Traditional photometric stereo (PS) algorithms, based on the assumption of uniform or ideal Lambertian illumination, have reduced applicability in such nonlinear, non-uniform discrete light fields, leading to distortions in 3D reconstruction.

[0004] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a visual-tactile deformation reconstruction device and reconstruction method based on lateral light guiding and spatial perception neural network, which aims to solve the problem of distortion in three-dimensional reconstruction caused by nonlinear and non-uniform discrete light fields in miniaturized reconstruction devices in the prior art.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] A visual-tactile deformation reconstruction device based on lateral light guidance and spatial perception neural networks, comprising:

[0008] Miniature camera;

[0009] A sleeve is configured to fit over the miniature camera, and a plurality of mounting holes are formed on the sleeve wall;

[0010] Light guide plate, with a reflective surface;

[0011] Elastic component with reflective layer;

[0012] Multiple optical fibers are located in corresponding mounting holes;

[0013] The light emitted from the optical fiber is reflected by the reflective surface to the elastic component and then illuminates the reflective layer.

[0014] When the elastic component comes into contact with the target, the reflective layer deforms, and the miniature camera is configured to capture a tactile image of the reflective layer in order to determine the tactile information of the target based on the tactile image.

[0015] The aforementioned visual-tactile deformation reconstruction device based on lateral light guidance and spatial perception neural networks, wherein the elastic component includes:

[0016] Elastomers;

[0017] A fixed bracket is connected to the sleeve and clamps the elastomer and the light guide plate.

[0018] The fixed bracket forms a through hole, and the elastic body extends outside the through hole;

[0019] The reflective layer is located on the side of the elastomer opposite to the light guide plate.

[0020] The aforementioned visual-tactile deformation reconstruction device based on lateral light guidance and spatial perception neural network, wherein the miniature camera is a built-in camera of an endoscope, and the sleeve is threadedly connected to the end of the endoscope.

[0021] The visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network, wherein the angle between the reflective surface and the central axis of the light guide plate is 30°~60°.

[0022] A reconstruction method for a visual-tactile deformation reconstruction device based on lateral light guidance and spatial perception neural networks as described in any of the above claims, comprising the following steps:

[0023] Tactile images of the target are acquired using a miniature camera, and a distance map of the single-channel hardware structure of the visual-tactile deformation reconstruction device based on lateral light guidance and spatial perception neural network is determined.

[0024] The tactile image and the single-channel hardware structure distance map are stitched together to obtain a feature image;

[0025] The feature image is input into a trained spatial perception neural network to obtain the tactile information of the target.

[0026] The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network, wherein the tactile information includes a three-dimensional normal vector map and a depth map;

[0027] The training steps of the trained spatial awareness neural network include:

[0028] Acquire real tactile images and determine the distance map of the single-channel hardware structure of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network;

[0029] The real tactile image and the distance map of the single-channel hardware structure are stitched together to obtain a feature image;

[0030] The feature image is input into a spatial perception neural network to obtain a three-dimensional normal vector map;

[0031] The three-dimensional normal vector map is rendered in reverse to obtain a simulated lighting image;

[0032] Based on the real tactile image and the simulated lighting image, the total loss value is calculated, and the parameters of the spatial perception neural network are adjusted to obtain a trained spatial perception neural network.

[0033] The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network, wherein the total loss value is the sum of the structural similarity loss value and the spatial gradient loss value.

[0034] The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network, wherein the simulated illumination image is:

[0035] ;

[0036] in, This represents a simulated lighting image. Indicates the reflectivity of the reflective layer. Indicates the number of optical fibers. This represents the initial luminous intensity of a single optical fiber. Indicates the first Light emitted from the optical fiber reaches the pixel. distance, and Represents the coordinates of a pixel. Pixels representing the surface of an elastomer The normal vector, Indicates the first Root fiber at pixel The incident light direction vector at point .

[0037] The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guidance and spatial perception neural network, wherein the distance map of the single-channel hardware structure is as follows:

[0038] ;

[0039] in, This represents a distance diagram of a single-channel hardware structure. Represents any pixel in the input image The corresponding feature value in the hardware structure distance map This indicates the effective inner diameter of the sleeve mapped to the diameter in the pixel coordinate system, which is a coordinate system with the center of the field of view of the miniature camera as the origin.

[0040] The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guidance and spatial perception neural network, wherein the spatial perception neural network adopts an encoder-decoder structure, the encoder-decoder structure comprising:

[0041] Encoder, spatial attention module, and decoder;

[0042] The convolutional layers of the encoder and the convolutional layers of the decoder are connected in a skip connection.

[0043] Beneficial effects: The axial light emitted by multiple optical fibers forms multiple lateral light under the action of the light guide plate, which forms a ring discrete dot matrix light source, improving the uniformity of light in the elastic component, thereby helping to improve the accuracy of three-dimensional deformation reconstruction. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network in an embodiment of the present invention.

[0045] Figure 2 This is the first exploded view of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network in an embodiment of the present invention.

[0046] Figure 3 This is the second exploded view of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network in this embodiment of the invention.

[0047] Figure 4 This is a schematic diagram of light propagation in the light guide plate and sleeve in an embodiment of the present invention.

[0048] Figure 5 This is the first flowchart of the reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network in this embodiment of the invention.

[0049] Figure 6 This is the second flowchart of the reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network in this embodiment of the invention.

[0050] Explanation of reference numerals in the attached figures:

[0051] 10. Sleeve; 11. Mounting hole;

[0052] 20. Light guide plate; 21. Reflective surface;

[0053] 30. Elastic component; 31. Elastomer; 32. Fixing bracket;

[0054] 40. Endoscope. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0056] Please also refer to Figures 1-4 The present invention provides some embodiments of a visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network.

[0057] like Figure 1 , Figure 2 and Figure 4 As shown, the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network of the present invention includes:

[0058] Miniature camera;

[0059] A sleeve 10 is configured to fit over the miniature camera, and a plurality of mounting holes 11 are formed on the sleeve wall;

[0060] The light guide plate 20 has a reflective surface 21;

[0061] Elastic component 30 has a reflective layer;

[0062] Multiple optical fibers are located in the corresponding mounting holes 11;

[0063] The light emitted from the optical fiber is reflected by the reflective surface 21 to the elastic component 30 and illuminates the reflective layer. When the elastic component 30 comes into contact with the target, the reflective layer deforms. The miniature camera is configured to capture an image of the reflective layer to obtain a tactile image, so as to determine the tactile information of the target based on the tactile image.

[0064] Specifically, the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural networks is small in size and can be applied in small spaces. By contacting the target, a tactile image is obtained and processed to obtain the target's tactile information, reflecting the target's geometric features. The optical fiber is located within the mounting hole 11 in the sleeve 10's wall, corresponding to the edge of the light guide plate 20, and the reflective surface 21 is located at the edge of the light guide plate 20. Light emitted from the optical fiber illuminates the emitting surface of the light guide plate 20, and the reflective surface 21 reflects the light emitted from the optical fiber from the edge of the light guide plate 20 to the center of the light guide plate 20, illuminating the elastic component 30 and reaching the reflective layer. The elastic component 30 is elastic; when the target contacts the elastic component 30, the elastic component 30 and the reflective layer will adapt to the target's shape and undergo three-dimensional deformation. The light reflected by the reflective layer can be captured by a miniature camera. Before and after the reflective layer deforms, the light reflected by the reflective layer changes, possibly producing deformation shadows. By capturing a tactile image of the reflective layer with the miniature camera, the target's tactile information can be obtained from the tactile image.

[0065] The light guide plate 20 can be made of acrylic sheet, which can guide the light emitted from the optical fiber to the elastic component 30. The optical fiber is RGB optical fiber, for example, including red, green and blue optical fibers. The red, green and blue optical fibers are located in different mounting holes 11. Fibers of the same color can be staggered. The axial light emitted from multiple optical fibers forms multiple lateral lights under the action of the light guide plate 20, which forms a ring-shaped discrete dot matrix light source, improving the uniformity of light within the elastic component 30, thereby helping to improve the accuracy of three-dimensional deformation reconstruction.

[0066] In a preferred implementation of this invention, such as Figures 2-3 As shown, the elastic component 30 includes:

[0067] Elastomer 31;

[0068] The fixed bracket 32 ​​is connected to the sleeve 10 and clamps the elastic body 31 and the light guide plate 20.

[0069] The fixed bracket 32 ​​forms a through hole, and the elastic body 31 extends outside the through hole; the reflective layer is located on the side of the elastic body 31 away from the light guide plate 20.

[0070] Specifically, the elastomer 31 is a transparent elastomer used to contact the target and deforms according to the shape of the target after contact. The fixing bracket 32 ​​is ring-shaped and fixes the elastomer 31 and the light guide plate 20 to the sleeve 10. The reflective layer can be a coating.

[0071] In a preferred implementation of this invention, such as Figures 1-3As shown, the miniature camera is a built-in camera of the endoscope 40, and the sleeve 10 is threadedly connected to the end of the endoscope 40.

[0072] Specifically, the miniature camera can be a built-in camera of the endoscope 40, connecting the sleeve 10 to the end of the endoscope 40 to facilitate the operation of the reconstruction device.

[0073] In a preferred implementation of this invention, such as Figures 3-4 As shown, the angle between the reflective surface 21 and the central axis of the light guide plate 20 is 30°~60°.

[0074] Specifically, the angle between the reflective surface 21 and the axis of the light guide plate 20 is 30° to 60°. By changing the angle, the angle of light reflected by the reflective surface 21 can be changed, thus changing the light distribution within the elastic component 30. For example, if the angle is 45°, the light reflected by the reflective surface 21 is nearly perpendicular to the central axis of the light guide plate 20.

[0075] Based on the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network described in any of the above embodiments, the present invention also provides a preferred embodiment of the reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network.

[0076] like Figure 5 As shown, the reconstruction method of this invention includes the following steps:

[0077] Step S100: Acquire tactile images of the target based on a miniature camera, and determine the single-channel hardware structure distance map (Radial Distance Map, R-map) of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network.

[0078] Step S200: The tactile image and the single-channel hardware structure distance map are stitched together to obtain a feature image;

[0079] Step S300: Input the feature image into the trained spatial perception neural network to obtain the tactile information of the target.

[0080] Specifically, when the elastic component contacts the target, a tactile image of the target is acquired using a miniature camera. Based on the reconstruction device, a single-channel hardware structure distance map is determined. This single-channel hardware structure distance map is a two-dimensional matrix representation of the physical distance from image pixels to the light source, specifically calculated based on the physical arrangement and geometric dimensions of the light source. Specifically, it can be obtained based on hardware physical dimensions such as the inner diameter of the sleeve, the angle of the fiber arrangement, the height of the reflective surface, and the polar coordinates of the radial distance of the fiber. The central region of the single-channel hardware structure distance map has values ​​close to 1 (a value close to 1 indicates a greater distance from the light source and more severe light attenuation), while the edge regions have values ​​close to 0 (a value close to 0 indicates a closer distance from the light source and weaker light attenuation). The tactile image and the single-channel hardware structure distance map are stitched together to obtain a feature image. This feature image is then input into a trained spatial perception neural network to obtain the tactile information of the target. The distance map of the single-channel hardware structure is stitched with the tactile image to form a multi-channel input. The spatial perception neural network is used to perceive the physical position of the pixel in a specific optical path, accurately compensate for nonlinear illumination attenuation, and solve the optical distortion problems such as discrete lattice light spots and subsurface scattering caused by miniaturization.

[0081] The tactile information includes a three-dimensional normal vector map and a depth map. The tactile information of the target can reflect the geometric features of the target in different ways, such as using a three-dimensional normal vector or a depth map.

[0082] like Figure 6 As shown, the training steps of the trained spatial perception neural network include:

[0083] Step A100: Acquire real tactile images and determine the single-channel hardware structure distance map of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network.

[0084] Step A200: Stitch the real tactile image and the single-channel hardware structure distance map together to obtain a feature image;

[0085] Step A300: Input the feature image into the spatial perception neural network to obtain a three-dimensional normal vector map;

[0086] Step A400: Reverse render the three-dimensional normal vector map to obtain a simulated lighting image;

[0087] Step A500: Based on the real tactile image and the simulated lighting image, calculate the total loss value and adjust the parameters of the spatial perception neural network to obtain the trained spatial perception neural network.

[0088] Specifically, during the training process of a spatial perception neural network, by performing reverse rendering on the output three-dimensional normal vector map, a simulated lighting image can be obtained by combining specific hardware optical path parameters. This image is then encoded into the forward propagation process of the neural network's self-supervised loss calculation, which is beneficial for accurate mathematical modeling of specific miniaturized hardware optical paths.

[0089] In the absence of real 3D deformation data, this invention employs a self-supervised learning framework and designs a specialized loss function to address the subsurface scattering whitening phenomenon unique to this hardware. The total loss is the sum of the structural similarity index Measure (SSIM) and the spatial gradient loss. The structural similarity loss focuses on the similarity of local image structures, while the spatial gradient loss focuses on matching high-frequency deformation features. These two loss functions are insensitive to changes in the absolute value of global brightness, aiming to reduce global whitening interference caused by subsurface scattering and achieve stable network convergence. Joint optimization using structural similarity loss and spatial gradient loss achieves feature-level consistency. By deeply integrating hardware prior knowledge with the neural network architecture and training loss function, this invention aims to achieve stable network convergence under complex non-uniform light fields and ultimately obtain high-precision, highly robust 3D deformation reconstruction results.

[0090] The simulated lighting image is:

[0091] ;

[0092] in, This represents a simulated lighting image. Indicates the reflectivity of the reflective layer. Indicates the number of optical fibers. This represents the initial luminous intensity of a single optical fiber. Indicates the first Light emitted from the optical fiber reaches the pixel. distance, and Represents the coordinates of a pixel. The height of the reflective surface, determined by the inner diameter of the sleeve, reflects the nonlinear spatial attenuation of near-field illumination. Pixels representing the surface of an elastomer The normal vector, , , , Let represent the coordinates of the normal vector, Indicates the first Root fiber at pixel The incident light direction vector at point [location missing]. Total internal reflection is employed using a reflecting surface; the zenith angle of the incident light is fixed by the hardware structure, while the azimuth angle is determined by the [missing information - likely a specific element or parameter]. The circumferential arrangement angle of the optical fibers on the outer wall of the sleeve is strictly determined. In the self-supervised reverse rendering module, the reflection parameters of the light guide plate's reflective surface (determining the incident light zenith angle) and the fiber position parameters (determining the incident light azimuth angle) are substituted into the rendering equation. When the angle of the light guide plate's reflective surface is slightly adjusted, the incident light zenith angle parameter in the algorithm's rendering equation and the distance calculation formula of the R-map are parsed and synchronously changed accordingly, maintaining the effectiveness of the hardware-software binding.

[0093] The distance diagram of the single-channel hardware structure is as follows:

[0094] ;

[0095] in, This represents a distance diagram of a single-channel hardware structure. Represents any pixel in the input image The corresponding feature value in the hardware structure distance map and Represents the coordinates of a pixel. This indicates the effective inner diameter of the sleeve mapped to the diameter in the pixel coordinate system, which is a coordinate system with the center of the field of view of the miniature camera as the origin.

[0096] Spatial perception neural networks can employ a Vision Transformer (ViT) architecture. The ViT architecture utilizes a global self-attention mechanism, combined with a single-channel hardware distance map as a supplement to position encoding, further enhancing the ability to perceive global illumination distortion. Spatial perception neural networks can also employ an encoder-decoder structure, which includes:

[0097] Encoder, spatial attention module, and decoder;

[0098] The convolutional layers of the encoder and the convolutional layers of the decoder are connected in a skip connection.

[0099] Specifically, the encoder-decoder structure is a U-shaped symmetrical convolutional neural network, comprising a downsampling encoder and an upsampling decoder, and fusing multi-scale features through skip connections. A spatial attention module is introduced between the encoder and decoder. This module utilizes physical distance information provided by the distance map of the single-channel hardware structure to generate a spatial weight mask, guiding the spatial awareness neural network to focus on areas with severe illumination attenuation or overlapping light spots. In this way, the spatial awareness neural network can "sense" the physical location of the current pixel, thereby compensating for nonlinear illumination attenuation.

[0100] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A reconstruction method for a visual-tactile deformation reconstruction device based on lateral light guidance and spatial perception neural networks, characterized in that, The visual-tactile deformation reconstruction device based on lateral light guidance and spatial perception neural network includes: Miniature camera; A sleeve is configured to fit over the miniature camera, and a plurality of mounting holes are formed on the sleeve wall; Light guide plate, with a reflective surface; Elastic component with reflective layer; Multiple optical fibers are located in corresponding mounting holes; The light emitted from the optical fiber is reflected by the reflective surface to the elastic component and then illuminates the reflective layer. When the elastic component comes into contact with the target, the reflective layer deforms, and the miniature camera is configured to capture an image of the reflective layer to obtain a tactile image, so as to determine the tactile information of the target based on the tactile image; The reconstruction method includes the following steps: Tactile images of the target are acquired using a miniature camera, and a distance map of the single-channel hardware structure of the visual-tactile deformation reconstruction device based on lateral light guidance and spatial perception neural network is determined. The tactile image and the single-channel hardware structure distance map are stitched together to obtain a feature image; The feature image is input into a trained spatial perception neural network to obtain the tactile information of the target; The distance diagram of the single-channel hardware structure is as follows: ; in, This represents a distance diagram of a single-channel hardware structure. Represents any pixel in the input image The corresponding feature value in the hardware structure distance map and Represents the coordinates of a pixel. This indicates the effective inner diameter of the sleeve mapped to the diameter in the pixel coordinate system, which is a coordinate system with the center of the field of view of the miniature camera as the origin.

2. The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network according to claim 1, characterized in that, The tactile information includes a three-dimensional normal vector map and a depth map; The training steps of the trained spatial awareness neural network include: Acquire real tactile images and determine the distance map of the single-channel hardware structure of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network; The real tactile image and the distance map of the single-channel hardware structure are stitched together to obtain a feature image; The feature image is input into a spatial perception neural network to obtain a three-dimensional normal vector map; The three-dimensional normal vector map is rendered in reverse to obtain a simulated lighting image; Based on the real tactile image and the simulated lighting image, the total loss value is calculated, and the parameters of the spatial perception neural network are adjusted to obtain a trained spatial perception neural network.

3. The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network according to claim 2, characterized in that, The total loss value is the sum of the structural similarity loss value and the spatial gradient loss value.

4. The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network according to claim 2, characterized in that, The simulated lighting image is: ; in, This represents a simulated lighting image. Indicates the reflectivity of the reflective layer. Indicates the number of optical fibers. This represents the initial luminous intensity of a single optical fiber. Indicates the first Light emitted from the optical fiber reaches the pixel. distance, Pixels representing the surface of an elastomer The normal vector, Indicates the first Root fiber at pixel The incident light direction vector at point .

5. The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network according to any one of claims 1 to 4, characterized in that, The spatial awareness neural network employs an encoder-decoder structure, which includes: Encoder, spatial attention module, and decoder; The convolutional layers of the encoder and the convolutional layers of the decoder are connected in a skip connection.

6. The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network according to claim 1, characterized in that, The elastic component includes: Elastomers; A fixed bracket is connected to the sleeve and clamps the elastomer and the light guide plate. The fixed bracket forms a through hole, and the elastic body extends outside the through hole; The reflective layer is located on the side of the elastomer opposite to the light guide plate.

7. The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network according to claim 1, characterized in that, The miniature camera is a built-in camera of the endoscope, and the sleeve is threadedly connected to the end of the endoscope.

8. The reconstruction method of the visual-tactile deformation reconstruction device based on lateral light guiding and spatial perception neural network according to claim 1, characterized in that, The angle between the reflective surface and the central axis of the light guide plate is 30°~60°.