A see-through skin sensing system that provides both visual and tactile perception

Abstract

This invention provides a see-through skin sensing system that simultaneously provides visual and tactile perception. First, a camera module uses a transparent silicone deformation layer to acquire a real-time image sequence containing both visual images of the external environment and positioning point patterns. Then, using image processing algorithms, the visual images of the external environment are efficiently extracted from the real-time image sequence as visual signals, and the displacement of the positioning points on the sensor surface is extracted as tactile signals. Simultaneously, the deformation layer of this invention is a fully transparent silicone deformation layer without a reflective layer, and the sensor is continuously illuminated internally. In other words, relying on a robust design based on "keyline positioning points," and utilizing a continuous internal light source and a fully transparent silicone deformation layer, the system can simultaneously extract tactile and visual information from a single frame image, ensuring the temporal consistency of multimodal information. This allows the robot to collaboratively utilize visual and tactile information, ultimately achieving efficient, robust, and simultaneous visual-tactile perception.

Description

Technical Field

[0001] This invention belongs to the field of robot perception technology, and in particular relates to a see-through skin sensing system that provides both visual and tactile perception. Background Technology

[0002] See-through-skin sensors (STS sensors) have become important sensing devices in recent years, providing robots with multimodal sensing signals such as tactile, fingertip vision, and proximity sensing. These sensors are typically mounted on the tips of a robot's fingers. An LED module and a camera are installed inside the sensor. A transparent silicone deformation layer (several millimeters thick) is mounted on one side of the sensor surface, and the outer surface is successively covered with a black silicone positioning point array and a white semi-transparent reflective coating. The camera faces this silicone deformation layer to simultaneously capture visual images of the sensor's external environment, the solid-color (usually black) positioning points, and the LED light reflected from the white semi-transparent reflective coating. The LED module is controllable: when the light is off, the captured images are primarily external visual scenes, providing visual perception and a proximity-like effect when the sensor approaches an object; when the light is on, the sensor operates similarly to a see-through-touch sensor, calculating the normal deformation of the silicone by analyzing the reflected image after the silicone deformation layer deforms upon contact with the object, and calculating the tangential deformation by tracking the displacement of the positioning points, thus providing tactile signals.

[0003] The existing technology mainly upgrades some components based on the above basic principles to improve the sensing capabilities of visual or tactile modalities, including: (1) using a binocular camera or microlens array to provide three-dimensional visual imaging in visual mode; (2) using a silicone deformation layer coated with an opaque reflective coating, and realizing the movement of the deformation layer through a mechanical structure, so that the sensor has the same tactile sensing function as the visual-tactile sensor and the visual sensing function that is not interfered with by the deformation layer.

[0004] The solution described in the background art has the following technical defects:

[0005] 1. Sensing Non-Simultaneity: Schemes that rely on adjusting light switches or mechanical structures to switch between visual and tactile modes forcibly separate the two sensing modes in time. This results in the loss of sensing information from one mode at any given moment and increases the complexity of the system's control logic. This non-simultaneity limits the sensor's performance in dynamic and complex tasks.

[0006] 2. Visual signals are severely interfered with: The translucent coating and opaque positioning points used to provide tactile mode signals become the imaging foreground in visual mode, causing the external visual images acquired by the camera to be partially obscured, weakened or interfered with, reducing the quality and clarity of visual perception.

[0007] 3. Poor robustness of tactile signals: In tactile mode, even with a semi-transparent coating, the stability of the detection and tracking of positioning points faces severe challenges. The reason is that the positioning point image is no longer based on controlled uniform internal background imaging, but is superimposed with dynamic, complex and unpredictable external environmental background, which will lead to two problems: (1) Missed detection problem: When using solid color positioning points, if the external background and the positioning point are the same or similar in color (such as black positioning point and black external environment), the edge of the positioning point will lose contrast and cannot be distinguished from the background, causing the detection algorithm to fail and resulting in missed detection; (b) False detection problem: The visual features of the external environment (such as texture and text) may appear as spot patterns in the image. The positioning point detection algorithm is prone to misjudging such noise as positioning points, resulting in false detection. Summary of the Invention

[0008] To address the aforementioned issues, this invention provides a see-through skin sensing system that simultaneously provides visual and tactile perception. It can extract tactile and visual signals from a single frame image, ensuring the temporal consistency of multimodal information and enabling robots to collaboratively utilize visual and tactile information.

[0009] A see-through skin sensing system that simultaneously provides visual and tactile perception includes a computing device and a sensor mounted at the end of an actuator. The sensor includes a sensor housing with an opening at the top, a camera module and an LED lighting module mounted inside the sensor housing, and a transparent silicone deformable layer mounted on the opening of the sensor housing via a bracket. The transparent silicone deformable layer has a keyline positioning point array with known original coordinates, and the coordinates of the keyline positioning point array change as the transparent silicone deformable layer contacts the end of the actuator.

[0010] The camera module is used to acquire real-time image sequences of the external environment through a transparent silicone deformation layer; wherein, the real-time image sequence is an image sequence superimposed by a keyline positioning point array and the external environment;

[0011] The LED lighting module is used to provide a continuous light source during the operation of the sensor;

[0012] The computing device is used to calculate the displacement of each keyline positioning point based on the real-time image sequence and provide the displacement as a tactile signal to the actuator; at the same time, the computing device also re-acquires the current coordinates of each keyline positioning point based on the displacement, and then generates a key point region mask to eliminate the keyline positioning point array on the real-time image sequence based on the current coordinates, and finally provides the obtained external environment image sequence as a visual signal to the actuator.

[0013] Furthermore, the keyline positioning point array is composed of multiple layers of concentric positioning points, and the contrast between each layer of positioning points is greater than a set value, the size of each positioning point is greater than the resolution of the camera module, and the center distance between adjacent positioning points is greater than the maximum tangential deformation that may occur in the transparent silicone deformation layer.

[0014] Furthermore, the transparent silicone deformation layer is a cuboid with a length and width in the centimeter range and a thickness in the millimeter range. Its surface is coated with two layers of key line positioning point arrays. The size of the positioning points in both layers is in the millimeter range, and the size of the positioning points in the bottom layer is smaller than that in the top layer. The color contrast between the two layers of positioning points is greater than a set value.

[0015] Furthermore, the method for preparing the transparent silicone deformable layer includes the following steps:

[0016] S1: 3D printing is used to create a support frame and mold frame for the transparent silicone deformable layer; a transparent acrylic support layer is embedded inside the support frame with glue, and the mold frame is fixed above the support frame; transparent polarizing silicone A and B components are mixed in a 1:1 ratio, and after being fully mixed, they are injected into the mold frame. After the silicone solidifies naturally, the mold frame is removed to form the silicone layer of the transparent silicone deformable layer.

[0017] S2: Cut out mask A and mask B with the same shape as the silicone layer using a laser cutting machine; cut out an array of circular holes at the same position on mask A and mask B respectively, with the position of the circular holes being the same as the position of the positioning point to be sprayed;

[0018] S3: Place mask A on the surface of the silicone layer, spray black pigment with a spray gun, remove mask A, and let it dry naturally to form black positioning points on the surface of the silicone layer.

[0019] S4: Place mask B on the surface of the silicone layer with black positioning points already formed, so that the array of black positioning points is aligned with the array of circular holes of mask B, and each black positioning point is located in the center of the corresponding circular hole; after spraying white pigment with a spray gun, remove mask B and let it dry naturally, so that white positioning points are formed on the surface of the silicone layer above the black positioning points, thus obtaining the final transparent silicone deformation layer.

[0020] Furthermore, the initial state At any given moment, the original coordinates of the keyline positioning point array The covariance is obtained through sensor fabrication and camera calibration parameter calculations, or through one-time manual annotation. Initialize to A matrix consisting entirely of zeros.

[0021] Furthermore, the method by which the computing device calculates the displacement of each keyline positioning point based on the real-time image sequence is as follows:

[0022] Step 1: For the current frame image Preprocessing is performed, which includes distortion correction, grayscale conversion, and minimum-maximum value normalization. Pixels with brightness below a brightness threshold in the preprocessed image are set to 0, and the rest are set to 255, resulting in the grayscale binary image of the current frame. ;

[0023] Step 2: Convert the current frame to a grayscale binary image. Perform a blob detection operation to obtain a candidate blob set. Among them, candidate spot set This includes both the actual location points and environmental noise points caused by the external environment;

[0024] Step 3: Reset the previous frame image The obtained posterior estimates of the coordinates of each positioning point Each point in the candidate blob set is matched with a posterior estimate of its coordinates. The coordinates of the nearest blob are used as the observations of each location point in the current frame. :

[0025]

[0026] in, Represents the candidate spot set The position coordinates of any spot in the array;

[0027] Step 4: Observe each positioning point in the current frame The input is a Kalman filter, and the output of the Kalman filter is the current frame image. The corresponding posterior estimates of the coordinates of each positioning point ;

[0028] Step 5: Calculate the current frame image respectively The corresponding posterior estimates of the coordinates of each positioning point Compared with the original coordinates Displacement between .

[0029] Furthermore, in step 4, the Kalman filter acquires the current frame image. The corresponding posterior estimates of the coordinates of each positioning point The method is as follows:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] in, For the current frame image The prior predicted coordinates of each location point. The current frame image And the previous frame image The posterior estimates of the coordinates of each location point. The current frame image And the previous frame image Kalman filter gain, The current frame image And the previous frame image The covariance of the prior predicted coordinates of each location point The current frame image And the previous frame image The covariance of the posterior estimates of the coordinates of each location point. The variance of the Gaussian distribution of process noise. To observe the Gaussian distribution variance of the noise, for The identity matrix.

[0036] Furthermore, a see-through skin sensing system that provides both visual and tactile perception also includes a power supply module and a data transmission circuit board.

[0037] The power supply module is used to supply power to the camera module and the LED lighting module;

[0038] The data transmission circuit board is used to transmit real-time image sequences to computing devices.

[0039] Furthermore, the computing device employs the Telea algorithm to eliminate the array of keyline positioning points on the real-time image sequence.

[0040] Beneficial effects:

[0041] 1. This invention provides a see-through skin sensing system that simultaneously provides visual and tactile perception. First, a camera module uses a transparent silicone deformation layer to acquire a real-time image sequence containing both visual images of the external environment and positioning point patterns. Then, using image processing algorithms, the visual images of the external environment are efficiently extracted from the real-time image sequence as visual signals, and the displacement of the positioning points on the sensor surface is extracted as tactile signals. Simultaneously, the deformation layer of this invention is a fully transparent silicone deformation layer without a reflective layer, and the sensor is continuously illuminated internally. In other words, this invention relies on a robust design based on "keyline positioning points," and with the help of a continuous internal light source and a fully transparent silicone deformation layer, the system can simultaneously extract tactile (positioning point) and visual (external scene) information from a single frame image, ensuring the temporal consistency of multimodal information. This allows the robot to collaboratively utilize visual and tactile information, improving performance in delicate, contact-rich tasks, and ultimately achieving efficient, robust, and simultaneous visual-tactile perception.

[0042] 2. This invention provides a see-through skin sensing system that simultaneously provides visual and tactile perception. It uses a Kalman filter to track each key point. After performing pixel brightness thresholding on the image containing the positioning point and the environmental background, it performs spot detection to obtain a candidate spot set. Then, it performs data association based on the position of the key point at the previous time and the distance of the spot. It matches the nearest spot from the candidate spot set as the observation value and updates the Kalman filter, thus achieving robustness of positioning point tracking.

[0043] 3. This invention provides a see-through skin sensing system that simultaneously provides visual and tactile perception. The key line positioning points adopt a high-contrast concentric layer design, which ensures that high-contrast edges are always visible regardless of the background color, thus fundamentally guaranteeing the detectability of the positioning points. Attached Figure Description

[0044] Figure 1 The basic sensor structure provided by this invention;

[0045] Figure 2 This is a schematic diagram illustrating an implementation example of the design of keyline positioning points provided by the present invention;

[0046] Figure 3 A flowchart illustrating the fabrication process of the silicone deformation layer and key line positioning points provided by this invention;

[0047] Figure 4 A schematic diagram illustrating the calculation process of visual and tactile signals provided by this invention. Detailed Implementation

[0048] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0049] It should be noted that, in order to overcome the following problems in the existing technology:

[0050] (1) The problem of non-simultaneity of visual and tactile perception;

[0051] (2) The visual modal part of the image signal is obstructed and interfered with by the semi-transparent coating and positioning points, resulting in poor clarity;

[0052] (3) The tactile modality in the image signal suffers from missed detection due to the loss of contrast of the positioning points in a complex background, and false detection due to environmental background noise.

[0053] This invention provides a see-through skin sensing system that simultaneously provides visual and tactile perception, enabling efficient, robust, and simultaneous visual-tactile perception. That is, it can efficiently extract visual images of the external environment and tactile signals representing the displacement of positioning points on the sensor surface from a single image captured by the sensor camera module, while being robust to the external environmental background.

[0054] Specifically, the see-through skin sensing system of the present invention includes a computing device, a power supply module, a data transmission circuit board, and a sensor installed at the end of the actuator, wherein, as... Figure 1 As shown, the sensor includes a sensor housing with an open top, a camera module and an LED lighting module installed inside the sensor housing, and a transparent silicone deformable layer mounted on the open top of the sensor housing via a bracket; wherein, a keyline positioning point array with known original coordinates is provided on the transparent silicone deformable layer, and the coordinates of the keyline positioning point array change as the transparent silicone deformable layer contacts the end of the actuator.

[0055] The camera module is used to acquire real-time image sequences of the external environment through a transparent silicone deformation layer; wherein, the real-time image sequence is an image sequence superimposed by a keyline positioning point array and the external environment;

[0056] The LED lighting module is used to provide a continuous light source during the operation of the sensor;

[0057] The computing device is used to calculate the displacement of each keyline positioning point according to the real-time image sequence and provide the displacement as a tactile signal to the actuator; at the same time, the computing device also re-acquires the current coordinates of each keyline positioning point according to the displacement, and then generates a key point region mask to eliminate the keyline positioning point array on the real-time image sequence based on the Telea algorithm, and finally provides the obtained external environment image sequence as a visual signal to the actuator.

[0058] The power supply module is used to supply power to the camera module and the LED lighting module;

[0059] The data transmission circuit board is used to transmit real-time image sequences to computing devices.

[0060] It should be noted that after the sensor assembly is completed, the present invention can connect the sensor and an external computing device (such as a computer) via a USB cable to provide power to the camera and LED light module, and can also capture real-time images from the camera on the external computing device.

[0061] The structure and preparation method of the transparent silicone deformable layer of the present invention will be described in detail below.

[0062] Existing see-through skin sensors typically have a semi-transparent coating to reflect light from the LED lighting module for calculating the normal deformation of the silicone. This invention, however, uses a fully transparent silicone deformation layer, i.e., without a reflective coating, such as... Figure 1 As shown, this design can improve the clarity of visual signals.

[0063] Although this design prevents the sensor of this invention from using the traditional visual-tactile principle to calculate the normal deformation of silicone as a tactile sensing signal, the transparent silicone deformation layer of this invention has the following characteristics:

[0064] (1) The key line positioning points diverge around the contact area, and the degree of divergence is positively correlated with the magnitude of the contact force;

[0065] (2) Before and after the actuator end comes into contact with the transparent silicone deformation layer, its contact area changes from silicone-air interface to silicone-object interface. The light from the LED module changes from partial reflection to almost complete reflection, resulting in a significant brightening of the area. Whether contact has occurred can be deduced by the change in image brightness.

[0066] This invention complements the two features mentioned above to improve information accuracy, thereby enabling the estimation of the contact area and the magnitude of the contact force, and using them as alternative tactile signals.

[0067] Furthermore, the "key line positioning point" proposed in this invention draws on the concept of "key line" used in the field of graphic design to mark the edges of color areas. Its core purpose is to ensure that the identifiable features (edges) of the positioning point have high visibility in various backgrounds.

[0068] like Figure 1As shown, the keyline positioning point array is set on the surface of the silicone deformation layer. The keyline positioning points consist of two concentric positioning point layers, each using a high-contrast color. The size of each positioning point is larger than the resolution of the camera module, meaning the positioning points need to be observable by the camera module, but not too numerous or too large to affect the visual modality. The layout of the positioning point array can be appropriately selected, with the center-to-center distance between adjacent positioning points greater than the maximum tangential deformation that the transparent silicone deformation layer may exhibit. Simultaneously, the transparent silicone deformation layer is a cuboid with dimensions on the order of centimeters and a thickness on the order of millimeters. Two layers of keyline positioning point arrays are sprayed onto its surface, with the size of the positioning points in both layers on the order of millimeters, and the size of the positioning points in the bottom layer being smaller than that in the top layer. The color contrast between the two layers of positioning points is greater than a set value.

[0069] For example, as shown in the appendix Figure 2 The implementation example shown: The two positioning points are concentric circles with radii of... The hole spacing is ,exist Silicone deformation layer surface spraying Array.

[0070] It should be noted that, to achieve the aforementioned double-layer concentric structure, this invention proposes a fabrication process based on a laser-cut mask and sequentially sprayed positioning points. For example... Figure 3 As shown, the preparation process of the silicone deformation layer mainly includes:

[0071] S1: 3D printing is used to create a support frame and mold frame for the transparent silicone deformable layer; a transparent acrylic support layer is embedded inside the support frame with glue, and the mold frame is fixed above the support frame; transparent polarizing silicone A and B components are mixed in a 1:1 ratio, and after being fully mixed, they are injected into the mold frame. After the silicone solidifies naturally, the mold frame is removed to form the silicone layer of the transparent silicone deformable layer.

[0072] S2: Cut out mask A and mask B with the same shape as the silicone layer using a laser cutting machine; cut out an array of circular holes at the same position on mask A and mask B respectively, with the position of the circular holes being the same as the position of the positioning point to be sprayed;

[0073] S3: Place mask A on the surface of the silicone layer, spray black pigment with a spray gun, remove mask A, and let it dry naturally to form black positioning points on the surface of the silicone layer.

[0074] S4: Place mask B on the surface of the silicone layer with black positioning points already formed, so that the array of black positioning points is aligned with the array of circular holes of mask B, and each black positioning point is located in the center of the corresponding circular hole; after spraying white pigment with a spray gun, remove mask B and let it dry naturally, so that white positioning points are formed on the surface of the silicone layer above the black positioning points, thus obtaining the final transparent silicone deformation layer.

[0075] It should be noted that in the above steps, the present invention uses laser cutting to prepare the mask and sequentially spraying positioning points to prepare the key line positioning points. However, other processes that can manufacture multi-layer fine structures, such as screen printing and pad printing, are also alternative solutions.

[0076] After completing the above steps, the transparent silicone deformation layer, together with the bracket, is fixed to the surface of the sensor opening with fasteners. When the sensor camera module images, the edges of the black positioning points in the inner layer are visible against any external environmental background. This is similar to the function of key lines in graphic design, which are used to mark the boundaries of color areas. This is called "key line positioning points".

[0077] It should be noted that the present invention uses two layers of concentric circles to construct the key line positioning points, but it can also use three or more layers (e.g., "white-black-white") to increase robustness in more complex backgrounds. The present invention will not elaborate on this.

[0078] The following details the process by which computing devices acquire tactile and visual signals from real-time image sequences, such as... Figure 4 As shown, it includes the following steps:

[0079] Step 0: Initial State At any given moment, the original coordinates of the keyline positioning point array are obtained through sensor fabrication and camera calibration parameter calculation. Alternatively, the original coordinates of the keyline positioning point array can be obtained through one-time manual annotation. At the same time, the corresponding covariance Initialize to A matrix consisting entirely of zeros.

[0080] Step 1: For the current frame image Preprocessing is performed, which includes distortion correction, grayscale conversion, and minimum-maximum value normalization. Pixels with brightness below a brightness threshold in the preprocessed image are set to 0, and the rest are set to 255, resulting in the grayscale binary image of the current frame. ;

[0081] Step 2: Convert the current frame to a grayscale binary image. Perform a blob detection operation to obtain a candidate blob set. Among them, candidate spot set This includes both the actual location points and environmental noise points caused by the external environment;

[0082] Step 3: Reset the previous frame image The obtained posterior estimates of the coordinates of each positioning point Each point in the candidate blob set is matched with a posterior estimate of its coordinates. The coordinates of the nearest blob are used as the observations of each location point in the current frame. :

[0083]

[0084] in, Represents the candidate spot set The position coordinates of any spot in the array;

[0085] Step 4: Observe each positioning point in the current frame The input is a Kalman filter, and the output of the Kalman filter is the current frame image. The corresponding posterior estimates of the coordinates of each positioning point The details are as follows:

[0086]

[0087]

[0088]

[0089]

[0090]

[0091] in, For the current frame image The prior predicted coordinates of each location point. The current frame image And the previous frame image The posterior estimates of the coordinates of each location point. The current frame image And the previous frame image Kalman filter gain, The current frame image And the previous frame image The covariance of the prior predicted coordinates of each location point The current frame image And the previous frame image The covariance of the posterior estimates of the coordinates of each location point. The variance of the Gaussian distribution of process noise. To observe the Gaussian distribution variance of the noise, for The identity matrix.

[0092] It should be noted that this invention uses a Kalman filter to track each key point, but any other time-series state estimation filter, such as a "particle filter" or "unscented Kalman filter (UKF)," can be used as a substitute. The tracking process of these filters will not be described in detail in this invention.

[0093] Step 5: Calculate the current frame image respectively The corresponding posterior estimates of the coordinates of each positioning point Compared with the original coordinates Displacement between .

[0094] Therefore, the tactile signal of the sensor in this invention can be characterized by the displacement of the positioning points. This invention sequentially processes the sensor image through grayscale and thresholding, and then uses a speckle detector to detect keyline positioning points. However, the detection results contain environmental noise (false detections) from non-positioning points, so a Kalman filter is needed to filter the detection results. Therefore, this invention uses a random walk model to construct the Kalman filter; then, at fixed time steps, real-time images are acquired from the camera module, and the positioning points in the real-time images are detected as observations. This updates the filter state according to the standard linear Kalman filter model; the updated state posterior estimate is then used to update the filter state. As the location of the corresponding positioning point.

[0095] Posterior estimates of the coordinates of each location point Afterwards, a key point region mask can be created, and the Telea algorithm can be used to interpolate the key point regions of the sensor image to remove the key point regions and obtain a visual sensing signal with a good field of view of the external environment and objects.

[0096] Based on this, the working principle of the transparent skin sensing system of the present invention can be summarized as follows:

[0097] During sensor operation, the internal LED lighting module remains continuously on. A single frame image captured by the camera includes both the external visual image obtained through the transparent silicone deformation layer and clearly defined keyline positioning points. Positioning point detection and tracking algorithms track the points, enabling continuous acquisition of tactile sensing signals under any external background, thus solving the problem of "poor robustness of tactile signals." High-quality visual sensing signals are obtained through positioning point image removal methods, solving the problem of "severe interference with visual signals." Both signals can be calculated from the sensor image at the same time, thus providing the ability to simultaneously provide tactile and visual sensing signals, resolving the issue of non-simultaneity between the two modalities.

[0098] In summary, compared with the prior art, the present invention has the following advantages:

[0099] 1. Achieve simultaneous multimodal sensing

[0100] Existing technologies enable visual and tactile modes separately by switching lighting switches, resulting in temporal separation of the tactile and visual modalities and a loss of information in the other modality. This invention relies on a robust design of "keyline localization points," utilizing a continuous internal light source and a fully transparent silicone deformation layer to enable the system to simultaneously extract tactile (localization points) and visual (external scene) information from a single frame image. This ensures the temporal consistency of multimodal information, allowing the robot to collaboratively utilize visual and tactile information, thus improving performance in fine, contact-rich tasks.

[0101] 2. Significantly improves the robustness of positioning point tracking.

[0102] (a) Solving the "missed detection" problem: The existing "solid color positioning points" will fail due to loss of contrast against the same color background. The "key line positioning points" of this invention adopt a high-contrast concentric layer design, which ensures that high-contrast edges are always visible regardless of the background color, thus fundamentally guaranteeing the detectability of the positioning points.

[0103] (b) Solving the problem of "false detection": Existing technologies are prone to misidentifying "environmental noise" as a location point.

[0104] The tracking algorithm of this invention, by combining Kalman filtering with distance-based data association, can actively filter out environmental noise specks and effectively eliminate false detections.

[0105] 3. Efficiency of the tracking algorithm

[0106] The tracking algorithm of this invention (including preprocessing, blob detection, data association, and filtering) has low computational overhead. Experiments show that compared with unfiltered blob detection, this invention introduces only minimal additional overhead (e.g., in one computational example, the average processing time for tracking keypoints in a cumulative 1628 frame sequences increases from 6.04 ms to 6.13 ms), supports high-frequency real-time operation (e.g., above 120 Hz), and has high engineering practicality.

[0107] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A see-through skin sensing system that simultaneously provides visual and tactile perception, characterized in that, The device includes a computing device and a sensor installed at the end of an actuator. The sensor includes a sensor housing with an open top, a camera module and an LED lighting module installed inside the sensor housing, and a transparent silicone deformable layer mounted on the open top of the sensor housing via a bracket. The transparent silicone deformable layer has a keyline positioning point array with known original coordinates, and the coordinates of the keyline positioning point array change as the transparent silicone deformable layer contacts the end of the actuator. The camera module is used to acquire real-time image sequences of the external environment through a transparent silicone deformation layer; wherein, the real-time image sequence is an image sequence superimposed by a keyline positioning point array and the external environment; The LED lighting module is used to provide a continuous light source during the operation of the sensor; The computing device is used to calculate the displacement of each keyline positioning point according to the real-time image sequence and provide the displacement as a tactile signal to the actuator; at the same time, the computing device also re-acquires the current coordinates of each keyline positioning point according to the displacement, and then generates a key point region mask to eliminate the keyline positioning point array on the real-time image sequence according to the current coordinates, and finally provides the obtained external environment image sequence as a visual signal to the actuator. The method by which the computing device calculates the displacement of each keyline positioning point based on a real-time image sequence is as follows: Step 1: For the current frame image Preprocessing is performed, which includes distortion correction, grayscale conversion, and minimum-maximum value normalization. Pixels with brightness below a brightness threshold in the preprocessed image are set to 0, and the rest are set to 255, resulting in the grayscale binary image of the current frame. ; Step 2: Convert the current frame to a grayscale binary image. Perform a blob detection operation to obtain a candidate blob set. Among them, candidate spot set This includes both the actual location points and environmental noise points caused by the external environment; Step 3: Reset the previous frame image The obtained posterior estimates of the coordinates of each positioning point Each point in the candidate blob set is matched with a posterior estimate of its coordinates. The coordinates of the nearest blob are used as the observations of each location point in the current frame. : in, Represents the candidate spot set The position coordinates of any spot in the array; Step 4: Observe each positioning point in the current frame The input is a Kalman filter, and the output of the Kalman filter is the current frame image. The corresponding posterior estimates of the coordinates of each positioning point Specifically: in, For the current frame image The prior predicted coordinates of each location point. The current frame image And the previous frame image The posterior estimates of the coordinates of each location point. The current frame image And the previous frame image Kalman filter gain, The current frame image And the previous frame image The covariance of the prior predicted coordinates of each location point The current frame image And the previous frame image The covariance of the posterior estimates of the coordinates of each location point. The variance of the Gaussian distribution of process noise. To observe the Gaussian distribution variance of the noise, for The identity matrix; Step 5: Calculate the current frame image respectively The corresponding posterior estimates of the coordinates of each positioning point Compared with the original coordinates Displacement between .

2. The see-through skin sensing system that simultaneously provides visual and tactile perception as described in claim 1, characterized in that, The keyline positioning point array consists of multiple layers of concentric positioning points, and the contrast between each layer of positioning points is greater than a set value. The size of each positioning point is greater than the resolution of the camera module, and the center distance between adjacent positioning points is greater than the maximum tangential deformation that may occur in the transparent silicone deformation layer.

3. A see-through skin sensing system that simultaneously provides visual and tactile perception as described in claim 2, characterized in that, The transparent silicone deformation layer is a cuboid with a length and width in the centimeter range and a thickness in the millimeter range. Its surface is coated with two layers of key line positioning point arrays. The size of the positioning points in both layers is in the millimeter range, and the size of the positioning points in the bottom layer is smaller than that in the top layer. The color contrast between the two layers of positioning points is greater than a set value.

4. A see-through skin sensing system that simultaneously provides visual and tactile perception as described in claim 1, characterized in that, The method for preparing a transparent silicone deformable layer includes the following steps: S1: 3D printing is used to create a support frame and mold frame for the transparent silicone deformable layer; a transparent acrylic support layer is embedded inside the support frame with glue, and the mold frame is fixed above the support frame; transparent polarizing silicone A and B components are mixed in a 1:1 ratio, and after being fully mixed, they are injected into the mold frame. After the silicone solidifies naturally, the mold frame is removed to form the silicone layer of the transparent silicone deformable layer. S2: Cut out mask A and mask B with the same shape as the silicone layer using a laser cutting machine; cut out an array of circular holes at the same position on mask A and mask B respectively, with the position of the circular holes being the same as the position of the positioning point to be sprayed; S3: Place mask A on the surface of the silicone layer, spray black pigment with a spray gun, remove mask A, and let it dry naturally to form black positioning points on the surface of the silicone layer. S4: Place mask B on the surface of the silicone layer with black positioning points already formed, so that the array of black positioning points is aligned with the array of circular holes of mask B, and each black positioning point is located in the center of the corresponding circular hole; after spraying white pigment with a spray gun, remove mask B and let it dry naturally, so that white positioning points are formed on the surface of the silicone layer above the black positioning points, thus obtaining the final transparent silicone deformation layer.

5. A see-through skin sensing system that simultaneously provides visual and tactile perception as described in claim 1, characterized in that, initial state At any given moment, the original coordinates of the keyline positioning point array The covariance is obtained through sensor fabrication and camera calibration parameter calculations, or through one-time manual annotation. Initialize to A matrix consisting entirely of zeros.

6. A see-through skin sensing system that simultaneously provides visual and tactile perception as described in claim 1, characterized in that, It also includes a power supply module and a data transmission circuit board; The power supply module is used to supply power to the camera module and the LED lighting module; The data transmission circuit board is used to transmit real-time image sequences to computing devices.

7. A see-through skin sensing system that simultaneously provides visual and tactile perception as described in claim 1, characterized in that, The computing device uses the Telea algorithm to eliminate the array of keyline localization points on the real-time image sequence.

Images