A high-density feature-based visual-haptic sensor and a feature generation and tracking method thereof

Abstract

This invention discloses a visual-tactile sensor based on high-density feature design and its image generation and tracking method. Addressing the problems of matching ambiguity and low tracking density caused by homogenized marker points in existing technologies, this invention first generates a globally unique feature pattern through an iterative algorithm with dual constraints in the spatial and color domains. Then, it utilizes adaptive field-of-view adjustment to compensate for feature distribution under different camera positions. This pattern is embedded in the sensor's tactile perception layer. During tracking, this invention employs a two-level cascaded matching logic, combined with reference frame modeling, dynamic resolution mechanisms, and mean normalization processing, to achieve displacement capture from pixel-level coarse matching to sub-pixel-level super-resolution. This invention significantly improves the sensor's displacement field tracking density, effectively eliminates feature confusion under large deformations, and enables high-precision perception at the full-field pixel level, providing a reliable high-density tactile feedback foundation for the refined operation of robots.

Description

Technical Field

[0001] This invention relates to the field of robotics and sensor technology, specifically to robot tactile perception and visual-tactile sensing technology, and particularly to a visual-tactile sensor based on high-density feature design and its image generation and tracking method. Background Technology

[0002] In the fields of precision robotic operations and human-computer interaction, visual-tactile sensors are key components for acquiring high-dimensional information about contact surfaces, and their performance directly affects a robot's ability to perceive the geometric features and force states of objects. Traditional visual-tactile sensors typically consist of a built-in camera, an illumination source, and a flexible sensing layer covering a rigid support layer. Discrete marker features are distributed inside or on the surface of the sensing layer. For example, the classic approach represented by GelSight typically uses a sparse dot matrix of 20×20 pixels printed on a flexible body as a tracking reference. When an external object comes into contact with the flexible sensing layer and causes deformation, the built-in camera captures the positional shift of these marker dots in image space, and combines this with relevant physical models or algorithms to estimate the morphology and force distribution of the contact surface.

[0003] However, existing marker feature tracking schemes have fundamental limitations when dealing with high-precision sensing requirements. Because marker feature points in existing technologies are typically designed as physical entities with completely identical shape and color—for example, all being black dots of the same diameter—each marker lacks spatially independent physical properties in optical imaging. In practical applications, when the flexible contact layer undergoes significant tensile, shear, or compressive deformation, the marker points experience significant physical displacement. For example, in a scenario involving severe shear deformation, a marker originally located in a certain area might move to the initial coordinate range of an adjacent marker. Since all marker points have extremely high visual similarity, tracking algorithms are prone to confusion during matching, failing to accurately determine which original marker in the reference frame the currently captured point corresponds to. This matching ambiguity caused by feature homogenization severely restricts the robustness of sensors under dynamic, large-deformation conditions.

[0004] To circumvent the aforementioned matching confusion problem, existing technologies are often forced to adopt a strategy of increasing the spacing between marker points in their physical design, sacrificing spatial distribution density for matching determinism. This compromise of trading distance for stability keeps the sensor's feature point coefficients at a consistently low level, typically achieving deformation field tracking resolution of only a few hundred sampling points. When faced with force field distributions that require the identification of minute textures, fine geometric shapes, or high spatial frequencies, this sparse sampling method causes the physical loss of crucial deformation information, creating a perception blind spot. Consequently, the displacement field tracking is not dense enough, making it difficult to achieve true full-field pixel-level deformation capture and high-precision topography reconstruction. Summary of the Invention

[0005] This invention provides a visual-tactile sensor based on high-density feature design and its image generation and tracking method, aiming to solve the technical problems of low displacement field tracking density and easy matching ambiguity under large deformation conditions caused by the sparse feature markers and insufficient specificity of existing visual-tactile sensors.

[0006] The core concept of this invention lies in actively designing and constructing a high-density feature layer image with globally unique identification characteristics through an algorithm. In the generation logic of the feature image, it abandons the traditional random or fixed dot matrix distribution and instead adopts an iterative design scheme that combines spatial and color domain constraints. This scheme ensures that the texture features of each local region in the image are unique across the entire field of view by imposing upper and lower limits on the spatial distribution distance of feature points and introducing distance constraints within the color space. This makes each pixel and its neighborhood within the feature layer a recognizable identifier, eliminating ambiguity in feature matching from the underlying data source level and laying the physical foundation for pixel-level dense tracking.

[0007] In terms of sensor hardware implementation, a multi-layered composite tactile sensing structure is constructed, integrating a hard transparent layer, an elastic transparent layer, a feature layer, and a contact layer sequentially from the inside out, and working in conjunction with a built-in optical imaging unit. To ensure consistent sensor perception across different application scenarios, this invention also provides an adaptive feature density distribution design for camera position. By dynamically adjusting the feature area coverage of different regions according to the camera's installation angle, the influence of affine transformation during optical imaging is effectively offset, ensuring that the final acquired image maintains uniform and stable feature resolution across the entire field of view.

[0008] Based on the aforementioned high-density feature layer, this invention further designs targeted feature tracking logic to obtain high-precision displacement field information. This method first establishes a stable reference benchmark by modeling multiple frames of static images, and then utilizes neighborhood interpolation technology to achieve sub-pixel enhancement of features at the mathematical level. In actual tracking, this invention employs a feature correlation algorithm combined with a dynamic resolution mechanism. Through dual comparison of pixel-level coarse matching and sub-pixel-level local refinement between the reference frame and the current frame, it achieves accurate capture of minute displacements in the flexible layer. This deeply coupled hardware and software solution not only greatly improves the spatial resolution of deformation tracking but also ensures the system's tracking robustness under scenarios of fluctuating illumination or large displacements.

[0009] Specifically, the first aspect of this invention provides a method for generating images for a visual-tactile sensor based on high-density feature design. First, seed points are randomly initialized within a preset image space, and minimum and maximum distribution spacing constraints are set for these points. Based on this, the scheme introduces color space distance constraints. An algorithm updates the color values ​​of the seed points in real time according to their physical displacement, ensuring that seed points that are physically close remain maximally far apart in the color domain. Finally, through iterative optimization combining spatial and color constraints, a feature pattern is generated where the local texture is uniquely identifiable globally. This scheme eliminates the homogenization defect of feature points at the underlying data source level, solves the matching ambiguity problem that easily arises under large deformation conditions, and lays a physical foundation for achieving high-density displacement tracking.

[0010] Preferably, during iterative optimization, the spatial repulsion force between each initial seed point is calculated by simulating a physical repulsion mechanism, and the resultant force is used as the driving force to dynamically update the spatial coordinates of each point. This iterative process continues until the distribution of all seed points fully satisfies the preset minimum and maximum spacing constraints. By calculating the spatial repulsion force, the uniformity of the sampling points across the entire field is ensured while preventing feature points from sticking together.

[0011] Furthermore, when calculating the spatial repulsion force, several virtual seed points are virtually set at the physical boundaries of the image to generate boundary repulsion forces. This boundary force, along with the mutual repulsion between the internal seed points, acts on the coordinate update algorithm to adjust the force balance of edge feature points. This avoids excessive clustering or sparseness of feature points in the image edge regions, ensuring consistent perception density between the sensor's field of view edges and the central region.

[0012] Preferably, based on the specific installation angle of the camera or video camera inside the sensor, such as a side-view camera position, the feature point area coverage of different regions of the image is dynamically compensated and adjusted during the pattern generation process. This effectively counteracts the effects of perspective projection or affine transformation during the optical imaging process, ensuring that the final captured deformed image maintains uniform feature resolution across the entire field of view.

[0013] Furthermore, when outputting feature patterns, morphological smoothing is performed on the patterns according to application requirements, thereby constructing a binary or grayscale image with clear edges. Alternatively, spatial diffusion is performed using a color Gaussian kernel function to generate a color image with a smooth color gradient. Based on this design, the quality and signal-to-noise ratio of the feature image can be improved, providing excellent edge feature support for subsequent high-precision mathematical interpolation and sub-pixel search.

[0014] Furthermore, a second aspect of the present invention provides a method for feature tracking using feature images generated by the aforementioned method. This method captures image sequences of the feature layer before and after force application, respectively serving as a reference frame and the current frame. In algorithm implementation, firstly, the color similarity between the current frame's sub-window and the search area of ​​the reference frame is calculated to lock pixel-level coarse matching coordinates. Then, a fine local search is performed within the neighborhood of the coarse matching coordinates to determine sub-pixel-level refined matching points. This two-stage cascaded tracking method achieves dense displacement tracking across orders of magnitude, improving the perceived spatial resolution to the pixel level or even sub-pixel level, greatly enhancing the ability to reconstruct fine shapes.

[0015] Preferably, a dynamic response logic is introduced into the tracking algorithm, which can automatically switch between lower pixel-level matching accuracy and extremely high sub-pixel-level matching accuracy based on the degree of deformation sensed in a local area. This significantly optimizes the allocation of computing resources and effectively improves the real-time computing efficiency of the sensor while ensuring high sensitivity to minute contact signals.

[0016] Furthermore, before initiating the tracking algorithm, multiple frames of static feature images are continuously acquired and their pixel averages are calculated to establish a highly stable reference frame baseline model. This operation effectively suppresses random electrical noise generated by the camera sensor, providing a clean and high-contrast baseline map for accurate displacement field calculation.

[0017] Furthermore, before performing similarity calculations for two-level cascaded matching, mean normalization processing can be performed on the sub-window regions of the current image in advance to offset image brightness shifts caused by external ambient light fluctuations or dynamic shadows. This can significantly improve the tracking stability of the sensor under complex lighting conditions.

[0018] A third aspect of the present invention provides a visual-tactile sensor having a physical entity comprising an optical imaging unit and a multi-layered tactile sensing layer including a hard transparent layer, an elastic transparent layer, a feature layer, and a contact layer. The feature layer carries a unique feature pattern generated through a specific constraint method and is equipped with the aforementioned feature tracking algorithm. Overall, a complete hardware closed loop from physical contact to high-density digital displacement output is achieved, providing robots with a tactile sensing hardware foundation possessing extremely high robustness and ultra-high resolution.

[0019] Compared to traditional visual-tactile sensor solutions based on discrete marker points, this invention significantly improves the spatial resolution of deformation sensing by implementing high-density digital feature design and corresponding pixel-level / sub-pixel-level tracking algorithms. Because this solution dramatically increases the resolution of displacement field tracking from the sparse sampling scale of traditional point arrays to the camera pixel level and even the sub-pixel level, the deformation field tracking density achieves an order-of-magnitude increase, resulting in approximately a hundredfold improvement in feature density compared to traditional methods. This extremely high tracking density and sub-pixel-level tracking accuracy enable the sensor to accurately capture subtle deformations of the contact surface, greatly enhancing its ability to perceive complex geometries and distributed force fields.

[0020] Furthermore, because the optimized digital patterns used in this invention possess highly specialized texture attributes, it ensures that every local region in the image has globally unique recognition features. This specialized design fundamentally solves the matching ambiguity and identity confusion problems that easily arise in traditional homogeneous marker points under large deformation scenarios. Even under extreme conditions such as uneven lighting, localized image blurring, or large physical displacement, this invention can still maintain robust tracking performance through stable feature association logic, significantly improving the reliability and robustness of the sensor in complex interactive environments.

[0021] Furthermore, this invention achieves better adaptability to different camera positions by dynamically adjusting the density distribution of feature images. Addressing the affine transformation problem caused by different installation angles such as frontal or side views, this invention can dynamically set the feature area coverage rate by dividing the region according to the angle between the camera and the feature layer. This effectively offsets the difference in near and far field resolution caused by perspective projection, ensuring that the sensor can acquire consistent and uniform feature tracking density at different image positions, thereby maintaining the uniformity of sensing performance across the entire field of view. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a typical visual-tactile sensor structure and a dot matrix in the prior art; Figure 2 This is a cross-sectional structural diagram of a visual-tactile sensor provided in a specific embodiment of the present invention; Figure 3 This is a flowchart illustrating a feature image generation method according to a specific embodiment of the present invention. Figure 4 This is a schematic diagram of dense features obtained by the above generation method according to a specific embodiment of the present invention, wherein the left image is black and white speckle, the middle image is grayscale speckle, and the right image is colored speckle; Figure 5 A schematic diagram of the logic flow of a feature tracking algorithm provided in a specific embodiment of the present invention; Figure 6This is a schematic diagram of the displacement field tracking effect based on the above tracking algorithm provided in a specific embodiment of the present invention; Figure 7 This is a schematic diagram of the overall logic flow of sensing and data processing provided in a specific embodiment of the present invention.

[0023] Reference numerals: 1-Outer shell; 2-Camera or camera; 3-Light source; 4-Rigid transparent layer; 5-Elastic transparent layer; 6-Feature layer; 7-Contact layer. Detailed Implementation

[0024] To better understand the technical solutions of the present invention, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that those skilled in the art can implement the present invention in other forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0025] like Figure 1 As shown, in existing visual-tactile sensor technologies, discrete marker arrays are typically deployed within the flexible sensing layer. Taking the typical GelSight solution as an example, it often uses a 20×20 black dot matrix as a reference for deformation tracking. During sensor operation, a built-in camera captures the spatial displacement of these dots as the flexible body is compressed, thus constructing a sparse displacement field distribution. However, a deeper analysis of the physical characteristics of this existing solution reveals that, because all markers maintain a high degree of consistency in color, shape, and geometric size, these features are essentially homogeneous in visual terms.

[0026] This homogenization of features leads to a significant technical bottleneck: when the sensor faces severe shearing or large-scale compression deformation, marker points in a certain area are prone to significant shifts and fall within the initial coordinate range of their neighboring marker points. Lacking unique identification features, the algorithm struggles to determine which original coordinates the point belongs to during image association matching, resulting in serious matching ambiguity or identity confusion. To mitigate this failure risk, existing technologies are forced to increase the physical spacing between marker points, sacrificing spatial distribution density for tracking stability. This design compromise often limits the final tracking density to the order of hundreds of sampling points, failing to capture sub-pixel-level fine textures and minute deformations.

[0027] To address the limitations of existing technologies, the core of this invention lies in breaking away from the conventional thinking of discrete marker points and proactively designing a high-density feature image with global uniqueness through algorithms. The high-density features employed in this invention are no longer simple geometric repetitions, but rather iteratively generated under dual constraints of the spatial and color domains, ensuring that every pixel and its neighborhood in the image possess an uncopyable texture fingerprint. This design not only eliminates the identity confusion problem under large deformations at its source but also enables deformation tracking resolution to leap directly from the hundreds of points to the camera pixel level, thus laying the foundation for achieving full-field pixel-level, highly robust visual-tactile perception.

[0028] like Figure 2 As shown, the visual-tactile sensor provided in this embodiment of the invention mainly consists of an optical imaging unit and a multi-layered composite tactile sensing layer. The optical imaging unit includes a camera or video camera 2 housed within the housing 1 and an illumination source 3 positioned near the tactile sensing layer, providing a stable imaging environment and capturing real-time images of the deformation process of the tactile sensing layer. The tactile sensing layer, as the physical core of the sensing mechanism, consists of a rigid transparent layer 4, an elastic transparent layer 5, a feature layer 6, and a contact layer 7, stacked sequentially from the inside out, i.e., from the side closest to the camera or video camera 2 to the side furthest away from it. These layers are physically bonded or chemically bonded to form a stable stacked coupling relationship.

[0029] The rigid transparent layer 4 is typically made of glass or a transparent polymer material with high light transmittance and high rigidity. Its main function is to support the overall structure, maintain the stability of the imaging optical path, and protect the internal optical components. Adjacent to the rigid transparent layer 4 is an elastic transparent layer 5, which is generally made of a low-modulus flexible transparent material, such as transparent silicone. Its physical properties allow it to undergo perceptible elastic deformation when subjected to external contact forces. The key aspect of the hardware design of this invention lies in the feature layer 6, which is precisely attached to the interface between the elastic transparent layer 5 and the contact layer 7. This feature layer 6 carries a high-density feature image with globally unique identification characteristics, generated through an iterative algorithm. In terms of manufacturing process, the feature layer 6 can be attached to the surface of the elastic transparent layer 5 through methods such as transfer printing, spraying, surface etching, masking, or projected patterns, forming a tight motion coupling with the elastomer. This ensures that the feature layer 6 can undergo synchronous, lag-free spatial displacement with the deformation of the elastic transparent layer 5.

[0030] The outermost contact layer 7 directly contacts the external target object, serving to transfer stress and protect the internal feature pattern. During the coordinated process of optical imaging and stress transfer, when the external target object presses on the contact layer 7, the contact stress is directly transferred to the feature layer 6 through the contact layer 7, inducing a microscopic or macroscopic displacement field in the feature layer 6 and its attached elastic transparent layer 5. Due to the high transparency of the hard transparent layer 4 and the elastic transparent layer 5, the light emitted by the light source 3 of the optical imaging unit can pass through the transparent medium and illuminate the feature layer 6, which is then captured by the camera or video camera 2 to show the dynamic evolution of the pattern on the feature layer 6 before and after the stress. This multi-layered composite architecture, through precise spatial layout, transforms complex physical contact information into high-contrast optical feature displacement, thus providing a highly reliable hardware platform for subsequent pixel-level / sub-pixel-level tracking algorithms. Furthermore, for different camera or video camera 2 positions, the feature layer 6 can be pre-set with an adaptive area coverage distribution in its spatial layout to counteract the effects of affine transformation at the hardware level, ensuring uniformity of the entire field of perception.

[0031] Based on the aforementioned hardware architecture, this embodiment of the invention designs and generates a high-density feature image with globally unique identification characteristics through an iterative algorithm that combines spatial and color domain constraints. For example... Figure 3 As shown, the specific process of this feature image generation method is as follows: First, N seed points P are randomly initialized within the preset image space. i (x i , y i This invention sets spatial distribution constraints for the algorithm. To prevent excessively dense feature points from causing adhesion during physical processing, and to ensure sufficient texture information within the imaging window to avoid feature loss regions, the present invention introduces the minimum distance d between seed points. min and maximum distance d max Limitations. Assume the coordinates of the generated seed point are... Require any two seed points P i With P j The distance between them satisfies , as well as .

[0032] In one embodiment, assuming the image size is 600×400, the diameter D of the circular spot is set to 5, and the area coverage η of the entire image is set to 50%, then the number of random seed points N can be calculated as N = (600×400×η) / (0.25×πD). 2 ), that is, N=6115. Minimum seed point distance d min It can be set to be slightly larger than the diameter of the circular spot, approximately 1.5 times the diameter D, with a maximum distance d. max Then it can be set to about twice the value of D.

[0033] During the iterative generation process, a repulsive force is defined between two seed points, which is inversely proportional to the distance. Let k be a set elastic coefficient, and the distance between the two seed points be... Then the repulsive force The definition is as follows.

[0034] Furthermore, to ensure the uniformity of feature distribution at image edges, a boundary repulsion force F generated by virtual seed points at image boundaries is introduced. boundary That is, during the calculation, an additional ring of virtual seed points is created at the image boundary, and these virtual seed points participate in the F... boundary Calculate, and the calculation method is the same as The results are consistent, but the virtual seed points do not participate in the iterative update and have fixed coordinates. Calculate the resultant force on each seed point. .

[0035] After the iteration begins, the seed point coordinates are updated based on the distance gradient. ,in This is the iteration step size. After multiple iterations, the iteration stops when the displacement change of all points is less than a set threshold.

[0036] After optimizing the spatial coordinates, a color dimension constraint is further introduced to enhance feature discriminability. By assigning colors to seed points, points that are physically close are geographically far apart in the color space. Initially, each seed point is randomly assigned a color C. This step can be completely randomized or selected from a preset high-contrast color sequence. Color space distance is defined as follows: And calculate the corresponding color repulsion force. This guides the seed point color C. i Update along the color gradient direction. In the color space distance formula, C... i and C j R represents the color of different seed points. i R j G i G j B i B j The values ​​of the three primary colors, red, green, and blue, are assigned to the seed. The color repulsion force is calculated using the following formula: After the iteration begins, update the seed point color according to the color gradient. ,in This is the iteration step size. After multiple iterations, the iteration stops when the color change of all points is less than a set threshold. This mechanism forces seed points that are close in physical space to be kept as far apart as possible in the RGB color space, thereby greatly improving the density of local textures.

[0037] like Figure 4 As shown, this method can produce dense feature maps of various forms depending on different perception requirements. Figure 4 The left side shows a binary speckle image. For a binary image, all the growing circular patches are set to 1 with the seed point as the center, and the rest of the background is set to 0. The merged patches are checked for excessively fine connections or sharp corners. Morphological operations are used to smooth the image to ensure that grayscale interference is not caused by anti-aliasing during imaging. Figure 4 The middle section is a grayscale speckle image, providing richer tonal gradations. For grayscale images, Gaussian speckles can be generated using a Gaussian kernel, ultimately producing a grayscale image. Figure 4 The image on the right is a color speckle image. Centered on a seed point, a color Gaussian kernel function is used to spatially diffuse the color of the seed point, forming smooth color patches. All color patches are then superimposed onto the background to generate the final color pattern.

[0038] The present invention ensures that every pixel in the feature image has traceable features through the above design. The neighborhood of each pixel in the full field of view, such as the 21×21 sub-window used later, has a unique texture fingerprint, thereby fundamentally eliminating the possibility of matching confusion.

[0039] Furthermore, for non-frontal viewing scenarios such as side-viewing by the camera or webcam, i.e., differences in the position and angle of the camera or webcam relative to the feature layer, this embodiment achieves field-of-view adaptation by dynamically adjusting the area coverage η of the feature image. Specifically, for a camera facing the feature map directly, the entire field of view is set to the same coverage to generate the feature map; for side-viewing features, the algorithm divides the feature layer region into m regions along the direction away from the camera, and the coverage η gradually increases from near to far, with the change value consistent with the angle between the camera and the feature layer. This dynamic compensation of distribution density effectively counteracts the influence of affine transformation during the imaging process, ensuring that the final captured image maintains a consistent feature tracking density at different positions.

[0040] After generating a high-density feature layer, this embodiment of the invention further achieves high-precision capture of the deformation field of the flexible layer through a two-stage cascaded feature tracking algorithm adapted to the image of this feature layer. For example... Figure 5 As shown, the tracking logic starts by modeling the reference frame, calculates the feature changes between the reference frame and the current frame, and outputs an ultra-high resolution pixel-level displacement field.

[0041] During the algorithm initialization phase, reference frame modeling is performed first. To eliminate ambient photoelectric noise and establish a robust baseline state, the system continuously acquires M images in a stationary state without sensor pressure; for example, M=30 in this embodiment. The reference frame model is constructed by calculating the following pixel mean.

[0042] Where M is the number of images, t is the count from 1 to M, and I is the average intensity value matrix of the image pixels. t It is the intensity value matrix of the image pixels of an image.

[0043] Subsequently, for the background image obtained from the modeling, sub-pixel-level feature enhancement is achieved using neighborhood third-order spline interpolation. A core design feature of this invention is the introduction of a dynamic resolution mechanism to balance accuracy and efficiency. In static regions where no deformation is detected, the system maintains lower pixel accuracy to optimize computational resource allocation. However, once a slight deformation trend is detected, the accuracy at that local location is automatically increased to a sub-pixel level of 0.1, thereby significantly improving the sensitivity to subtle contact signals while ensuring system real-time performance.

[0044] In actual tracking, the algorithm performs a first-stage pixel-level coarse matching for the current frame image. Taking a feature image with a resolution of 600×400 as an example, a region with a size of, for example, 21×21 pixels is selected as a sub-window. For a sampling point with coordinates (u, v) in the current frame, the algorithm delineates a search region of ±30 pixels around the coordinates of its corresponding reference frame, and slides the sub-window with a step size of 1 pixel, calculating the cross-correlation coefficient between the current frame sub-window and the corresponding reference frame one by one.

[0045] To eliminate interference from lighting fluctuations or shadows, the algorithm first performs mean normalization on the color values ​​within the current sub-window, calculated using the following formula: , Among them, f i The original color value of the current child window. The normalized color value.

[0046] Subsequently, the algorithm calculates the similarity of the energy distribution of the current frame sub-window and the reference frame search window in the R, G, and B channel components. Let the normalized window of the current frame be... The reference frame sub-window to be calculated is normalized to The three channel components are respectively and Define the product of corresponding pixels within the ik window and then sum them as: , and The total energy of pixels within a sub-window is defined as the sum of the squares of the pixels. and The specific calculation method for the cross-correlation coefficient is as follows: .

[0047] By searching for the maximum value of the correlation coefficient C within the region, preliminary matching coordinates at the integer pixel level are determined.

[0048] After obtaining the initial matching coordinates, the algorithm enters the second stage of sub-pixel-level high-precision refinement. Within the locked matching coordinates ±1 pixel range, virtual pixels are generated using sub-pixel interpolation accuracy of 0.1, and the above cross-correlation matching process is repeated. This cascaded strategy from global coarse search to local refinement ensures that the tracking results can accurately reflect the subtle displacements of the feature layer at the micrometer scale.

[0049] This ultimately results in high-density pixel tracking. For example... Figure 6 As shown, this invention ultimately outputs an extremely high-density deformation tracking field. Compared to traditional dot matrix schemes that can only provide sparse tracking points of approximately 20×20, this invention achieves a resolution of 600×400, or 240,000 tracking points, even without considering sub-pixel accuracy. Furthermore, by combining sub-pixel tracking technology, the equivalent tracking density can be increased to over 2000×2000. This leap from sparse to dense allows the sensor to detect dense feature displacement vectors, such as... Figure 6 The middle arrow indicates that the full-field pixel-level displacement distribution of the flexible layer under contact pressure is accurately reproduced, providing a complete data foundation for subsequent high-precision morphology reconstruction and force field analysis.

[0050] It should be noted that the specific structures, processes, algorithm logic, and parameters described in the foregoing embodiments are merely preferred embodiments of the present invention and do not constitute an absolute limitation on the scope of protection of the present invention. Those skilled in the art should understand that, without departing from the core concept of the present invention, equivalent substitutions or modifications can be made to the above solutions according to specific application scenarios and hardware conditions.

[0051] Regarding the physical construction of the feature layer, this invention encompasses a variety of processing methods. Besides water transfer printing, the feature layer can also be directly attached to the surface of the elastic transparent layer using precision processing techniques such as spraying, surface etching, masking, or vacuum deposition. In some non-physical contact implementation scenarios, the feature layer can even project specific patterns onto the surface of the elastic transparent layer using a projection device to form a controlled optical feature layer. Furthermore, the shape of the feature pattern is not limited to two-dimensional planar images; it can also be constructed using three-dimensional processing methods such as micro-nano fabrication or 3D printing to create dense feature structures with three-dimensional topological features on or within the elastic layer, thereby capturing more complex tactile deformation information in multi-dimensional space.

[0052] At the level of feature image generation algorithms, in addition to iterative schemes based on simulated repulsion, this invention also supports other generation logics with spatial constraints. For example, the Voronoi diagram method can be used to divide the target image region into several sub-regions, and seed points can be randomly generated within each sub-region according to a specific distribution pattern. Alternatively, a generation method based on Poisson disk sampling can be used, dynamically determining whether seed points meet preset maximum / minimum distance limits when randomly adding them, and performing retention or discard operations accordingly, thereby generating high-density feature patterns with equal uniqueness.

[0053] In implementing feature tracking algorithms, besides the aforementioned normalized cross-correlation algorithms based on grayscale or color, those skilled in the art can also invoke other computer vision feature operators for matching, depending on real-time or robustness requirements. For example, feature operators such as SURF (Speed-Up Robust Features) or SIFT (Scale-Invariant Feature Transform) can be used to extract and calculate the local feature vector of each sampling point in the image. By calculating the Euclidean distance, Manhattan distance, or Hamming distance between the feature vectors of the current frame and the reference frame, accurate localization and tracking of high-density feature points can also be achieved.

[0054] The aforementioned changes in physical structure, generation algorithm, or tracking operator essentially utilize the core inventive concept of this invention: by actively constructing dense features with unique identifiability, the invention eliminates matching ambiguity at the physical and algorithmic levels and achieves full-field pixel-level deformation tracking.

[0055] Combination Figure 7 As shown, the visual-tactile perception system based on high-density feature design provided in this embodiment of the invention constructs a complete lifecycle closed loop from digital feature design to digital restoration of deformation field through deep collaboration between software and hardware.

[0056] First, in the feature image generation stage, this system utilizes the aforementioned spatial and color domain dual-constraint iterative algorithm to synthesize high-density feature patterns with globally unique recognizability within the computer. This stage is the source of perception; by actively intervening in the feature distribution through the algorithm, it ensures that every local region within the entire field of view possesses non-repeatable texture recognition features, laying a mathematical foundation for eliminating subsequent matching ambiguities.

[0057] Secondly, in the processing and transfer stage, the system transforms the digital pattern designed above into a physical entity. The feature image is precisely attached to the interface between the elastic transparent layer and the contact layer through precision processing techniques such as water transfer printing, spraying, etching, or masking. This step realizes the leap from virtual features to a physical sensing layer, enabling the feature layer to undergo synchronous displacement mapping with the compressive deformation of the flexible body.

[0058] Next, in the feature extraction stage, the optical imaging unit captures the dynamic evolution of the feature layer in real time and performs preprocessing. The system calculates the mean of multiple static images to establish a high signal-to-noise ratio reference frame model, and introduces neighborhood third-order spline interpolation technology to perform sub-pixel-level mathematical enhancement of the features. The purpose of this stage is to extract the clearest and most stable feature benchmarks from the image sequence, preparing for high-precision displacement analysis.

[0059] Subsequently, the pixel tracking stage begins. Based on the reference frame and the current frame, the system executes a two-stage cascaded feature correlation algorithm. Under the scheduling of the dynamic resolution mechanism, the algorithm first eliminates illumination interference through mean normalization and performs pixel-level coarse cross-correlation matching within a preset range. Then, within the initially locked coordinate neighborhood, local fine-tuning is performed using sub-pixel precision of 0.1. This stage achieves precise capture of the displacement of each pixel in the feature layer.

[0060] Thus, the technical solution of this invention can complete the recovery of the deformation field. The system integrates tens of thousands of characteristic displacement vectors and maps them to physical space, ultimately outputting high-density full-field displacement or deformation field data.

[0061] This invention completely breaks the mutual constraint between sampling density and tracking robustness in traditional visual-tactile sensors by coupling unique features with cascaded tracking depth, achieving a tracking density a hundred times higher than that of traditional dot matrix schemes, and providing complete data support for robots to achieve pixel-level precision shape reconstruction and fine tactile feedback.

[0062] The technical solutions of the present invention have been described in detail above with reference to specific embodiments. However, the above embodiments are only used to illustrate the technical concept of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or substitutions can be made to the technical solutions involved in the present invention without departing from the overall technical concept of the present invention. All equivalent substitutions, modifications, or improvements made to the present invention by those skilled in the art based on their understanding of the technical solutions of the present invention, as long as they do not depart from the technical concept and essence of the present invention, should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A method for generating feature images of a visual-tactile sensor based on high-density feature design, characterized in that, Includes the following steps: Randomly initialize seed points within a preset image space, and set spatial constraints for the minimum and maximum distribution spacing of the seed points; and / or, By introducing color space distance constraints, the color values ​​of various child points are dynamically updated based on the physical distance between seed points, so that physically adjacent seed points remain far apart in the color domain. Iterative optimization is performed based on the spatial constraints and / or the color space distance constraints to generate feature patterns with globally unique identifiable local textures.

2. The method according to claim 1, characterized in that, During the iterative optimization process, the coordinates of each point are dynamically updated by calculating the spatial repulsion force between seed points until the distribution state between various seed points satisfies the constraints of the minimum distribution spacing and the maximum distribution spacing.

3. The method according to claim 2, characterized in that, When calculating the spatial repulsion force, a boundary repulsion force generated by a virtual seed point at the image boundary is introduced, so that the boundary repulsion force and the spatial repulsion force between seed points work together to update the coordinates of the seed points.

4. The method according to any one of claims 1 to 3, characterized in that, It also includes a field-of-view adaptive adjustment step that dynamically adjusts the feature point area coverage in different regions of the image based on the installation angle of the imaging camera.

5. The method according to any one of claims 1 to 3, characterized in that, When generating the feature pattern, morphological smoothing is performed on the pattern to construct a binary image or a grayscale image, or a color image with a color gradient is constructed using color Gaussian diffusion.

6. A method for feature tracking using a feature image generated by the method according to any one of claims 1 to 5, characterized in that, Includes the following steps: The feature pattern carried by the feature layer is obtained before and after being subjected to force, and is used as the reference frame and the current frame, respectively. A two-level cascaded matching is performed. First, the color component similarity between the current frame sub-window and the search area of ​​the reference frame is calculated to determine the pixel-level coarse matching coordinates. Then, a local search is performed in the neighborhood of the coarse matching coordinates to determine the sub-pixel-level fine matching coordinates, thereby obtaining high-density displacement field data.

7. The method according to claim 6, characterized in that, A dynamic resolution mechanism is introduced to automatically switch between pixel-level matching accuracy and sub-pixel-level matching accuracy based on the degree of deformation in local areas.

8. The method according to claim 6 or 7, characterized in that, Before performing the two-level cascaded matching, a reference frame modeling process is also included, which involves acquiring multiple frames of static images and calculating the pixel mean to establish a reference frame baseline model.

9. The method according to claim 6 or 7, characterized in that, Before performing the two-level cascaded matching, mean normalization is performed on the sub-windows of the current image to eliminate the interference of external ambient light fluctuations on the matching calculation.

10. A visual-tactile sensor, characterized in that, Includes an optical imaging unit and a multi-layered composite tactile sensing layer; The tactile sensing layer comprises, from the inside out, a hard transparent layer, an elastic transparent layer, a feature layer, and a contact layer; The feature layer carries a feature pattern generated using the method described in any one of claims 1 to 5; and the sensor performs feature tracking on the image acquired by the optical imaging unit using the method described in any one of claims 6 to 9.

Images