Tactile perception method and device of visual tactile sensor, storage medium, program product and computing equipment

Abstract

A tactile perception method and device of a visual tactile sensor, a storage medium, a program product and a computing device, the visual tactile sensor has a flexible layer and a camera, N marker layers are arranged in the flexible layer, a plurality of markers are distributed in each marker layer, and N is a positive integer greater than 1; the tactile perception method comprises the following steps: acquiring a to-be-processed image of a current frame, wherein the to-be-processed image of the current frame is obtained by performing image acquisition on a flexible layer by a camera; obtaining to-be-analyzed hierarchy information of the current frame, wherein the to-be-analyzed hierarchy information of the current frame is used for indicating at least one target marker layer in the N marker layers; processing the to-be-processed image of the current frame to at least determine a displacement field of at least one target marker layer of the current frame; and according to the displacement field of the at least one target marker layer of the current frame, determining an external force applied to the flexible layer of the current frame. According to the scheme provided by the embodiment of the invention, the visual tactile sensor can realize more accurate tactile perception.

Description

Technical Field

[0001] This application relates to the field of sensor technology, specifically to a tactile sensing method and apparatus, storage medium, program product, and computing device for a visual-tactile sensor. Background Technology

[0002] With the rapid development of technologies such as robotics and human-computer interaction, acquiring high-precision, multi-dimensional tactile information has become one of the important ways for intelligent systems to perceive their environment. As an emerging technology that combines visual and tactile perception capabilities, the visual-tactile sensor has attracted much attention due to its simple structure, ease of integration, and ability to perceive rich information. Summary of the Invention

[0003] One of the technical objectives of this application is to provide an improved tactile perception method, so as to enable visual tactile sensors to achieve more accurate tactile perception.

[0004] In a first aspect, embodiments of this application provide a tactile perception method for a visual-tactile sensor. The visual-tactile sensor has a flexible layer and a camera. N marker layers are disposed within the flexible layer, and multiple markers are distributed within each marker layer, where N is a positive integer greater than 1. The method includes: acquiring a to-be-processed image of the current frame, wherein the to-be-processed image of the current frame is obtained by the camera capturing an image of the flexible layer; acquiring layer information to be analyzed in the current frame, wherein the layer information to be analyzed in the current frame is used to indicate at least one target marker layer among the N marker layers; processing the to-be-processed image of the current frame to at least determine the displacement field of at least one target marker layer of the current frame; and determining the external force acting on the flexible layer of the current frame based on the displacement field of the at least one target marker layer of the current frame.

[0005] Optionally, the markers in different marker layers have different identifiable features; processing the image to be processed in the current frame to determine at least the displacement field of at least one target marker layer in the current frame includes: segmenting the image of the target marker layer from the image to be processed in the current frame based on the identifiable features of the markers in each target marker layer; and determining the displacement field of the target marker layer in the current frame based on the image of each target marker layer in the current frame.

[0006] Optionally, the identifiable feature is color. Before processing the image to be processed in the current frame to determine at least the displacement field of at least one target marker layer of the current frame, the method further includes: determining the standard color to which the pixel belongs based on the color characterization value of each pixel in the image to be processed in the current frame and the standard color characterization value of each layer marker; for each pixel, if the proportion of pixels with the same standard color in the neighborhood of the pixel is less than a preset proportion threshold, then the pixel is judged as noise and discarded; wherein, the size of the neighborhood is positively correlated with the depth of the marker to which the standard color of the pixel belongs in the flexible layer.

[0007] Optionally, the markers in different marker layers respond to different wavelengths, and the visual-tactile sensor further includes: a multi-wavelength light source; before acquiring the image to be processed in the current frame, the method further includes: sending a control command to the multi-wavelength light source to control the multi-wavelength light source to emit light of a target wavelength, wherein the image to be processed in the current frame is an image acquired during the process of the multi-wavelength light source emitting light of the target wavelength.

[0008] Optionally, different marker layers correspond to different displacement thresholds, and the displacement threshold of each marker layer is positively correlated with the depth of the marker layer in the flexible layer. The depth refers to the distance between the object contact surface and the flexible layer. Among the N marker layers, the depth of the i-th marker layer is less than the depth of the (i+1)-th marker layer, where i is a positive integer and 1≤i≤N. If the current frame is the initial frame, the target marker layer of the current frame is the first marker layer among the N marker layers. If the current frame is not the initial frame, and the maximum displacement value in the displacement field of the target marker layer in the previous frame is greater than or equal to the displacement threshold corresponding to the target marker layer in the previous frame, then the target marker layer of the current frame is the next marker layer of the target marker layer in the previous frame. If the current frame is not the initial frame, and the maximum displacement value in the displacement field of the target marker layer in the previous frame is less than the displacement threshold corresponding to the target marker layer in the previous frame, then the target marker layer of the current frame is the target marker layer of the previous frame.

[0009] Optionally, different marker layers correspond to different displacement thresholds, and the displacement threshold corresponding to each marker layer is positively correlated with the depth of the marker layer in the flexible layer, where the depth refers to the distance between the object contact surface and the flexible layer; processing the image to be processed in the current frame to determine at least one displacement field of the target marker layer in the current frame includes: processing the image to be processed in the current frame to determine the displacement field of each marker layer in the current frame; wherein, if the current frame is the initial frame, the target marker layer of the current frame is the first marker layer among the N marker layers; if the current frame is not the initial frame, and the displacement field of the j-th marker layer in the current frame is greater than the displacement field of the j-th marker layer in the previous frame, and the maximum displacement value in the displacement field of the j-th marker layer in the current frame is less than the displacement threshold corresponding to the j-th marker layer, then the target marker layer of the current frame is the j-th marker layer, where j is a positive integer and 1≤j≤N.

[0010] Optionally, the number of target marker layers indicated by the hierarchical information to be analyzed in the current frame is multiple. Determining the external force on the flexible layer in the current frame based on the displacement field of at least one target marker layer in the current frame includes: splicing the displacement fields of multiple target marker layers to obtain a spliced ​​displacement field; and determining the external force on the flexible layer in the current frame based on the spliced ​​displacement field.

[0011] Secondly, embodiments of this application provide a tactile sensing device for a visual-tactile sensor. The visual-tactile sensor has a flexible layer and a camera. N marker layers are disposed within the flexible layer, and multiple markers are distributed within each marker layer, where N is a positive integer greater than 1. The device includes: an acquisition module for acquiring a current frame image to be processed, wherein the current frame image to be processed is obtained by the camera capturing images of the flexible layer; an attention determination module for acquiring the layer information to be analyzed in the current frame, wherein the layer information to be analyzed in the current frame is used to indicate at least one target marker layer among the N marker layers; a displacement calculation module for processing the current frame image to be processed to determine at least the displacement field of at least one target marker layer in the current frame; and a force calculation module for determining the external force acting on the flexible layer in the current frame based on the displacement field of at least one target marker layer in the current frame.

[0012] Thirdly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when run by a processor, performs the steps of the tactile perception method of the visual-tactile sensor provided in the first aspect.

[0013] Fourthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the tactile perception method of the visual-tactile sensor provided in the first aspect.

[0014] Fifthly, embodiments of this application also provide a computing device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the steps of the tactile perception method of the visual-tactile sensor provided in the first aspect when running the computer program.

[0015] Compared with the prior art, the technical solution of this application embodiment has the following beneficial effects:

[0016] In the scheme of this application embodiment, the image to be processed in the current frame is obtained, which is obtained by the camera capturing the image of the flexible layer; the layer information to be analyzed in the current frame is obtained, which is used to indicate at least one target marker layer among the N marker layers; the image to be processed in the current frame is processed to determine at least the displacement field of at least one target marker layer in the current frame; and the external force on the flexible layer in the current frame is determined based on the displacement field of at least one target marker layer in the current frame.

[0017] In the above scheme, the displacement field of the target marker layer is decoupled from the image to be processed based on the hierarchical information to be analyzed, and the external force on the flexible layer is calculated based on the displacement field of the target marker layer. This scheme utilizes marker layers of different depths to sense external forces of different degrees. For example, shallow marker layers are used to accurately sense shear forces or small normal forces, while deeper marker layers are used to accurately sense larger shear forces or normal forces. This scheme can accurately sense external forces of different degrees or natures, improving the sensing range of the visual-tactile sensor and achieving wide dynamic range sensing.

[0018] Furthermore, considering that the camera captures the flexible layer from below, the shallower the marker layer, the greater the probability of occlusion, and therefore the smaller the image size of the marker in the image; conversely, the greater the depth, the lower the probability of occlusion, and the larger the image size of the marker. Therefore, when determining whether a pixel is noise, the size of the neighborhood used for noise detection is set based on the depth of the marker layer. This approach helps to accurately determine whether a pixel is noise, thereby further improving the accuracy of displacement field decoupling between different marker layers.

[0019] Furthermore, in the scheme of this application embodiment, the displacement field of the target marker layer of the previous frame is used as a prediction variable. When the variable reaches the corresponding displacement threshold, the target marker layer is switched to shift the center of gravity of perception (i.e., attention) to the next target marker layer after the target marker layer of the previous frame. This switching strategy can ensure that tactile perception is always performed with an appropriate marker layer during the force application process, better simulating the layered feeling of biological skin "first touching the surface, then sensing the pressure", and realizing refined and phased tactile perception of the process of applying external force (such as from light touch to heavy pressure). Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of a visual-tactile sensor according to an embodiment of this application;

[0021] Figure 2 This is a flowchart illustrating the first tactile sensing method in the embodiments of this application;

[0022] Figure 3 This is a flowchart illustrating the second tactile sensing method in the embodiments of this application;

[0023] Figure 4 This is a flowchart illustrating the third tactile perception method in the embodiments of this application;

[0024] Figure 5 This is a flowchart illustrating the fourth tactile perception method in the embodiments of this application;

[0025] Figure 6 This is a flowchart illustrating the fifth tactile sensing method in the embodiments of this application;

[0026] Figure 7 This is a schematic diagram of the structure of a tactile sensing device according to an embodiment of this application. Detailed Implementation

[0027] As described in the background section, a visual-tactile sensor is a device that captures visual information using visual means and calculates tactile information from the visual information. In related technologies, visual-tactile sensors typically use a flexible layer as the force-bearing object, and embed markers on or inside the flexible layer. When an external force is applied to the surface of the flexible layer, a camera tracks the displacement of these markers, and the external force is perceived and estimated based on the tracked displacement.

[0028] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a visual-tactile sensor in an embodiment of this application.

[0029] like Figure 1 As shown, the visual-tactile sensor 100 may include: a flexible layer 10, a camera 30, a light source 40, and a housing 60.

[0030] The housing 60 provides mechanical support for the various components inside the visual-tactile sensor 100, thereby providing a stable working platform for the visual-tactile sensor 100. In specific implementations, the material of the housing 60 can be engineering plastics, metals, etc., but is not limited to these, and this document does not limit it.

[0031] Furthermore, the outer casing 60 has an opening, and the flexible layer 10 is disposed at the opening of the outer casing 60, so that the outer surface of the flexible layer 10 is directly exposed to the external environment, so that the flexible layer 10 can come into contact with an object. For example, the flexible layer 10 can be fixed to the opening of the outer casing 60 by means of bolts or adhesive, but is not limited thereto.

[0032] In specific implementations, the flexible layer 10 can be transparent and elastic. That is, the flexible layer 10 can be a transparent elastomer. In specific implementations, the flexible layer 10 can be made of a material with high light transmittance and high elasticity, such as transparent silicone, polyurethane elastomer, transparent hydrogel, special transparent rubber, or other optically transparent polymer materials.

[0033] It should be noted that the shape of the flexible layer 10 is not limited in this paper. The flexible layer 10 can be pointed, planar, curved (such as spherical), tubular, balloon-shaped, irregular curved, etc., to adapt to different robot end effectors or meet the needs of application scenarios.

[0034] Furthermore, markers are provided within the flexible layer 10. A marker can refer to an object used for optical tracking that can undergo identifiable displacement and / or deformation when the flexible layer 10 is subjected to force. "Marker" can be a pattern, structure, component, etc., on the flexible layer 10, and this application is not limited thereto.

[0035] In the embodiments of this application, N marker layers are provided within the flexible layer 10, and multiple markers 20 are distributed within each marker layer, where N is a positive integer greater than 1. Specifically, N marker layers are provided along the depth direction of the flexible layer 10, wherein marker layers in different marker layers have different identifiable features. In other words, markers 20 with the same identifiable features have the same depth within the flexible layer 10, and multiple markers 20 with the same depth can be considered as one marker layer. Specifically, the depth of the i-th marker layer among the N marker layers is less than the depth of the (i+1)-th marker layer. i is a positive integer, 1≤i≤N. Figure 1 Different identifiable markers are distinguished by different filling patterns.

[0036] The depth direction of the flexible layer 10 is perpendicular to the surface on which the flexible layer 10 is located. For example... Figure 1 As shown, Figure 1 In the diagram, the U direction is the upward direction along the depth direction, and the D direction is the downward direction along the depth direction.

[0037] It should be noted that, in this article, "depth" refers to the distance between the marker 20 and the outer surface of the flexible layer 10. The outer surface of the flexible layer 10 refers to the surface furthest from the camera 30. For example, a identifiable feature could be color. That is, the marker 20 in different marker layers may have different colors. For instance, with N=3, the marker 20 in the first marker layer is red, the marker 20 in the second marker layer is green, and the marker 20 in the third marker layer is red.

[0038] As another example, the identifiable feature can be shape. That is, the markers 20 in different marker layers have different shapes.

[0039] As another example, the material of marker 20 can be a fluorescent material, and the identifiable feature can refer to the excitation wavelength of the fluorescent material. That is, the excitation wavelength of marker 20 in different marker layers is different. In other words, marker 20 in different marker layers responds to different wavelengths. In this case, light source 40 can be a multi-wavelength light source, capable of emitting light of different wavelengths.

[0040] As another example, the identifiable feature could refer to the spectral band of the light reflected by the marker. Taking N=3 as an example, the three marker layers reflect only specific narrowband infrared light at 850 nm, 940 nm, and 1550 nm, respectively. In this case, the camera could be a multispectral or hyperspectral camera.

[0041] In one possible implementation, the markers 20 in the N marker layers can be arranged in a staggered array. Specifically, the camera 30 captures images from below towards the flexible layer 10. To avoid lower marker layers obscuring upper marker layers and to ensure that all markers 20 in each marker layer can be clearly observed, the positions of the markers 20 in each marker layer within the observation plane can be designed to be staggered. This ensures that the image captured by the camera 30 can contain complete images of the markers 20 in each marker layer. The observation plane can be a plane perpendicular to the depth direction of the flexible layer 10. The projections of the markers 20 in the N marker layers onto the observation plane do not overlap.

[0042] In another possible implementation, the markers 20 in each marker layer can be randomly distributed. For example, the markers 20 in each marker layer can be randomly and densely distributed in the flexible layer 10. It should be noted that the random distribution of the markers 20 here refers to the random position of the markers 20 on the observation plane, not that the depth of the markers 20 is random.

[0043] Furthermore, the camera 30 can be disposed at the bottom of the housing 60, and the camera 30 is used to acquire images of the flexible layer 10. In a specific implementation, the camera may include an image sensor and a lens, wherein the image sensor may be a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) sensor, but is not limited to these.

[0044] Furthermore, such as Figure 1 As shown, the visual-tactile sensor 100 may also include a light source 40, which can be used to create a uniform optical environment for the visual sensor 100 to improve the image quality acquired by the camera 30. Specifically, the light source 40 can provide illumination for image acquisition by the camera 30. For example, the light source 40 can be used to illuminate the marker 20 in the flexible layer 10.

[0045] In a specific implementation, the light source 40 can be located at the bottom of the housing 60. Alternatively, the light source 40 can also be located on the side wall of the housing 60.

[0046] It should be noted that this article does not limit the number of light sources 40. For example, there can be multiple light sources 40, and multiple light sources 40 can be arranged around the camera 30.

[0047] Furthermore, the visual-tactile sensor 100 may also include a data interface ( Figure 1 (Unspecified), the visual-tactile sensor 100 can be connected to an external processor 70 via a data interface. The data interface can be a wired communication interface or a wireless communication interface; this document does not limit the type of data interface.

[0048] In one embodiment of this application, the camera 30 can transmit the captured image to a processor 70 outside the visual-touch sensor 100 for processing.

[0049] In another embodiment of this application, the visual-touch sensor 100 may include a processor (not shown), which can process or analyze the image captured by the camera 30 to obtain information about the external force. Alternatively, the processor in the visual-touch sensor 100 can partially process the image captured by the camera 30 to obtain intermediate results, and then send the intermediate results to a processor 70 outside the visual-touch sensor 100, whereby the processor outside the visual-touch sensor 100 obtains information about the external force based on the intermediate results.

[0050] It should be noted that the processor mentioned in this article can refer to a device or equipment with data processing capabilities. More specifically, a processor can be a microcontroller unit (MCU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a personal computer (PC), etc.

[0051] based on Figure 1 The visual-tactile sensor with the structure shown captures images of markers 20 at multiple marker layers with different depths in the image acquired by the camera 30. These markers 20 at different depths are projected indiscriminately onto the same two-dimensional image plane, resulting in the loss of depth information and the mixing of displacement fields from markers at different depths into a single entity. This information confusion leads to problems such as low accuracy of the perceived mechanical information and a limited sensing range.

[0052] The solution in this application takes into account that a shallower marker layer is more sensitive to shear force and a smaller normal pressure, and can more accurately sense shear force and a smaller normal pressure. When the normal pressure applied to the flexible layer is large, the displacement field of the shallower marker layer can no longer be accurately characterized. At this time, it will be more accurate to determine the external force based on the displacement field of the deeper marker layer.

[0053] In view of this, embodiments of this application provide a tactile perception method for a visual-tactile sensor, comprising: acquiring a current frame image to be processed, wherein the current frame image to be processed is obtained by the camera capturing an image of the flexible layer; acquiring layer information to be analyzed in the current frame, wherein the layer information to be analyzed in the current frame is used to indicate at least one target marker layer among the N marker layers; processing the current frame image to be processed to determine at least the displacement field of at least one target marker layer in the current frame; and determining the external force acting on the flexible layer in the current frame based on the displacement field of at least one target marker layer in the current frame.

[0054] In the above scheme, the displacement field of the target marker layer is decoupled from the image to be processed based on the hierarchical information to be analyzed, and the external force on the flexible layer is calculated based on the displacement field of the target marker layer. This scheme utilizes marker layers of different depths to sense external forces of different degrees. For example, shallow marker layers are used to accurately sense shear force or small normal pressure, while deeper marker layers are used to accurately sense larger normal pressure. This scheme can accurately sense external forces of different degrees or natures, improving the sensing range of the visual-tactile sensor and achieving wide dynamic range sensing.

[0055] It should be noted that the tactile sensing method provided in this application embodiment can be applied to a processor. Specifically, the tactile sensing method provided herein can be executed by a single processor, or by multiple processors working together. For example, different processors can execute different steps of the tactile sensing method described herein, or different processors can execute different sub-steps within the same step of the tactile sensing method described herein. The multiple processors may include processors inside the visual tactile sensor and processors outside the sensor, but are not limited thereto.

[0056] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first tactile sensing method in the embodiments of this application. Figure 2 The methods shown may include S21, S22, S23 and S24.

[0057] S21, Obtain the image to be processed in the current frame.

[0058] In this process, the image to be processed in the current frame is obtained by the camera in the visual-tactile sensor capturing images of the flexible layer. In practice, the camera can capture images at a preset frame rate to obtain the image to be processed, and then transmit the image to the processor for processing and analysis.

[0059] The image to be processed contains images of markers in N marker layers. Furthermore, in the embodiments of this application, the image to be processed also carries N layer information, which can be used to indicate the N marker layers from the image to be processed.

[0060] Specifically, since markers in different marker layers have different identifiable features, the hierarchical information can be characterized by the identifiable features of the markers. The hierarchical information enables the processor to determine the marker layer to which a marker belongs from the image captured by the camera.

[0061] In the scheme of this application embodiment, by setting different identifiable features for the markers of the marker layer, when the camera 30 captures images of the flexible layer, the obtained images carry the hierarchical information of the marker layer. The hierarchical information makes each marker layer independent of each other in the image to be processed and not confused.

[0062] S22, obtain the layer information to be analyzed in the current frame. The layer information to be analyzed is used to indicate at least one target marker layer among N marker layers.

[0063] Specifically, in S32, the layer information to be analyzed in the current frame is determined from the N layer information carried in the image to be processed in the current frame. The layer information to be analyzed is used to indicate one or more target marker layers. In other words, the image to be processed in the current frame contains images of markers in N marker layers. In S22, the target marker layer of the current frame is determined from the N marker layers so that the external force on the flexible layer can be calculated subsequently based on the displacement field of the markers in the target marker layer. The information of other marker layers in the N marker layers other than the target marker layer is ignored, that is, other marker layers other than the target marker layer are not used to calculate the external force on the flexible layer.

[0064] In one possible implementation, different marker layers correspond to different displacement thresholds, and the displacement threshold of each marker layer is positively correlated with its depth in the flexible layer. That is, the greater the depth of the marker layer in the flexible layer, the greater the displacement threshold corresponding to the marker layer.

[0065] For example, if the current frame is the initial frame, the target marker layer of the current frame can be determined as the first marker layer among N marker layers. If the current frame is not the initial frame, and the maximum displacement value in the displacement field of the target marker layer of the previous frame is greater than or equal to the displacement threshold corresponding to the target marker layer of the previous frame, then the target marker layer of the current frame is the next marker layer after the target marker layer of the previous frame; if the current frame is not the initial frame, and the maximum displacement value in the displacement field of the target marker layer of the previous frame is less than the displacement threshold corresponding to the target marker layer of the previous frame, then the target marker layer of the current frame is the target marker layer of the previous frame. The displacement field of the target marker layer of the previous frame can be obtained based on the image to be processed in the previous frame.

[0066] For example, assuming the target marker layer of the previous frame is the first marker layer, if the maximum value in the displacement field of the first marker layer is greater than or equal to the displacement threshold corresponding to the first marker layer, then the second marker layer can be determined as the target marker layer of the current frame; otherwise, the target marker layer of the current frame is still the first marker layer.

[0067] Specifically, if the maximum displacement value in the displacement field of the target marker layer in the previous frame is greater than or equal to the displacement threshold corresponding to the target marker layer in the previous frame, it indicates that the external force has exceeded the optimal perception range of the target marker layer in the previous frame. In this case, attention is shifted to the next marker layer to continue perception.

[0068] Therefore, the layer information to be analyzed in the current frame can be determined through S22, that is, the target marker layer of the current frame can be determined.

[0069] S23, process the image to be processed in the current frame to determine the displacement field of the target marker layer in the current frame.

[0070] Specifically, the image of the target marker layer can be segmented from the image to be processed in the current frame based on the identifiable features of the markers in each target marker layer; then, the displacement field of the target marker layer in the current frame can be determined based on the image of the target marker layer in the current frame.

[0071] In practice, image segmentation algorithms can be used to segment the image of the target marker layer from the image to be processed in the current frame based on the identifiable features of the markers in each target marker layer.

[0072] In one embodiment of this application, the identifiable feature is color. In S23, the standard color to which the pixel belongs can be determined based on the color representation value of each pixel in the image to be processed in the current frame and the standard color representation information of each marker layer. Further, the image to be processed is restored according to the reference representation value of the standard color to which each pixel belongs, thereby obtaining a restored image. Further, the image of the target marker layer is segmented from the restored image. The color representation value can be a value used to represent color, for example, it can be an RGB value, but is not limited to this.

[0073] The standard color representation information for each marker layer can be pre-configured or pre-calibrated. Specifically, the standard color representation information for each marker layer can be a color profile obtained statistically from a reference image of that marker layer. For example, taking the i-th marker layer as an example, only the i-th marker layer is set in the visual-tactile sensor. An image of the i-th marker layer is acquired under the standard lighting environment of the visual-tactile sensor to obtain a reference image for the i-th marker layer. Further, the standard color representation information for the i-th marker layer is determined by statistically analyzing the color representation values ​​of each pixel in the reference image of the i-th marker layer. Taking the color of the marker in the i-th marker layer as red as an example, its standard color representation information is a Gaussian distribution centered at (220, 30, 15) with a specific variance.

[0074] Furthermore, for each pixel in the image to be processed, the confidence level of the pixel belonging to each marker layer can be determined based on the pixel's color representation value and the standard color representation information of each marker layer. The color of the marker in the marker layer with the highest confidence level is then used as the standard color of the pixel. In practice, a classifier (such as a Mahalanobis distance-based classifier, a Bayesian classifier, etc.) can be used to calculate the confidence level.

[0075] The above scheme pre-configures standard color representation information for various colors under standard lighting conditions of the visual-tactile sensor. Then, it calculates the color representation value of each pixel and the confidence level of the standard color representation information for each color, determining the standard color to which the pixel belongs based on the confidence level. This scheme can accurately determine the standard color of a pixel even when color mixing occurs in the image to be processed, thereby improving the accuracy and reliability of decoupling markers from different layers in the image to be processed.

[0076] In one possible implementation, noise in the image to be processed can be removed before restoration. For example, for each pixel, if the proportion of pixels with the same standard color in its neighborhood is less than a preset threshold, the pixel can be identified as noise and removed. For instance, the size of the neighborhood can be positively correlated with the depth of the marker corresponding to the pixel's standard color in the flexible layer. That is, the greater the depth of the marker corresponding to the pixel's standard color in the flexible layer, the larger the size of the neighborhood. This approach is adopted because the camera captures the flexible layer from below; the shallower the marker layer, the greater the probability of occlusion, and therefore the smaller the size of the marker's image in the image; conversely, the greater the depth, the lower the probability of occlusion, and the larger the size of the marker's image in the image. Therefore, when determining whether a pixel is noise, the size of the neighborhood needs to be set according to the depth of the marker layer. This approach helps to accurately determine whether a pixel is noise.

[0077] In another possible implementation, the identifiable feature is shape. In this case, computer vision techniques can also be used to identify the images of individual markers in the target marker layer of the image to be processed.

[0078] Furthermore, for each target marker layer image, the displacement field of that target marker layer is determined. Specifically, reference images corresponding to each marker layer can be pre-configured when the flexible layer is not under stress. Further, for each target marker layer, based on the image obtained in S23 and the pre-configured reference image, the displacement field of the target marker layer can be determined.

[0079] For example, optical flow can be used to determine the displacement field of the target marker layer. Using optical flow can yield an accurate and complete displacement field, especially when the markers are randomly and densely distributed within the flexible layer; in this case, optical flow can effectively improve the integrity and accuracy of the displacement field.

[0080] S24, determine the external force on the flexible layer of the current frame based on the displacement field of the target marker layer of the current frame.

[0081] For example, the displacement field of the target marker layer in the current frame can be input into a pre-obtained force calculation model to obtain the characterization parameters of the external forces output by the force calculation model. These characterization parameters may include at least one of the following: magnitude, direction, and type (e.g., normal force, shear force, torque, etc.).

[0082] The force calculation model can be a model establishing the relationship between elastic body deformation and applied external forces through finite element simulation, and then the perceived parameters of the external forces obtained in S24 through inverse finite element analysis. Alternatively, the force calculation model can be a machine learning model, such as a convolutional network model, Transformer, self-supervised learning model, etc., but is not limited to these. Alternatively, the force calculation model can be a model establishing the relationship between elastic body deformation and applied external forces by combining elasticity theory and experimental data.

[0083] In one variation, there can be multiple target marker layers. In this case, the displacement fields of multiple target marker layers can be fused to obtain a fused displacement field, and then the external force on the flexible layer can be determined based on the fused displacement field.

[0084] From the above, in Figure 2 The illustrated scheme acquires the hierarchical information of the current frame to be analyzed, determines the target marker layer based on this information, decouples the displacement field of the target marker from the image to be processed in the current frame, and calculates the characterization parameters of the external force based on the displacement field of the target marker. Furthermore, the scheme defines a corresponding displacement threshold for each target marker layer, and controls the switching of the target marker layer by comparing the displacement threshold with the displacement field of the target marker layer in the previous frame. This switching strategy ensures that a suitable marker layer is always used for tactile perception during the force application process, better simulating the layered perception of biological skin—"first touching the surface, then sensing pressure"—and enabling refined and phased analysis of the external force application process (e.g., from light touch to heavy pressure). (Refer to...) Figure 3 , Figure 3 This is a flowchart illustrating the second tactile sensing method in an embodiment of this application. Figure 3 In the scheme shown, the light source in the visual-tactile sensor is a multi-wavelength light source, the material of the marker can be a fluorescent material, the identifiable feature can refer to the excitation wavelength of the fluorescent material, and the markers in different marker layers respond to different wavelengths. Figure 3 The methods shown may include S31, S32, S33 and S34.

[0085] S31, Obtain the layer information to be analyzed in the current frame.

[0086] In one possible implementation, if the current frame is the initial frame, the target marker layer of the current frame can be determined as the first marker layer among N marker layers. If the current frame is not the initial frame, and the maximum displacement value in the displacement field of the target marker layer of the previous frame is greater than or equal to the displacement threshold corresponding to the target marker layer of the previous frame, then the target marker layer of the current frame is the next marker layer after the target marker layer of the previous frame; if the current frame is not the initial frame, and the maximum displacement value in the displacement field of the target marker layer of the previous frame is less than the displacement threshold corresponding to the target marker layer of the previous frame, then the target marker layer of the current frame is the target marker layer of the previous frame. The displacement field of the target marker layer of the previous frame can be obtained based on the image to be processed in the previous frame.

[0087] In another possible implementation, the hierarchical information of the current frame to be analyzed can be obtained from an external source.

[0088] For more information about S31, please refer to the description of S22 above.

[0089] S32, based on the hierarchical information to be analyzed in the current frame, sends control commands to the multi-wavelength light source to control the multi-wavelength light source to emit light of the target wavelength.

[0090] S33, obtain the image to be processed in the current frame.

[0091] Specifically, the image to be processed in the current frame is an image acquired while light of the target wavelength is emitted by a multi-wavelength light source. More specifically, the markers in the target marker layer are able to respond to light of the target wavelength, causing the markers in the target marker layer to emit light while the markers in other target marker layers do not emit light. Thus, the image to be processed in the current frame acquired by the camera only contains images of the markers in the target marker layer.

[0092] S34, process the image to be processed in the current frame to determine the displacement field of the target marker layer in the current frame.

[0093] In S34, since the image to be processed in the current frame only contains the image of the target marker layer, it is not necessary to segment the image of the target marker layer through image processing algorithms. The displacement field of the target marker layer can be directly determined based on the image to be processed and the reference image of the target marker layer.

[0094] S35, determine the external force on the flexible layer of the current frame based on the displacement field of the target marker layer of the current frame.

[0095] For details regarding the specific content shown in S35, please refer to the relevant description of S24 above, which will not be repeated here.

[0096] From the above, it is different from Figure 2 The proposed scheme Figure 3 The illustrated scheme controls multiple wavelength light sources to emit light of the target wavelength in a time-division manner, allowing the camera to sequentially acquire images containing only the target marker layer in the image to be processed. This approach eliminates the need for image processing algorithms to segment the image of the target marker layer, completely avoiding crosstalk between different marker layers within the same image and ensuring the accuracy and reliability of information decoupling.

[0097] Reference Figure 4 , Figure 4 This is a flowchart illustrating the third tactile sensing method in the embodiments of this application. It should be noted that... Figure 4 In the scheme shown, each marker layer in the flexible layer can be regarded as a target marker layer. Figure 4 The methods shown may include: S41, S42 and S43.

[0098] S41, Obtain the image to be processed in the current frame.

[0099] For details regarding S41, please refer to the description of S21 above.

[0100] S42, process the image to be processed in the current frame to determine the displacement field of each marker layer in the current frame.

[0101] In practice, the image of each marker layer can be segmented from the image to be processed in the current frame based on the identifiable features of the markers in each marker layer; then, the displacement field of the marker layer in the current frame can be determined based on the image of each marker layer in the current frame.

[0102] thus, Figure 4 In the scheme shown, the displacement fields of each marker layer can be decoupled from the image to be processed in the current frame, so that the displacement fields of different marker layers are independent of each other.

[0103] For more information on S42, please refer to the description of S23 above, which will not be repeated here.

[0104] S43, determine the external force on the flexible layer of the current frame based on the displacement field of each marker layer in the current frame.

[0105] Specifically, the displacement fields of the N marker layers can be fused first to obtain a fused displacement field. Furthermore, the fused displacement field can be input into a pre-prepared force calculation model to obtain the external force characterization parameters output by the force calculation model.

[0106] In one possible implementation, the displacement fields of each marker layer can be stitched together to obtain a stitched displacement field, which is the fused displacement field. For example, the displacement field of each marker layer is a two-dimensional vector, which can be stitched together in a third dimension, resulting in a three-dimensional feature vector.

[0107] In another possible implementation, the displacement fields of each marker layer can be weighted to obtain a weighted displacement field, which is the fused displacement field. For example, the weights of the displacement fields of each marker layer can be determined based on the hierarchical information of each marker layer.

[0108] From the above, Figure 4 The illustrated scheme separates the images of each marker layer in the image to be processed to obtain the images of each marker layer, and then determines the displacement field of each marker layer based on its image. This scheme decouples the displacement fields of each marker layer, and then re-fuses them to obtain the fused displacement field. This process enables the fused displacement field to carry the hierarchical information of each displacement field, which is beneficial for achieving wide dynamic range sensing of the visual-tactile sensor.

[0109] Reference Figure 5 , Figure 5 This is a flowchart illustrating the fourth tactile perception method in the embodiments of this application. Figure 5 The methods shown may include S51, S52, S53 and S54.

[0110] S51, Obtain the image to be processed in the current frame.

[0111] For details about S51, please refer to the above description of S41.

[0112] S52, process the image to be processed in the current frame to determine the displacement field of each marker layer in the current frame.

[0113] For details regarding S52, please refer to the descriptions of S42 and S23 above.

[0114] S53, obtain the layer information to be analyzed in the current frame.

[0115] Specifically, if the current frame is the initial frame, then the target marker layer of the current frame is the first marker layer among the N marker layers.

[0116] If the current frame is not the initial frame, and the displacement field of the j-th marker layer in the current frame is greater than the displacement field of the j-th marker layer in the previous frame, and the maximum displacement value in the displacement field of the j-th marker layer in the current frame is less than the displacement threshold corresponding to the j-th marker layer, then the target marker layer in the current frame is the j-th marker layer, where j is a positive integer and 1≤j≤N.

[0117] In other words, if the current frame is not the initial frame, it is possible to determine whether each marker layer is a target marker layer based on the displacement fields of each marker layer in the current frame. Taking the j-th marker layer as an example, if the displacement field of the j-th marker layer in the current frame is greater than the displacement field in the previous frame, and the maximum displacement value in the displacement field of the j-th marker layer is less than the displacement threshold corresponding to the j-th marker layer, then the j-th marker layer can be determined as the target marker layer.

[0118] For more information on S53, please refer to the relevant description of S22 above.

[0119] S54, determine the external force on the flexible layer of the current frame based on the displacement field of the target marker layer of the current frame.

[0120] For details about S54, please refer to the description of S24 above.

[0121] From the above, Figure 5 In the illustrated scheme, the displacement fields of all marker layers in the current frame are first determined. The target marker layer is then determined based on the displacement fields of all marker layers. Furthermore, the external force acting on the flexible layer is determined based on the displacement field of the target marker layer. That is, it differs from... Figure 2 The proposed solution Figure 5 The scheme combines the displacement field of the current frame to control the switching of the target marker layer.

[0122] Reference Figure 6 , Figure 6 This is a flowchart illustrating the fifth tactile sensing method in the embodiments of this application. Figure 6 In the scheme shown, the light source in the visual-tactile sensor is a multi-wavelength light source, the material of the marker can be a fluorescent material, the identifiable feature can refer to the excitation wavelength of the fluorescent material, and the markers in different marker layers respond to different wavelengths. Figure 6 The methods shown may include S61, S62, and S63.

[0123] S61, obtain the current image group to be processed, wherein the i-th image to be processed in the current image group contains only the image of the i-th marker layer.

[0124] S62, based on the current group of images to be processed, obtain the displacement fields of the current N marker layers.

[0125] In step S61, N control commands can be sent sequentially to the multi-wavelength light source. The i-th control command among these N commands controls the multi-wavelength light source to emit the wavelength corresponding to the i-th marker layer. When the multi-wavelength light source emits the wavelength corresponding to the i-th marker layer, the image to be processed captured by the camera only contains the image of the i-th marker layer. Thus, N images to be processed can be obtained, and the i-th image among these N images contains only the image of the i-th marker layer.

[0126] For example, firstly, a multi-wavelength light source emits the wavelength corresponding to the first marker layer, exciting only the markers in the first marker layer to emit light, and simultaneously capturing the first image to be processed. Further, the multi-wavelength light source emits the wavelength corresponding to the second marker layer, exciting only the markers in the second marker layer to emit light, and simultaneously capturing the second image to be processed. This process continues in a similar manner, completing the sequential excitation and image acquisition of all marker layers in a very short time, thereby obtaining a group of images representing different depths that are temporally separated.

[0127] Furthermore, based on the image of the i-th marker layer in the N images to be processed, the displacement field of the i-th marker layer can be determined, thereby allowing the displacement field of each marker layer to be determined separately.

[0128] S63, determine the external force currently acting on the flexible layer based on the displacement field of each marker layer.

[0129] For details about S63, please refer to the above description of S43.

[0130] From the above, Figure 6 In this scheme, multi-wavelength light sources emit light of different wavelengths at different times, allowing the camera to sequentially acquire images of individual marker layers, thereby obtaining the displacement field of each marker layer. This approach eliminates the need for software algorithms to decouple the displacement fields of each marker layer, completely avoiding crosstalk between different marker layers in the same image. It enables more thorough separation of the displacement fields of each marker layer, ensuring the accuracy and reliability of information decoupling.

[0131] As described above, the solution of this application decouples the displacement information of multiple marker layers, and determines the external force on the flexible layer based on the displacement field of one or more marker layers obtained by decoupling, thus achieving richer and more accurate tactile perception. For example, it can distinguish the fine texture of the object surface (reflected by the displacement field of shallow markers) and the overall pressure intensity (reflected by the displacement field of deep markers). This achieves a breakthrough in tactile perception in terms of dimension, accuracy, and breadth.

[0132] It should be noted that the tactile sensing method provided in this paper can be applied to androids or industrial applications, enabling the dexterous hands of androids or the grippers of industrial robots to perform precise object manipulation and adaptive grasping control. For example, the solution provided in this paper can be applied to flexible wearable sensing, medical rehabilitation, and medical palpation, but is not limited to these applications.

[0133] It should be noted that the various embodiments, possible implementations, possible methods of implementation, various examples, and solutions in various situations provided in this article can be used individually or in combination to achieve different technical effects.

[0134] It is understandable that, in specific implementation, the above method can be implemented using a software program that runs in the processor integrated inside the chip or chip module; or, the method can be implemented using hardware or a combination of hardware and software, such as using a dedicated chip or chip module, or using a dedicated chip or chip module combined with a software program.

[0135] Reference Figure 7 , Figure 7 This application discloses a tactile sensing device for a visual-tactile sensor. The visual-tactile sensor has a flexible layer and a camera. N marker layers are disposed within the flexible layer, and each marker layer contains multiple markers, where N is a positive integer greater than 1. The device includes:

[0136] The acquisition module 71 is used to acquire the image to be processed in the current frame, wherein the image to be processed in the current frame is obtained by the camera capturing the image of the flexible layer;

[0137] Attention determination module 72 is used to obtain the hierarchical information to be analyzed in the current frame, wherein the hierarchical information to be analyzed in the current frame is used to indicate at least one target marker layer among the N marker layers;

[0138] Displacement calculation 73 is used to process the image to be processed in the current frame to determine at least the displacement field of at least one target marker layer in the current frame.

[0139] Force calculation module 74 is used to determine the external force on the flexible layer in the current frame based on the displacement field of at least one target marker layer in the current frame.

[0140] For more information on the working principle, working method, and beneficial effects of the tactile sensing device in the embodiments of this application, please refer to the relevant description of the tactile sensing method above, which will not be repeated here.

[0141] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is run by a computer, the above-described method is executed. The storage medium may include read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. The storage medium may also include non-volatile memory or non-transitory memory, etc.

[0142] This application also provides a computing device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it performs the steps of the method described above. In specific implementations, this computing device can be integrated into a hygienist robot.

[0143] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it performs the steps of the above-described tactile sensing method. The storage medium may include ROM, RAM, a magnetic disk, or an optical disk, etc. The storage medium may also include non-volatile memory or non-transitory memory, etc.

[0144] It should be understood that in the embodiments of this application, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0145] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0146] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or in a combination of hardware and software functional units. For example, for various devices or products applied to or integrated into a chip, each module / unit can be implemented using hardware such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware such as circuits; for various devices or products applied to or integrated into a chip module, each module / unit can be implemented using hardware such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0149] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0150] In this application's embodiments, "multiple" refers to two or more. The descriptions of "first," "second," etc., appearing in this application's embodiments are merely illustrative and for distinguishing the described objects; they have no order and do not indicate a specific limitation on the number of devices in this application's embodiments, nor do they constitute any limitation on the embodiments of this application.

[0151] While this application discloses the above information, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application shall be determined by the scope defined in the claims.

Claims

1. A tactile perception method using a visual-tactile sensor, characterized in that, The visual-tactile sensor has a flexible layer and a camera. The flexible layer has N marker layers, and each marker layer has multiple markers distributed in it. N is a positive integer greater than 1. The method includes: The image to be processed in the current frame is obtained by the camera capturing images of the flexible layer. Obtain the layer information to be analyzed in the current frame, wherein the layer information to be analyzed in the current frame is used to indicate at least one target marker layer among the N marker layers; The image to be processed in the current frame is processed to determine the displacement field of at least one target marker layer in the current frame; The external force exerted on the flexible layer in the current frame is determined based on the displacement field of at least one target marker layer in the current frame.

2. The tactile perception method according to claim 1, characterized in that, The markers in different marker layers have different identifiable features; Processing the image to be processed in the current frame to determine the displacement field of at least one target marker layer in the current frame includes: Based on the identifiable features of the markers in each target marker layer, the image of that target marker layer is segmented from the image to be processed in the current frame; Based on the image of each target marker layer in the current frame, determine the displacement field of that target marker layer in the current frame.

3. The tactile perception method according to claim 2, characterized in that, The identifiable feature is color. Before processing the image to be processed in the current frame to determine at least the displacement field of at least one target marker layer in the current frame, the method further includes: Based on the color representation value of each pixel in the image to be processed in the current frame and the standard color representation information of each marker layer, the standard color to which the pixel belongs is determined. For each pixel, if the proportion of pixels with the same standard color in the neighborhood of that pixel is less than a preset threshold, then that pixel is identified as noise and removed. The size of the neighborhood is positively correlated with the depth of the marker to which the standard color of the pixel belongs in the flexible layer.

4. The tactile perception method according to claim 1, characterized in that, The markers in different marker layers respond to different wavelengths, and the visual-tactile sensor also includes a multi-wavelength light source; Before acquiring the image to be processed in the current frame, the method further includes: A control command is sent to the multi-wavelength light source to control the multi-wavelength light source to emit light of the target wavelength. The image to be processed in the current frame is an image acquired during the process of the multi-wavelength light source emitting light of the target wavelength.

5. The tactile perception method according to claim 1, characterized in that, Different marker layers correspond to different displacement thresholds, and the displacement threshold of each marker layer is positively correlated with the depth of the marker layer in the flexible layer. The depth refers to the distance between the object contact surface and the flexible layer. Among the N marker layers, the depth of the i-th marker layer is less than the depth of the (i+1)-th marker layer, where i is a positive integer and 1≤i≤N. If the current frame is the initial frame, then the target marker layer of the current frame is the first marker layer among the N marker layers; If the current frame is not the initial frame, and the maximum displacement value in the displacement field of the target marker layer of the previous frame is greater than or equal to the displacement threshold corresponding to the target marker layer of the previous frame, then the target marker layer of the current frame is the next marker layer of the target marker layer of the previous frame. If the current frame is not the initial frame, and the maximum displacement value in the displacement field of the target marker layer in the previous frame is less than the displacement threshold corresponding to the target marker layer in the previous frame, then the target marker layer of the current frame is the target marker layer of the previous frame.

6. The tactile perception method according to claim 1, characterized in that, Different marker layers correspond to different displacement thresholds, and the displacement threshold of each marker layer is positively correlated with the depth of the marker layer in the flexible layer, where the depth refers to the distance between the object contact surface of the flexible layer and the object. Processing the image to be processed in the current frame to determine the displacement field of at least one target marker layer in the current frame includes: The image to be processed in the current frame is processed to determine the displacement field of each marker layer in the current frame; Wherein, if the current frame is the initial frame, then the target marker layer of the current frame is the first marker layer among the N marker layers; If the current frame is not the initial frame, and the displacement field of the j-th marker layer in the current frame is greater than the displacement field of the j-th marker layer in the previous frame, and the maximum displacement value in the displacement field of the j-th marker layer in the current frame is less than the displacement threshold corresponding to the j-th marker layer, then the target marker layer of the current frame is the j-th marker layer, where j is a positive integer and 1≤j≤N.

7. The tactile perception method according to claim 1, characterized in that, The number of target marker layers indicated by the hierarchical information to be analyzed in the current frame is multiple. Based on the displacement field of at least one target marker layer in the current frame, the external force on the flexible layer in the current frame is determined, including: The displacement fields of multiple target marker layers are spliced ​​together to obtain the spliced ​​displacement field; Based on the spliced ​​displacement field, the external force on the flexible layer in the current frame is determined.

8. A tactile sensing device for a visual-tactile sensor, characterized in that, The visual-tactile sensor has a flexible layer and a camera. The flexible layer has N marker layers, and each marker layer has multiple markers distributed in it. N is a positive integer greater than 1. The device includes: The acquisition module is used to acquire the image to be processed in the current frame, which is obtained by the camera capturing images of the flexible layer. An attention determination module is used to obtain the hierarchical information to be analyzed in the current frame, wherein the hierarchical information to be analyzed in the current frame is used to indicate at least one target marker layer among the N marker layers; Displacement calculation is used to process the image to be processed in the current frame to determine the displacement field of at least one target marker layer in the current frame. The force calculation module is used to determine the external force on the flexible layer in the current frame based on the displacement field of at least one target marker layer in the current frame.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the tactile perception method of the visual-tactile sensor as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the tactile perception method of the visual-tactile sensor as described in any one of claims 1 to 7.

11. A computing device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the tactile perception method of the visual-tactile sensor according to any one of claims 1 to 7.

Images