Tactile sensing system and method, electronic device, storage medium, and program product

Abstract

The embodiment of the invention provides a touch sensing system and method, electronic equipment, a storage medium and a program product. The touch sensing system comprises a contact assembly, a projection assembly, a collection assembly and a processing assembly, and is configured to be capable of being in contact with an external object and generating deformation; the projection assembly is configured to project an active light field with a time modulation characteristic to a first surface of the contact assembly, the active light field is projected on the first surface to form a plurality of pattern areas, the light intensity of adjacent pattern areas changes based on different time sequence codes, and the first surface is opposite to a second surface, making contact with an external object, of the contact assembly; the acquisition assembly comprises a plurality of acquisition units arranged in an array, and is configured to acquire a plurality of pattern areas and output a time stamp acquisition data sequence.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202610145134.5 (22) Application Date 2026.02.02 (71) Applicant Shenzhen Ruishi Zhixin Technology Co., Ltd. Address 518055 No. 3370 Liuxian Avenue, Fuguang Community, Taoyuan Street, Nanshan District, Shenzhen, Guangdong Province, Building 3, Chongwen Park, Nanshan Zhiyuan (72) Inventors Deng Jian, Chen Liming, Liu Chang, Chen Qiyue (74) Patent Agency Beijing Yingsai Jia Hua Intellectual Property Agency Co., Ltd. 11204 Patent Attorney Wang Dazuo, Wang Yanchun (51) Int.Cl. G06F 3 / 01 (2006.01) G06F 3 / 041 (2006.01) (54) Invention Title Tactile Sensing System and Method, Electronic Device, Storage Medium, Summary of the Program Product (57) Embodiments of this disclosure provide a tactile sensing system and method, an electronic device, a storage medium, and a program product. The tactile sensing system includes a contact component, a projection component, a acquisition component, and a processing component, configured to contact an external object and generate deformation; the projection component is configured to project an active light field with time-modulated characteristics onto a first surface of the contact component, the active light field forming multiple patterned regions on the first surface, the light intensity of adjacent patterned regions changing based on different time-series encodings, the first surface facing away from a second surface of the contact component that contacts the external object; the acquisition component includes multiple acquisition units arranged in an array and is configured to acquire multiple patterned regions and output a time-stamped acquisition data sequence. Claims 3 pages, Description 30 pages, Drawings 8 pages, CN 121614037 A 2026.03.06 CN 1 21 61 40 37 A 1. A tactile sensing system, characterized in that it comprises: a contact component configured to contact an external object and generate deformation; a projection component configured to project an active light field having time modulation characteristics onto a first surface of the contact component, the active light field forming a plurality of patterned regions on the first surface, the light intensity of adjacent patterned regions varying based on different time-series codes, the first surface facing away from a second surface of the contact component that contacts the external object; a collection component comprising a plurality of collection units arranged in an array, configured to collect a plurality of the patterned regions and output a time-stamped collection data sequence; and a processing component configured to perform at least the following: obtaining an observation code sequence for each collection unit based on the received time-stamped collection data sequence; identifying the plurality of observation code sequences to determine the time-series code to which the observation code sequence belongs. Based on the time series encoding corresponding to each of the acquisition units, the three-dimensional morphology of the contact component on one side of the first surface is reconstructed;1. Based on the reconstructed three-dimensional topography, the pressure distribution of the external object acting on the contact component is inverted; and tactile data is output based on the pressure distribution. 2. The system according to claim 1, wherein the tactile data includes at least one of the following: contact state information between the contact component and the external object; pressure distribution information of the external object acting on the contact component; total normal force of the external object acting on the contact component; shear force of the external object acting on the contact component; and slippage information of the external object relative to the contact component. 3. The system according to claim 1, wherein the processing component is further configured to: process the time-stamped acquisition data sequence using a spatial intersection algorithm based on fixed position parameters between the projection component and the acquisition component to reconstruct the three-dimensional topography. 4. The system according to claim 1, wherein the projection component is further configured to: vary the light intensity of two pattern regions spatially greater than or equal to a predetermined distance according to the same time-series encoding. 5. The system of claim 1, wherein the projection component is further configured to: set the Hamming distance between the time series codes corresponding to adjacent pattern regions to be greater than or equal to a predetermined threshold. 6. The system of claim 1, wherein the processing component is further configured to: filter the received time-stamped acquisition data sequence before obtaining the observation coding sequence of each acquisition unit, including: counting the acquisition data of each acquisition unit within the coding period of the time series code; and excluding acquisition units whose acquisition data count is lower than a response rate threshold to form a set of acquisition units for subsequent processing. 7. The system of claim 1, wherein the contact component comprises: an outer layer for receiving contact from the external object and including the second surface; and an inner layer, stacked with the outer layer and including the first surface, wherein the inner layer comprises at least one of: a diffuse reflection layer, a scattering layer with a microstructure, a deformable speckle layer, and a texture with anisotropic optical characteristics. 8. The system of claim 7, wherein the contact component further comprises: an intermediate layer located between the outer layer and the inner layer, wherein the hardness of the intermediate layer near the outer layer is less than the hardness of the intermediate layer near the inner layer; and the hardness of the intermediate layer increases in a gradient. 9. The system of claim 1, wherein the system has a first operating mode and a second operating mode, and the material of the contact component includes a light-transmitting material.In the first operating mode, the processing component is further configured to detect changes in light intensity transmitted through the light-transmitting material to the side of the first surface based on the time-stamped data sequence; and to trigger the system to switch to the second operating mode in response to the change in light intensity satisfying a preset condition, so that the projection component projects the active light field onto the first surface. 10. The system according to claim 1, wherein the active light field comprises: structured light, the pattern of which is selected from at least one of speckle pattern, stripe pattern, Gray code pattern, and binarized dot matrix pattern. 11. The system according to claim 1, further comprising: a timing control module integrated in the projection component, the acquisition component, or the processing component, and configured to align the timing of the time-stamped data sequence with the timing of the time-series encoding. 12. A tactile sensing method, characterized in that it comprises: projecting an active light field with time-modulated characteristics onto a first surface of a contact component by a projection component, the active light field forming multiple patterned regions on the first surface, the light intensity of adjacent patterned regions varying based on different time-series codes, the first surface facing away from a second surface of the contact component in contact with an external object; acquiring multiple patterned regions by an acquisition component and outputting a time-stamped acquisition data sequence, wherein the acquisition component includes multiple acquisition units arranged in an array; and performing at least some of the following based on the received time-stamped acquisition data sequence: obtaining an observation code sequence for each acquisition unit based on the received time-stamped acquisition data sequence; identifying the multiple observation code sequences to determine the time-series code to which the observation code sequence belongs; reconstructing a three-dimensional topography of the contact component on one side of the first surface based on the time-series code corresponding to each acquisition unit; inverting the pressure distribution of the external object acting on the contact component based on the reconstructed three-dimensional topography; and outputting tactile data based on the pressure distribution. 13. The method according to claim 12, wherein reconstructing the three-dimensional morphology of the contact component on one side of the first surface based on the time series encoding corresponding to each of the acquisition units comprises: processing the time-stamped acquisition data sequence using a spatial intersection algorithm based on fixed position parameters between the projection component and the acquisition component to reconstruct the three-dimensional morphology. 14. The method according to claim 12, wherein projecting an active light field with time-modulated characteristics onto the first surface of the contact component by the projection component, the active light field forming multiple patterned regions on the first surface, and the light intensity of adjacent patterned regions changing based on different time series encodings, comprises:The light intensity of two patterned regions spatially greater than or equal to a predetermined distance is varied according to the same time-series encoding. 15. The method of claim 12, wherein the projection of an active light field with time-modulation characteristics onto a first surface of a contact component by a projection component, the active light field forming a plurality of patterned regions on the first surface, the light intensity of adjacent patterned regions varying based on different time-series encodings, further comprises: setting the Hamming distance between the time-series encodings corresponding to adjacent patterned regions to be greater than or equal to a predetermined threshold. 16. The method of claim 12, wherein the method further comprises: filtering the received time-stamped acquisition data sequence before obtaining the observation encoding sequence of each acquisition unit, including: counting the acquisition data of each acquisition unit within the encoding period of the time-series encoding; and excluding acquisition units whose count of the acquisition data is lower than a response rate threshold to form a set of acquisition units for subsequent processing. 17. The method of claim 12, wherein the material of the contact component comprises a light-transmitting material, and the method further comprises: acquiring a data sequence based on the time stamp, detecting a change in light intensity transmitted through the light-transmitting material to the side where the first surface is located; and triggering the projection component to project the active light field onto the first surface in response to the change in light intensity satisfying a preset condition. 18. The method of claim 12, wherein the method further comprises: aligning the timing of the time-stamped data sequence with the timing of the time-series encoding. 19. An electronic device, comprising: a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement at least a portion of the tactile sensing method of any one of claims 12 to 18. 20. A non-volatile computer-readable storage medium storing computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement at least a portion of the tactile sensing method of any one of claims 12 to 18. 21. A computer program product, characterized in that it comprises a computer program, which, when executed by a processor, implements at least some of the steps of the tactile sensing method according to any one of claims 12 to 18. Claims 3 / 3 Page 4 CN 121614037 A Tactile Sensing System and Method, Electronic Device, Storage Medium, Program Product Technical Field

[0001] Embodiments of this disclosure relate to the field of robot tactile sensing and three-dimensional vision measurement, and particularly to tactile sensing.Systems, tactile sensing methods, electronic devices, non-volatile computer-readable storage media, and computer program products. Background Art

[0002] With the rapid development of emerging applications such as robot operation, intelligent manufacturing, medical rehabilitation, and virtual reality or augmented reality, the demand for refined and dynamic tactile sensing between end effectors and the environment is becoming increasingly urgent.

[0003] Taking robot operation as an example, the robot's fingers not only need to detect whether there is contact with external objects, but also need to accurately perceive the contact position, contact area, pressure distribution information of the external object acting on the contact position, total normal force, shear force, etc., and also need to accurately perceive the sliding trend and friction changes of the external object relative to the contact position.

[0004] Therefore, the industry urgently needs a new tactile sensing solution that can take into account high frame rate, low power consumption, strong anti-interference ability, and high sensitivity to dynamic contact changes, so as to improve the adaptability and reliability of intelligent devices such as robots in complex interactive tasks.

[0005] Embodiments of this disclosure provide tactile sensing systems, tactile sensing methods, electronic devices, non-volatile computer-readable storage media, and computer program products that can at least partially solve the above-mentioned problems or other problems in the art.

[0006] In a first aspect, embodiments of the present disclosure provide a tactile sensing system, comprising: a contact component configured to contact an external object and generate deformation; a projection component configured to project an active light field having time modulation characteristics onto a first surface of the contact component, the active light field forming multiple patterned regions on the first surface, the light intensity of adjacent patterned regions varying based on different time-series codes, the first surface facing away from a second surface of the contact component that contacts the external object; a collection component including multiple collection units arranged in an array and configured to collect multiple patterned regions and output a time-stamped collection data sequence; and a processing component configured to perform at least the following: obtaining an observation code sequence for each collection unit based on the received time-stamped collection data sequence; identifying the multiple observation code sequences to determine the time-series code to which the observation code sequence belongs; reconstructing a three-dimensional topography of one side of the contact component on the first surface based on the time-series code corresponding to each collection unit; inverting the pressure distribution of the external object acting on the contact component based on the reconstructed three-dimensional topography; and outputting tactile data based on the pressure distribution.

[0007] In a second aspect, embodiments of this disclosure provide a tactile sensing method, the method comprising: projecting an active light field with time modulation characteristics onto a first surface of a contact component by a projection component; the active light field being projected onto the first surface to form multiple patterned regions; the light intensity of adjacent patterned regions changing based on different time-series encodings; the first surface facing away from a second surface of the contact component that is in contact with an external object; and acquiring multiple patterned regions by a acquisition component and outputting time-stamped acquisition data.The sequence, wherein the acquisition component includes multiple acquisition units arranged in an array; and based on the received time-stamped acquisition data sequence, performing at least part of the following: obtaining an observation coding sequence for each acquisition unit based on the received time-stamped acquisition data sequence; identifying multiple observation coding sequences to determine the time series coding to which the observation coding sequence belongs; reconstructing the three-dimensional morphology of one side of the contact component on the first surface based on the time series coding corresponding to each acquisition unit; inverting the pressure distribution of an external object acting on the contact component based on the reconstructed three-dimensional morphology on page 1 / 30 of the specification 5 CN 121614037 A; and outputting tactile data based on the pressure distribution.

[0008] In a third aspect, an electronic device is provided, the electronic device including a processor, the processor being usable to implement the tactile sensing method in the second aspect and any possible implementation of the second aspect described above.

[0009] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, having stored thereon computer program instructions, which, when executed by a processor, implement the tactile sensing method in the second aspect and any possible implementation of the second aspect described above.

[0010] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements at least a portion of the steps of the tactile sensing method in the second aspect and any possible implementation of the second aspect.

[0011] It should be understood that the content described in this section is not intended to identify key or important features of embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0012] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0013] Some exemplary embodiments of this disclosure will now be described with reference to the accompanying drawings. These accompanying drawings are for illustrative purposes only and are not intended to limit the claimed technical solutions. In the drawings: Figure 1 is a schematic diagram of the structure and working principle of a tactile sensing system provided according to an exemplary embodiment of the present disclosure; Figure 2 is a schematic diagram of the structure of a cavity provided according to an exemplary embodiment of the present disclosure; Figures 3 and 4 are schematic diagrams of the working principle of a processing component provided according to an exemplary embodiment of the present disclosure; Figure 5 is a timing diagram of time-series encoded projected structured light and time-stamped acquisition data provided according to an exemplary embodiment of the present disclosure; Figure 6 is a timing diagram of time-series encoded projected structured light and time-stamped acquisition data provided according to an exemplary embodiment of the present disclosure; Figure 7 is a schematic diagram of the principle of spot aggregation and center positioning provided according to an exemplary embodiment of the present disclosure; Figure 8 is a schematic diagram of the principle of three-dimensional reconstruction based on spatial intersection provided according to an exemplary embodiment of the present disclosure.Figure 9 is a flowchart illustrating a tactile sensing method provided according to an exemplary embodiment of the present disclosure; Figure 10 shows a schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure. Detailed Description

[0014] To better understand the present disclosure, various aspects of the present disclosure will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely descriptions of exemplary embodiments of the present disclosure and are not intended to limit the scope of the present disclosure in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression “and / or” includes any and all combinations of one or more of the associated listed items.

[0015] It should be noted that in this specification, the expressions first, second, etc., are used only to distinguish one feature from another and do not indicate any limitation on the features. Therefore, the first surface discussed below may also be referred to as the second surface without departing from the teachings of the present disclosure. Specification 2 / 30 pages 6 CN 121614037 A

[0016] It should also be understood that expressions such as “comprising,” “including,” “having,” “including,” and / or “containing” are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Additionally, the use of “exemplarily” is intended to refer to an example or illustration.

[0017] Unless otherwise defined, all terms used in this disclosure (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It should also be understood that terms (e.g., terms as defined in common dictionaries) should be understood to have the meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense, unless expressly so defined in this disclosure.

[0018] It should be noted that, without conflict, embodiments and features in embodiments of this disclosure can be combined with each other. Furthermore, unless explicitly limited or contradicted by the context, the specific steps included in the methods described in this disclosure are not necessarily limited to the order in which they are described, but can be performed in any order or in parallel. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] FIG1 is a schematic diagram of the structure and working principle of a tactile sensing system 1000 provided according to an exemplary embodiment of this disclosure.

[0020] As shown in FIG1, an embodiment of this disclosure provides a tactile sensing system 1000, which includes: a contact component 100, a projection component 200, a collection component 300, and a processing component 400, wherein the contact component 100 is configured to contact an external object and generate deformation; the projection component 200 is configured to project a device onto a first surface of the contact component 100.An active light field with time-modulated characteristics is projected onto a first surface to form multiple patterned regions. The light intensity of adjacent patterned regions varies based on different time-series codes. The first surface faces away from the second surface of the contact component 100, which is in contact with an external object. The acquisition component 300 includes multiple acquisition units arranged in an array and is configured to acquire multiple patterned regions and output a time-stamped acquisition data sequence. The processing component 400 is configured to perform at least the following: obtaining an observation code sequence for each acquisition unit based on the received time-stamped acquisition data sequence; identifying multiple observation code sequences to determine the time-series code to which the observation code sequence belongs; reconstructing the three-dimensional topography of one side of the contact component 100 on the first surface based on the time-series code corresponding to each acquisition unit; inverting the pressure distribution of the external object acting on the contact component 100 based on the reconstructed three-dimensional topography; and outputting tactile data based on the pressure distribution.

[0021] The tactile sensing system 1000 proposed in the embodiments of this disclosure is designed based on a projection component 200 and a collection component 300. The collection component 300 includes multiple collection units arranged in an array and generates a time-stamped collection data sequence based only on changes in light intensity, thus making it suitable for tactile sensing such as high-speed sliding or vibration. In addition, since the collection component 300 generates a time-stamped collection data sequence based only on changes in light intensity, it detects less redundant data, which can significantly reduce the operating power consumption of the tactile sensing system 1000.

[0022] Furthermore, the contact component 100 includes a first surface and a second surface arranged opposite to each other, wherein an external object can contact the second surface, and the projection component 200 can project an active light field onto the first surface. The active light field is projected onto the first surface to form multiple patterned regions. The light intensity of adjacent patterned regions changes based on different time-series encodings. The collection component 300 can collect the light intensity changes of the reflected light from the active light field on the first surface and output a time-stamped collection data sequence.

[0023] When the contact component 100 is not deformed, the reflected light is affected by the intensity change of the active light field, resulting in a corresponding intensity change. The acquisition component 300 can output a time-stamped acquisition data sequence based on the intensity change of the reflected light. When the contact component 100 is deformed, the reflected light is affected not only by the intensity change of the active light field but also by the deformation of the first surface. In this case, the acquisition component 300 can still output a time-stamped acquisition data sequence based on the intensity change of the reflected light. The output time-stamped acquisition data sequence includes the deformation information of the contact component 100. Therefore, the tactile sensing system 1000 can reconstruct the three-dimensional shape of one side of the first surface of the contact component 100 based on this time-stamped acquisition data sequence. Based on this, the tactile sensing system 1000 can also obtain the pressure distribution of an external object acting on the contact component 100.The tactile data of the external object acting on the contact component 100, etc.

[0024] The time-stamped acquisition data sequence may include a data stream acquired by an event camera or other photoelectric detection array that generates synchronous or asynchronous acquisition data based on changes in light intensity. The acquisition data in the time-stamped acquisition data sequence includes at least: the time information, position information, and light intensity change information characterizing the direction or amplitude of the change of the reflected light. When the acquisition unit receives the light intensity change caused by the reflected light and the change amplitude exceeds a preset light intensity threshold, an acquisition data can be generated.

[0025] For example, the time-stamped acquisition data sequence may include any of the following different data sequences: a light intensity change event sequence output by an event camera (hereinafter referred to as an event sequence); a light intensity pulse signal sequence output by a pulse camera; a photon-triggered timestamp sequence output by a photodetector array or SPAD (Single-Photon Avalanche Diode) array; a jump time sequence obtained based on light intensity acquisition by a high-speed image sensor or an in-pixel (ADC-in-pixel) converter array; or other non-frame acquisition data sequences capable of recording the change time or phase information of reflected light signals.

[0026] The time-stamped acquisition data sequence acquired by the event camera can be described as an event sequence, which may include multiple events. The multiple events in the event sequence are asynchronous events that are independent of each other in terms of time, spatial location, and event polarity. Specifically, the event camera may include multiple event pixels (hereinafter referred to as pixels) with different pixel positions, and these pixels may form a pixel array. Reflected light illuminating the pixel array can form a light spot area. Based on the reflected light collected by one or more pixels in the light spot area, an event sequence characterizing the spatiotemporal information of the reflected light can be generated. The events in the event sequence may include the time information, location information, and polarity information of the change in light intensity of the reflected light. When a pixel receives a brightness change caused by the reflected light, and the change amplitude exceeds a preset threshold, an event can be generated.

[0027] The working mode of the event camera may include an asynchronous working mode and a synchronous working mode. In the asynchronous working mode, each pixel of the event camera can operate independently, and it can trigger an event instantly when the light intensity change exceeds a set threshold. The event may contain an independent timestamp. When the event camera receives samples in the synchronous working mode, all pixels in the pixel array can be sampled at the same time, and the light intensity difference between the current round and the previous round is calculated to generate a sparse event matrix. It can be understood that each frame shares a unified timestamp. Therefore, in the synchronous working mode, the event sequence generated by the event camera can be understood as a sequence of sampled data obtained at a fixed sampling frequency.

[0028] As an option, the event sequence may include multiple events, such as multiple events occurring in chronological order. Events may include timestamps of changes in the intensity of reflected light, location information, and polarity information. For example, the event sequence may be represented as a data set {t_1, t_2, ..., t_n}, where t_1, t_2, and t_n may be multiple events occurring in chronological order. Alternatively, multiple events in the event sequence may be arranged in ascending order according to their timestamps.

[0029] Taking robot operation as an example, the robot's fingers, as the end effectors of an intelligent device, not only need to detect whether there is contact with an external object, but also need to accurately perceive the contact position, contact area, pressure distribution information of the external object acting on the contact position, total normal force, shear force, etc., and also need to accurately perceive multi-dimensional information such as the sliding trend and friction changes of the external object relative to the contact position. The skin at the tip of the robot's finger can be understood as a specific instance of a contact component 100, which can contact an external object and deform. The first surface of the contact component 100 can be understood as the inner surface of the contact component 100 facing the projection component 200 and the acquisition component 300. This surface is the direct projection target of the active light field and the main acquisition interface for deformation optical information. In contrast, the second surface of the contact component 100 can be understood as the contact surface of the contact component 100 with an external object in the external environment. Specification 4 / 30 page 8 CN 121614037 A

[0030] One side of the contact component 100 is exposed to the external environment for interaction with an external object. The external object may be a physical object (e.g., a workpiece grasped by a robot, a human finger, etc.). When the external object contacts the contact component 100 and applies pressure, the contact component 100 can convert the pressure into its own deformation. It is understood that causing deformation does not necessarily depend on direct physical contact. Non-contact physical actions, such as air pressure, liquid pressure (e.g., underwater applications), or forces applied by a changing electromagnetic field, may also cause the contact component 100 to produce measurable deformation.

[0031] Existing tactile sensing technologies include array sensors based on resistance or capacitance changes, piezoelectric material sensors, fiber optic sensors, and three-dimensional vision measurement schemes based on frame cameras and active light fields (e.g., structured light).

[0032] Taking the three-dimensional vision measurement scheme as an example, due to its reliance on frame cameras for image acquisition, the inherent exposure and readout delays result in high motion delays and strict frame rate limits. In high-speed sliding or high-frequency vibration scenarios, it is prone to failure due to image blurring, making it difficult to achieve true real-time dynamic tactile sensing. Furthermore, this three-dimensional vision measurement scheme utilizes three-dimensional reconstruction algorithms (e.g., grayscale correlation matching), which are computationally complex, time-consuming, and have large matching windows, leading to limited three-dimensional data.The low output rate and high system power consumption make it difficult to deploy on embedded or mobile platforms that are sensitive to computing power and power consumption. In addition, for tactile surfaces covered with soft materials, relying solely on a single deformation or height information for mechanical inversion makes its accuracy and robustness susceptible to changes in the material, geometry, and working conditions of external objects, thus limiting its reliable application in real and variable environments.

[0033] To at least solve the above-mentioned technical problems, the tactile sensing system 1000 proposed in the embodiments of this disclosure projects a time-series encoded active light field onto the first surface of the contact component 100 through the projection component 200, and uses a collection component 300 for detection. The collection component 300 includes multiple collection units arranged in an array, and generates a time-stamped collection data sequence based solely on changes in light intensity. Since the collection unit only responds to changes in light intensity and outputs collection data with precise timestamps, the system has no exposure delay or fixed frame rate limitation, and can capture microsecond-level deformation signals caused by high-speed sliding or vibration in near real-time, thereby realizing real-time dynamic tactile sensing of dynamic contact processes.

[0034] The time-stamped data sequence only includes information about changes in light intensity, resulting in low data redundancy. This significantly reduces the amount of data that the system needs to transmit and process, and also reduces the system's power consumption. Furthermore, the time-stamp-based temporal decoding method can replace the traditional complex grayscale correlation matching algorithm, thereby greatly reducing the computational complexity and computing power required for 3D reconstruction, making this system very suitable for deployment on embedded or mobile platforms.

[0035] Furthermore, regardless of whether the contact component 100 deforms, the acquisition component 300 can output a time-stamped acquisition data sequence based on the intensity change of the reflected light. When the contact component 100 does not deform, the reflected light from the contact component 100 is affected by the intensity change caused by the time-series encoding of the active light field, resulting in a corresponding intensity change. The acquisition component 300 can output a time-stamped acquisition data sequence based on this intensity change. When the contact component 100 deforms, the reflected light is affected not only by the intensity change of the active light field but also by the deformation of the first surface. In this case, the acquisition component 300 can still output a time-stamped acquisition data sequence based on the intensity change of the reflected light, where the output time-stamped acquisition data sequence includes the deformation information of the contact component 100. This consistently effective data acquisition mechanism provides a reliable foundation for subsequent stable and high-precision deformation field reconstruction and pressure distribution inversion.

[0036] Existing three-dimensional vision measurement schemes are based on static structured light methods, whose three-dimensional reconstruction relies on the correlation and matching of texture or gray-scale space features in the image. When the surface of the soft material in the contact component deforms and its reflectivity changes due to contact with different objects, these spatial characteristics will change accordingly, leading to matching errors or failures. The embodiments provided in this disclosure...In the tactile sensing system 1000, the projection component 200 projects an active light field onto the first surface of the contact component 100. The active light field forms multiple patterned regions on the first surface. The light intensity of adjacent patterned regions changes based on different time-series encoded variations. This transfers the features upon which the above-mentioned correlation matching depends from the unstable spatial domain to the stable temporal domain. The matching process is based on the preset temporal light intensity variation pattern of the patterned regions. As long as the light intensity variation of the patterned region can be detected, it can be accurately identified. Its matching robustness is basically unaffected by surface reflection characteristics, contact material, or deformation geometry.

[0037] Furthermore, three-dimensional reconstruction can rely on matching and associating the pixel coordinates on the imaging plane of the acquisition component 300 with the preset encoded pattern coordinates in the projection component 200. The above-mentioned robust matching based on time-series encoding ensures the uniqueness and accuracy of this association, thereby guaranteeing the high precision and high stability of the three-dimensional shape reconstruction results.

[0038] The active light field may include structured light. The structure and working principle of the tactile sensing system 1000 will be described below using structured light as an example. However, those skilled in the art will understand that the embodiments described herein are merely exemplary and intended to clearly demonstrate the technical principles and solutions of the present invention, rather than constituting any limitation on the scope of protection of the present invention.

[0039] FIG2 is a schematic diagram of the structure of a cavity 1001 provided according to an exemplary embodiment of the present disclosure.

[0040] As shown in FIG1 and FIG2, in some embodiments of the present disclosure, the projection component 200 and the acquisition component 300 may be at least partially placed within the cavity 1001. Furthermore, considering that open optical path designs are susceptible to interference from ambient light, affecting the stability and reliability of measurements, the cavity 1001 may be a closed cavity, and the projection component 200 and the acquisition component 300 may be at least partially placed within this closed cavity 1001, thereby reducing the interference of ambient light on structured light projection and time-stamped data sequence acquisition.

[0041] Optionally, the contact assembly 100 may include an outer layer 110 and an inner layer 120. The outer layer 110 is configured to receive contact from an external object and includes a second surface; the inner layer 120 may be stacked on top of the outer layer 110 and includes a first surface. Alternatively, the inner layer 120 may include at least one of a diffuse reflection layer, a scattering layer with a microstructure, a deformable speckle layer, and a texture with anisotropic optical characteristics.

[0042] Furthermore, the contact assembly 100 may also include an intermediate layer 130, which may be located between the outer layer 110 and the inner layer 120, wherein the hardness of the intermediate layer 130 near the outer layer 110 is less than the hardness of the intermediate layer 130 near the inner layer 120. The hardness of the intermediate layer 130 may increase in a gradient.

[0043] Specifically, the side of the contact component 100 that contacts an external object can be defined as the outer side, and the opposite side can be defined as the inner side. The contact component 100 can serve as an end effector of a smart device, implemented as a robot finger, a tactile pad surface (e.g., a robotic arm tactile pad), a gripper liner, a tactile structure within a flexible wearable device, etc.

[0044] The outer layer 110 can be made of an elastomeric material, such as silicone / TPU (Thermoplastic Polyurethane) / PDMS (Polydimethylsiloxane), etc. Exemplarily, the diffuse reflection layer can be at least one of a white diffuse reflection coating and a microtextured film; the scattering layer with microstructures can be at least one of microbumps, microprisms, and random textures; the deformable speckle layer can be embedded within the intermediate layer 130 to ensure the stability of the structured light features. In addition, the inner layer 120 can also introduce a texture with anisotropic optical characteristics. This texture has directional optical characteristics, and when the surface on which it is located undergoes shear displacement, the shape or orientation of the reflected light spot will produce a specific change related to the displacement direction. This design facilitates the detection and quantification of in-plane displacement caused by shear force, thereby directly serving the estimation of shear force and slip information in tactile data.

[0045] The above measures can optimize optical measurement conditions, ensure that structured light can be effectively detected and analyzed, thereby improving the accuracy and reliability of three-dimensional reconstruction or surface topography measurement, and enhancing diffuse reflection and reducing specular reflection.

[0046] Specular reflection or highlights can cause some areas to be overexposed or saturated and lose structured light information. The direction of light reflection depends on the surface normal, which may cause some angles to not receive light signals. The diffuse reflection layer can make the incident light scatter uniformly in all directions, avoiding specular reflection interference. In addition, the first surface can be white. White surfaces have high reflectivity, which can reflect more structured light back to the acquisition component 300, enhancing the brightness and contrast of the stripes in the image. The diffuse reflection characteristic makes the light intensity distribution mainly depend on the incident light intensity and surface geometry, which facilitates subsequent processing. (Page 6 / 30 of the specification, CN 121614037 A)

[0047] The diffuse reflection layer, the microstructured scattering layer, and the deformable speckle layer may have the following optical characteristics, including but not limited to: high reflectivity and a flat spectrum; high reflectivity to the wavelength of the structured light used, and consistent reflection across different wavelengths, avoiding the introduction of color deviation; close to an ideal diffuse reflector, with reflected light intensity independent of the viewing angle; surface roughness much smaller than the period of the structured light fringes; uniform and smooth thickness, etc. Furthermore, textures with anisotropic optical characteristics may have the following optical characteristics, including but not limited to: random and uniform texture, spatial frequency higher than the camera resolution, avoiding the generation of moiré fringes.

[0048] It should be noted that the outer layer 110, inner layer 120 and intermediate layer 130 are functionally layered, and at least two of them can be formed by physical layering or integral molding.

[0049] In some embodiments of this disclosure, at least some modules of the projection component 200 and at least some modules of the acquisition component 300 can be disposed within the cavity 1001. Exemplarily, the cavity 1001 can be a closed cavity.

[0050] At least some modules of the projection component 200 and the acquisition component 300 can be disposed in a closed cavity with the inner surface (e.g., the first surface) of the contact component 100 in an optically accessible manner. Optically accessible can be understood as the projection component 200 being able to project structured light onto the first surface of the contact component 100, and the acquisition component 300 being able to capture light intensity change events on the first surface of the contact component 100. As an alternative, the projection component 200, which is accessible via an optical path, can be disposed on one side of the first surface of the contact component 100, with its optical axis pointing towards the first surface at a certain angle, thereby projecting a structured light pattern carrying encoded information onto the first surface; the acquisition component 300 is also disposed on the same side of the first surface and is spatially separated from the projection component 200, with its optical axis pointing towards the first surface from a different angle, thereby observing the first surface illuminated by the structured light and capturing changes in the intensity of its reflected light. The contact component 100 serves as the observed target, and its first surface is located in the intersection area of ​​the projection optical path and the acquisition optical path.

[0051] Optionally, to reduce the space of the cavity 1001 and achieve miniaturization of the tactile sensing system 1000, the projection component 200 and the acquisition component 300 can also be arranged side by side in the same direction. In this embodiment, the tactile sensing system 1000 may further include at least one optical mirror (not shown) disposed within the cavity 1001, for refraction and reflection of structured light emitted from the projection component 200 to the first surface, and / or refraction and reflection of reflected light from the first surface to the acquisition component 300, thereby also realizing a triangular optical path layout.

[0052] At least some modules of the projection component 200 disposed within the cavity 1001 may include modules related to directly projecting structured light; at least some modules of the acquisition component 300 disposed within the cavity 1001 may include modules related to directly capturing changes in light intensity. Optionally, the processing component 400 may also reconstruct the three-dimensional topography by processing the time-stamped acquisition data sequence through a spatial intersection algorithm based on fixed position parameters between the projection component 200 and the acquisition component 300. In other words, the modules related to directly capturing changes in light intensity in the acquisition component 300 and the modules related to directly projecting structured light in the projection component 200 have a fixed geometric relationship, which facilitates the calibration of the intrinsic and extrinsic parameters of the acquisition component 300 for spatial intersection point cloud recovery.

[0053] Spatial intersection algorithms can utilize two-dimensional information obtained from multiple observation devices with known spatial positions and attitudes, through...The unique coordinates of the target point in three-dimensional space are determined by the intersection of geometric models. Therefore, in some embodiments of this disclosure, the spatial intersection algorithm can utilize the fixed spatial geometric relationship (i.e., "position parameters") between the projection component 200 and the acquisition component 300 to inversely calculate three-dimensional spatial points from two-dimensional observation data. The spatial intersection algorithm may include triangulation, least squares solution, epipolar geometric constraint-based solution, bundle adjustment, and other equivalent stereo reconstruction methods.

[0054] In this invention, regardless of the specific algorithm used, as long as it utilizes a fixed projection-acquisition spatial geometric relationship to intersect the two-dimensional observation position of the coded pattern matched in the time-stamped acquisition data sequence with the projection ray and the observation line of sight in space, thereby calculating the three-dimensional topographic data, it should be considered a specific implementation of the "spatial intersection algorithm" and fall within the protection scope of this invention.

[0055] Furthermore, the remaining modules in the projection component 200 or the acquisition component 300 used for driving, control, and signal processing can be flexibly selected to be all located inside the cavity 1001, partially located inside the cavity 1001, or all located outside the cavity 1001, according to the design requirements of the tactile sensing system 1000 as described on page 7 / 30 of the specification (CN 121614037 A). This disclosure does not limit this.

[0056] The processing component 400 can be communicatively connected to the acquisition component 300 to receive the time-stamped acquisition data sequence output by the acquisition component 300 and process the time-stamped acquisition data sequence into tactile event data. The processing component 400 can be located inside or outside the cavity 1001. Alternatively, the processing component 400 may also include multiple processing modules (not shown), of which some processing modules can be located inside the cavity 1001 and others can be located outside the cavity 1001.

[0057] Optionally, the inner wall of the cavity 1001 may be coated / attached with a light-absorbing material to absorb stray reflected light and prevent spurious events that interfere with the measurement. Similarly, the cavity 1001 may also be provided with at least one of a light-absorbing wall 1004 and a light-shielding structure 1005, wherein the light-absorbing wall 1004 may be made of a light-absorbing material, and the light-absorbing wall 1004 and the light-shielding structure 1005 may be arranged in the optical path where stray light reflection may occur to absorb stray light and suppress spurious events.

[0058] Referring again to FIG1 and FIG2, in some embodiments of this disclosure, the projection assembly 200 is used to project time-series encoded structured light onto a first surface of the contact assembly 100, the structured light being projected onto the first surface to form a plurality of patterned regions, the light intensity of adjacent patterned regions varying based on different time-series encodings.

[0059] Optionally, the projection assembly 200 may vary the light intensity of two patterned regions that are spatially greater than or equal to a predetermined distance according to the same time-series encoding. Optionally, the projection component 200 can also correspond to adjacent pattern areas.The Hamming distance between time-series codes is set to be greater than or equal to a predetermined threshold.

[0060] Specifically, the projection component 200 may include an emitter, optical elements, a drive controller, and a timing control module. The emitter in the projection component 200 may be one or a combination of two or more of the following: a VCSEL (Vertical-Cavity Surface-Emitting Laser) array, a laser diode array, a microprojector, or a multi-channel LED (Light-Emitting Diode).

[0061] Alternatively, the pattern of the structured light may be one or more coded patterns selected from speckle patterns, stripe patterns, Gray code patterns, and binarized dot matrix patterns.

[0062] For example, a VCSEL array has the characteristics of low power consumption, high beam quality, easy two-dimensional integration into a dense dot matrix, and high modulation bandwidth, and is suitable for generating high-density speckle patterns. A laser diode array can provide higher single-point optical power. Microprojectors based on digital micromirror devices (DLP) or liquid crystal on silicon (LCOS) can generate complex patterns (e.g., Gray code stripes) through dynamic programming, exhibiting excellent programmability. Multi-channel LEDs offer advantages in terms of cost and system integration. Furthermore, hybrid projection schemes can be employed, such as combining a VCSEL array to generate background speckle and using a DLP projector to project specific auxiliary coded patterns to achieve multi-scale, highly robust depth perception. Therefore, in practical designs, a suitable emitter can be selected by comprehensively considering factors such as projection distance, spatial resolution, system cost, and power consumption.

[0063] The optical elements in the projection assembly 200 may include DOEs (Diffractive Optical Element), diffusers, collimating lenses, etc., to form the projected structured light into a multi-spot array (e.g., dot matrix), speckle, or stripes. DOEs can be used to generate random speckle or regular dot matrices, with micro- or nano-structures inside or on the surface, which can diffract light to form a certain number of tiny spots with unique distribution characteristics. A diffuser can be used to homogenize the light beam and expand the illumination angle, ensuring a uniform illumination field. A collimating lens can be used to convert the diverging light source output into parallel light, ensuring that the pattern formed by the structured light is not distorted within the effective distance.

[0064] Alternatively, the above-mentioned emitter and optical elements, as related modules for directly projecting structured light in the projection assembly 200, can be disposed in the cavity 1001.

[0065] The drive controller in the projection assembly 200 can control the output timing of multiple independent "logic channels", each of which is 12 CN on page 8 / 30 of the specification.The coded light from the 121614037 A logic channels is projected onto the first surface of the contact component 100, illuminating and forming a corresponding "pattern area". In other words, the drive controller can be used to generate multi-channel time-series encoding to independently control the light intensity output timing of multiple logic channels. Multi-channel time-series encoding can be understood as multiple logic channels being assigned a distinct M-bit binary code, and the light intensity state (on or off) of each logic channel is driven sequentially to present the value of the corresponding bit in its encoding sequence, where the total number of bits M can be a natural number greater than 1.

[0066] Specifically, multi-channel time-series encoding can be understood as assigning M codes to logic channel 1 within an M-bit time sequence, for example, [0,1,0,1,1,0,0,…, 1]. Therefore, within consecutive time steps, the light intensity of logic channel 1 will change sequentially according to the light intensity change timing sequence of "off, on, off, on, on, off, off, off,…, on". Furthermore, within the aforementioned time series, M codes are assigned to logic channel 2, for example, [1,0,1,0,0,1,1,…,0], wherein the light intensity change sequence of logic channel 2 may differ from that of logic channel 1.

[0067] When the independently encoded structured light is emitted from the projection component 200 and illuminates the first surface of the contact component 100, the light from different logic channels will form a corresponding pattern area on the first surface, wherein the pattern may be a light spot or stripes, etc. Since the light intensity of adjacent pattern areas is based on different time series encoding changes, the feature matching basis relied upon by existing three-dimensional reconstruction can be shifted from "spatial grayscale or texture features" which are easily affected by surface reflection characteristics to stable "time series light intensity change features". The feature matching process is based on the light intensity of the pattern area being changed by a preset time series encoding. As long as the light intensity change of the pattern area can be detected, it can be accurately identified, and its matching robustness is basically unaffected by surface reflection characteristics, contact material, or deformation geometry.

[0068] The multi-channel time-series encoding of the drive controller is to drive the projected structured light spots / speckles / stripes in a time sequence so that they can exhibit bright / dark changes according to the encoded pulses. For ease of description, the spot / speckle / stripe is referred to as a light-emitting unit below. One or more light-emitting units can be programmed into a logic channel, and the light-emitting units within the same logic channel can exhibit the same light-emitting rhythm according to the same time-series encoding.

[0069] As an option, the drive controller can control the encoding of each logic channel independently or in groups. Independent control can be understood as driving and controlling each logic channel independently, which can achieve fully parallel signal output. Under the independent control strategy, in order to enhance the distinguishability of different channels during decoding, time-series encoding corresponding to adjacent logic channels can be set.The Hamming distance between them is greater than or equal to a predetermined threshold. In this implementation, the Hamming distance between the time series codes corresponding to adjacent pattern areas can be set to be greater than or equal to a predetermined threshold. Maintaining a sufficient Hamming distance can effectively prevent different logic channels from being confused during decoding due to signal interference.

[0070] The Hamming distance is the number of times that the corresponding bits of two coded sequences of equal length are different. For example, the time series codes "1011" and "1101" have 2 different corresponding bits, so the Hamming distance is 2. The predetermined threshold of the Hamming distance can be used to determine whether two time series codes are distinguishable or whether there is a valid error. For example, the predetermined threshold of the Hamming distance can be set to 2. If the Hamming distance between the codes of two logic channels is greater than or equal to 2, they can be significantly distinguished.

[0071] Group control can be understood as dividing multiple logic channels into multiple groups, using a unified control strategy within each group, and a differentiated strategy between groups. For example, the same time-series encoding is assigned to the logical channels corresponding to two pattern regions that are spatially far apart and have a distance greater than or equal to a predetermined distance, and they are grouped together for unified driving. Different time-series encodings are assigned to the logical channels corresponding to spatially adjacent or close pattern regions. This spatial distance-based grouping method significantly optimizes the system's encoding resources and control complexity, achieving a balance between performance and efficiency.

[0072] In addition, to enhance robustness, the embodiments of this disclosure may adopt one or more of the following encoding design methods, including but not limited to: Specification 9 / 30 pages 13 CN 121614037 A (i) By optimizing the design of the time-series encoding, the number of light intensity state (on / off) transitions within one encoding cycle is increased. More state transitions are beneficial for the acquisition component 300 to capture light intensity change events more densely, thereby making the encoding features more significant. For example, the original light intensity change sequence "on-on-off-off" is optimized to "on-off-on-off", changing from 1 transition to 3 transitions, which greatly increases the event occurrence rate.

[0073] (ii) Repeatedly project the same complete time series encoding multiple times. Utilizing this temporal redundancy, by comparing and fusing multiple periodic data, the interference of random noise can be suppressed, improving the decoding accuracy. For example, by replicating the 4ms encoding period three times, the total projection time of the structured light is 12ms. Applying mean filtering to the three sets of time series encodings can eliminate encoding errors caused by single-pulse noise.

[0074] (iii) Increase the length of the time series encoding, which can be understood as the total number of bits M described above. Longer encoding can provide a larger encoding space, thereby accommodating more unique encodings while satisfying the minimum Hamming distance constraint, significantly improving the system's anti-aliasing capability. For example, upgrading the 4-bit binary encoding to 8 bits expands the encoding space from 16 to 256.

[0075] Furthermore, in some embodiments of this disclosure, the encoding of structured light may include periodic, Gray code, binary code, pseudo-random code, etc.

[0076] Specifically, periodic encoding can be used as the encoding scheme for time series encoding. In this embodiment, periodic encoding can fix the period as a unit, allowing the emitting unit to switch between on / off or bright / dark states according to a repetitive temporal pattern. The temporal sequence within the period serves as a unique identifier, and redundancy verification is achieved through multi-period acquisition. For example, the period of the complete time series encoding can be set to 4 frames, and the light intensity change sequence of a certain emitting unit is "on-off-on-off", which can be periodically repeated in subsequent projections.

[0077] Optionally, Gray code encoding can be used as the encoding scheme for time series encoding. In this embodiment, a complete Gray code sequence is preset for the emitting unit, so that when it is driven on a continuous time step, only one bit of binary number changes between adjacent light intensity states. For example, the adjacent light intensity state sequence may be 000→001→011→010…. This characteristic of only one bit changing between adjacent states brings a fault-tolerance advantage. Even if the data acquisition fails at a certain time step, the complete encoding sequence can still be uniquely derived based on the correct states of the time steps before and after it.

[0078] Optionally, binary code encoding can be used as the encoding scheme for time series encoding. In this implementation, the light intensity state (on / off) of the emitting unit in each encoding time slot (or time step) is mapped to a binary bit (e.g., 1 / 0). The sequence composed of the states of multiple consecutive time slots constitutes the unique time series encoding of the emitting unit. During decoding, the matching of the observation sequence with the preset time series encoding can be completed through bit operations. For example, using 4-bit binary code, the time sequence of a certain emitting unit is "on-on-off-on" corresponding to binary "1101" (or, 0010), and the time sequence of another light point is "off-on-on-off" corresponding to "0110" (or, 1001).

[0079] Optionally, pseudo-random code encoding can be used as the encoding scheme for time series encoding. In this implementation, the pseudo-random code encoding can be based on a pseudo-random sequence generation algorithm to assign a time sequence with no obvious pattern to each luminous unit. High robustness matching is achieved by utilizing the autocorrelation of the sequence, avoiding confusion with the encoding sequences of adjacent logic channels. For example, using an M-bit sequence pseudo-random code, the light intensity change time sequence of a certain luminous unit is "on-off-on-on-off-off-on," which has characteristics similar to white noise. Even if light spots spatially overlap, different luminous units can still be distinguished by calculating the autocorrelation coefficient between the acquired sequence and the preset sequence.

[0080] In some embodiments of this disclosure, the tactile sensing system 1000 also includes a timing control module. The timing control module can align the timing of the time-stamped acquired data sequence with the timing of the time sequence encoding, providing a reference for the processing component 400.Decoding provides a foundation. Optionally, the timing control module can be integrated into the projection component 200, the acquisition component 300, or the processing component (page 10 / 30, CN 121614037 A 400), or it can exist as an independent hardware unit. This disclosure does not limit the specific hardware structure, implementation circuit, or software organization of the timing control module. It can be a dedicated clock generator, a programmable logic unit, a timer module in a microcontroller, or functional logic composed of a synchronization algorithm running on a processor.

[0081] It should be noted that any functional module or mechanism that can realize or maintain the above-mentioned timing correspondence by generating clock signals, issuing hardware triggers, performing software timestamp calibration, or using any other technical means, and ensure that the encoded state and event acquisition can be reliably associated on the time axis, falls within the conceptual scope of the timing control module described in this disclosure and can be used to implement the technical solution of this invention.

[0082] For example, the following description uses the projection component 200 including the above-mentioned timing control module as an example. The timing control module in the projection component 200 can be used to establish and maintain the timing alignment relationship between the projection component 200 and the acquisition component 300.

[0083] Specifically, the processing component 400 can restore the light intensity state (on or off) observed by each acquisition unit of the acquisition component 300 within the encoding time window defined by the projection component 200 based on the timestamp in the time-marked acquisition data sequence, so as to form an observation encoding sequence. If there is a deviation between the timing reference of the projection component 200 and the acquisition component 300, the acquisition data cannot be accurately mapped to its corresponding encoding time window, resulting in state judgment and decoding errors.

[0084] Therefore, in order to make the acquisition data accurately reflect the encoding characteristics of structured light, the timestamp of the time-marked acquisition data sequence can be time-aligned with the encoding time sequence of the projection component 200. Optionally, one or a combination of the following methods can be used: on the one hand, it can be achieved through hardware timing alignment, for example, by using a global clock signal, hardware trigger line or precision time protocol to unify the clock reference of the projection component 200 and the acquisition component 300; on the other hand, it can be achieved through software calibration, by having the acquisition component 300 record the known reference timing signal emitted by the projection component 200, calculate the time deviation and perform software compensation, thereby establishing a time alignment relationship at the data processing level.

[0085] Referring again to Figures 1 and 2, as described above, the acquisition component 300 can be used to capture the light intensity change caused by the reflection of structured light through the first surface and output a time-stamped acquisition data sequence.

[0086] It should be noted that the "time-stamped acquisition data sequence" is a general term for non-frame-based sampling data generated based on the light intensity change of the reflected light signal and capable of recording the information of the change time, and is not limited to a specific hardware or data format.Each data acquisition in the column includes at least: time information, location information, and light intensity change information representing the direction or magnitude of the change. When the acquisition unit receives the light intensity change caused by the reflected light, and the change magnitude exceeds a preset light intensity threshold, a data acquisition can be generated.

[0087] Taking the acquisition component 300 including an event camera as an example, the event camera may include an independent EVS (Event based Vision Sensor), or a hybrid sensor (HVS) that integrates an event sensor and an active pixel sensor (APS). The HVS has the function of outputting event data and frame images. In the embodiments of this disclosure, the HVS can perform analysis based on event data to realize tactile perception, but it is not excluded that it integrates frame image data for auxiliary analysis and verification.

[0088] Exemplarily, the event camera may include an optical system, an event vision sensor, an analog front end, and a processor, etc. The optical system can image the reflected light signal onto the photosensitive surface of the sensor, which may include a lens group, an aperture, etc. The event vision sensor includes multiple pixels arranged in an array, where each pixel can be understood as a collection unit of the acquisition component 300. Each pixel can work independently and triggers and outputs an event data only when it detects that the intensity change of the reflected light it receives exceeds a preset intensity threshold. Therefore, the redundancy of the detected data is low. This significantly reduces the amount of data that the system needs to transmit and process, and reduces the system's operating power consumption.

[0089] Events can be represented in data form. Alternatively, events can be represented in the form of a quadruple of (x, y, t, p), where t represents the timestamp of the event, which can be accurate to the millisecond or microsecond level; (x, y) represents the pixel position where the event occurs; and p (polarity) represents the polarity of the event, which characterizes the trend of light intensity change. For example, when p is +1, it can represent an increase in light intensity; when p is 0, it can represent no change in light intensity; and when p is -1, it can represent a decrease in light intensity.

[0090] The output of the event camera can also be a 1-bit or 2-bit matrix representing the states of all pixels. In a 1-bit matrix, the value of any element represents whether the corresponding pixel has an event output. If there is an event output, the matrix element value is 1; if there is no event output, the matrix element value is 0. In a 2-bit matrix, the value of any element represents whether the corresponding pixel outputs a positive or negative event, or no event output. For example, if there is a positive event output, the matrix element value is +1; if there is a negative event output, the matrix element value is -1; if there is no event output, the matrix element value is -1.The matrix element value is 0, where the first bit of any element in the 2-bit matrix can represent the sign.

[0091] Therefore, in at least one embodiment of this disclosure, since the acquisition component 300 generates a time-stamped acquisition data sequence based solely on the intensity change of reflected light, its acquisition unit has a fast response speed, and the system does not have the exposure delay and fixed frame rate limitation of traditional cameras. Therefore, it can capture microsecond-level deformation signals caused by high-speed slip or high-frequency vibration in near real-time.

[0092] In addition, the time-stamped acquisition data sequence contains only light intensity change information, and the data redundancy is extremely low. This significantly reduces the data transmission load, processing burden, and overall operating power consumption of the system. The processing component 400 can perform time-series decoding based on precise timestamps, replacing the computationally complex grayscale correlation matching algorithm, reducing the computational complexity and computing power required for 3D reconstruction, making the system very suitable for deployment on embedded or mobile platforms that are sensitive to power consumption and computing power.

[0093] In addition, regardless of whether the contact component 100 is deformed, the acquisition component 300 can output a time-stamped acquisition data sequence based on the intensity change of reflected light. In the absence of contact, the time-stamped data sequence reflects the encoding changes of the structured light itself; in the presence of contact deformation, the time-stamped data sequence reflects the encoding changes of the structured light itself and the deformation information of the first surface. This data acquisition mechanism, which is effective in any state, provides a reliable foundation for subsequent stable and high-precision deformation field reconstruction and pressure distribution inversion.

[0094] Figures 3 and 4 are schematic diagrams of the working principle of the processing component 400 (as shown in Figure 1) provided according to an exemplary embodiment of the present disclosure.

[0095] As shown in Figures 1, 3, and 4, in some embodiments of the present disclosure, the processing component 400 can perform time-series structured light feature extraction and recognition based on EVS. Specifically, the processing component 400 obtains the observation encoding sequence of each acquisition unit based on the received time-stamped data sequence; and identifies multiple observation encoding sequences to determine the time-series encoding to which the observation encoding sequence belongs. As an alternative, before obtaining the observation coding sequence of each acquisition unit, the processing component 400 may also filter the received time-stamped acquisition data sequence. This filtering may include: counting the acquisition data of each acquisition unit within the coding period of the time series coding; and excluding acquisition units whose acquisition data count is lower than the response rate threshold, to form a set of acquisition units for subsequent processing.

[0096] Optionally, identifying multiple observation coding sequences to determine the time series coding to which the observation coding sequence belongs may include determining the coding sequence to which the observation coding sequence belongs through Hamming distance metric, correlation operation, frequency domain demodulation, statistical similarity metric, maximum likelihood or other equivalent time series feature identification methods. To simplify the above description, the following will refer to the coding sequence to which the observation coding sequence belongs.The content is described as matching the time series code corresponding to each acquisition unit based on the observed coding sequence, or "matching".

[0097] The processing component 400 can be used for real-time processing, timing calibration, format conversion, etc. of time-stamped acquisition data sequences. The processing component 400 can be built based on high-performance FPGA (Field-Programmable Gate Array), MCU (Microcontroller Unit), DSP (Digital Signal Processor), GPU (Graphics Processing Unit), or NPU (Neural Processing Unit), etc., and has parallel computing capabilities. The processing component 400 can also integrate a parameter configuration interface to dynamically adjust sensor thresholds, filtering parameters, etc. from the outside to adapt to different coding schemes and application scenarios. Therefore, the processing component 400 can perform low-latency processing on the acquired data in a large number of time-stamped acquisition data sequences.

[0098] Optionally, the processing component 400 may also perform at least one of the following processing: performing timing calibration on the acquired data in the time-stamped acquisition data sequence, and correcting the timestamp deviation caused by hardware transmission by combining the clock reference provided by the timing control module in the projection component 200, so as to ensure that the timestamp of the acquired data is aligned with the encoded time sequence of the projection component 200; filtering the received time-stamped acquisition data sequence, etc.

[0099] Optionally, the processing component 400 may perform a series of processing on the received time-stamped acquisition data sequence and output tactile data reflecting the tactile perception of the contact component 100. For example, the tactile data may include at least one of the following data: contact state information between the contact component 100 and the external object; pressure distribution information of the external object acting on the contact component 100; total normal force of the external object acting on the contact component 100; shear force of the external object acting on the contact component 100; and slippage information of the external object relative to the contact component 100.

[0100] The processing component 400 is optional in hardware implementation and may adopt different integrated architectures. For example, the processing component 400 can be a highly integrated single-chip architecture, implemented as a System on Chip (SoC). This SoC integrates a central processing unit, dedicated hardware accelerators, memory controllers, and other modules to form a complete computing system. Alternatively, the processing component 400 can also be a distributed, collaborative, modular architecture, comprising multiple processing modules (not shown), which can be discrete but collaborative hardware and software modules. The processing performed by the processing component can be handled by dedicated...Executed using chips, programmable chips, general-purpose processors, etc.

[0101] Exemplarily, the processing component 400 may be configured to perform a series of processes to extract tactile information from a time-stamped acquisition data sequence. The processing may include: filtering the time-stamped acquisition data sequence, wherein the acquisition data of each acquisition unit within the encoding period of the time-series encoding is counted; and excluding acquisition units whose acquisition data count is lower than a response rate threshold to form a set of acquisition units for subsequent processing. By setting a response rate threshold, the data corresponding to acquisition units with an average data volume lower than the threshold are excluded from subsequent calculations, thereby obtaining the set of acquisition units. This directly sparsifies the data to be processed, which can significantly reduce the complexity of subsequent operations.

[0102] Optionally, the above processing may further include: performing time-series decoding on the set of acquisition units, wherein the observation encoding sequence characterizing the light intensity state observed by each acquisition unit in the set of acquisition units can be reconstructed based on the acquisition data of each acquisition unit, the observation encoding sequence may be a light intensity state sequence, and the observation encoding sequence is matched with the multi-channel time-series encoding of the projection component 200 to determine the logical channel to which each acquisition unit belongs.

[0103] Optionally, the above processing may further include: performing spot aggregation, aggregating multiple acquisition units that are determined to be matched with the same logical channel and are spatially adjacent into a spot region. The center coordinates of each spot region are calculated through center positioning.

[0104] Optionally, the above processing may further include: comparing the center coordinates of the spot regions corresponding to the same logical channel with the preset projection coordinates of that channel in the projection component 200 through parallax calculation to obtain parallax information. Using the principle of spatial intersection, the coordinates of the center of each spot region in three-dimensional space are calculated, wherein the centers of all spot regions together constitute the three-dimensional shape of the contact component 100 on one side of the first surface. In other words, based on the fixed position parameters between the projection component 200 and the acquisition component 300, the time-stamped acquisition data sequence can be processed by the spatial intersection algorithm to reconstruct the above three-dimensional shape.

[0105] Optionally, the above processing may further include: constructing a deformation field based on the three-dimensional shape of the contact component 100 on one side of the first surface, quantifying the deformation and deformation gradient of the first surface relative to the non-contact state of the contact component 100.

[0106] Optionally, the above processing may further include: performing pressure inversion based on the deformation field, converting the deformation information into pressure / pressure distribution of the external object acting on the contact component 100 through a physical model or data-driven model (page 13 / 30 of the specification, CN 121614037 A), and outputting tactile data.

[0107] Specifically, taking the acquisition component 300 including an event camera and the time-stamped acquisition data sequence as an event sequence as an example, the event can be marked as formula (1), (1) where, is the i-th event;is the event sequence number. In the case of whole-frame encoding, k refers to the kth frame; is the x-coordinate of the pixel where the i-th event occurs in the image coordinate system; is the y-coordinate of the pixel where the i-th event occurs in the image coordinate system; is the timestamp of the i-th event; is the polarity of the i-th event.

[0108] In the step of filtering the time-stamped data sequence, the event response rate can be statistically analyzed to determine the set of acquisition units, wherein the event response rate can be defined as the ratio of the number of valid events to the number of frames in a single period. The event count (or event rate) of each pixel in the encoding period can be expressed by formula (2), .

[0109] Where, represents the pixel; represents the pixel coordinate of the i-th event; is the timestamp of the i-th event; represents the encoding period duration of the time series encoding; represents the event count of the pixel in the encoding period; represents the start time of the encoding period; represents the end time of the encoding period.

[0110] Based on this, the event response rate can be expressed by formula (3), (3) where, represents a pixel; is the event response rate of the pixel; represents the event count of the pixel in the encoding period; M is the total number of bits of the time sequence encoding in the encoding period, which is also called the encoding step number.

[0111] Figure 5 is a timing diagram of the projection structured light and time stamp acquisition data of the time sequence encoding provided in the exemplary embodiment of this disclosure.

[0112] As shown in Figures 1 and 5, the encoding step size M=6 can be set, that is, 6 bits represent one encoding. is one encoding period. The projection component 200 can project light spots of "off-on-on-off-on-off" respectively in the time according to the time sequence encoding [011010].

[0113] The event camera can detect events in sequence at the same pixel position where the above light spots are projected: 0 (the event polarity is set to 0 in the first frame), +1, 0, -1, +1, -1. The event count is 4, and the event response rate of the pixel is 4 / 6=0.667.

[0114] Optionally, a response rate threshold can be set, and the corresponding pixel is excluded when the event response rate is lower than the response rate threshold. At that time, the pixel can be assigned to the acquisition unit set.

[0115] In other words, the event sequence and the time series encoding do not directly correspond numerically. Based on the acquisition data of the pixel, the observation encoding sequence representing the light intensity state can be reconstructed. The observation encoding sequence can be a light intensity state sequence. By matching the observation encoding sequence with the multi-channel time series encoding of the projection component 200, the logical channel to which the pixel belongs can be determined.

[0116] Figure 6 is a timing diagram of the projected structured light and time-marked acquisition data according to the exemplary embodiment of the present disclosure.

[0117] As shown in Figures 1 and 6, the structured light of different logical channels is projected into different areas of the first surface.The encoding of channel A can be different from each other, and the Hamming distance between the encodings of different channels can be greater than or equal to a predetermined threshold.

[0118] In addition, the embodiments of this disclosure use structured light with brightness jumps, so at least one jump will occur within one encoding cycle. Pixels at locations not affected by structured light projection will not cause events to be generated by the event camera and will be excluded. Alternatively, before obtaining the observation encoding sequence of each acquisition unit, the received time-stamped acquisition data sequence is filtered, wherein the acquisition data of each acquisition unit within the encoding cycle of the time-series encoding is counted; and acquisition units with acquisition data counts lower than the response rate threshold are excluded to form a set of acquisition units for subsequent processing.

[0119] In some implementations of this disclosure, the response rate threshold used for the above filtering can be adaptive according to the "contact state". For example, contact between the contact component 100 and an external object can cause a change in the intensity of reflected light. Dynamically adjusting the above response rate threshold can balance the sensitivity and false detection rate of the tactile sensing system 1000.

[0120] Optionally, the tactile sensing system 1000 may have a first working mode and a second working mode, and the material of the contact component 100 may include a light-transmitting material. In the first working mode, the processing component 400 may collect a data sequence based on a time stamp and detect the change in light intensity transmitted through the light-transmitting material to the side where the first surface is located; in response to the change in light intensity satisfying a preset condition, the tactile sensing system 1000 is triggered to switch to the second working mode so that the projection component 200 projects structured light onto the first surface.

[0121] The response rate threshold may be defined as having different filtering thresholds at different times. For example, the material of the contact component 100 may include a light-transmitting material, so the contact component 100 is light-transmitting. If an external object approaches the second surface of the contact component 100, it will block some external light from entering the side where the first surface of the contact component 100 is located, resulting in a decrease in light intensity at that location. In response to the change in light intensity satisfying a preset condition, the projection component 200 can project structured light onto the first surface. Therefore, the event camera can normally detect at a relatively low frame rate, and once a change in the contact state is detected, it can immediately switch to high frame rate operation and dynamically redefine the response rate threshold.

[0122] In some other implementations of this disclosure, "local ROI (Region of Interest) adaptation" can also be added. On the imaging plane of the acquisition component 300, those pixel areas that are predicted to correspond to the occurrence of physical contact can use a lower response rate threshold to maintain high sensitivity; while for other areas on the imaging plane, a higher response rate threshold can be used to suppress noise and reduce the operating power consumption of the tactile sensing system 1000. In this embodiment, the response rate threshold can be defined as, which has different response rate thresholds in different regions of the imaging plane of the acquisition component 300. It should be noted that can be understood as a data pair, representing the position coordinates in the imaging plane.

[0123] In some embodiments of this disclosure, temporal decoding of the acquisition unit set may include: decoding matching of pixels in the acquisition unit set on the time axis. One flash corresponds to a whole frame, and time decoding matching is performed on multiple frames (e.g., 4 frames or 6 frames) of events on the same pixel.

[0124] The temporal sequence encoding of logical channel j is represented as, where the temporal sequence encoding is a sequence of M-bit 0 and 1 combinations. In the case of M=6, the example temporal sequence encoding can be [011010], [110110], [100100], etc., and there can be 64 different encodings in the encoding space. Different encodings can be used for different logical channels, and according to the previous description, the Hamming distance between the temporal sequence encodings corresponding to adjacent pattern regions can be set to be greater than or equal to a predetermined threshold so that the light intensity changes of adjacent pattern regions are significantly distinguishable.

[0125] The observation encoding sequence of the pixel is represented as follows, and the value in the sequence is obtained by whether the event is triggered in each encoding step. As shown in Table 1 below, the time sequence encoding of logic channel 1 can be [011010]. The position of the structured light modulated by it projected onto the first surface corresponds to the pixel coordinates (a, b) of the event camera. The flashing of the structured light causes the event camera to record the event sequence at pixel coordinates (a, b). If the recording is accurate, it will correspond to the time sequence encoding. At another position on the first surface, corresponding to the coordinates (c, d) of the event camera, the flashing of the structured light causes the event camera to record the event sequence at pixel coordinates (c, d) as [001010], which does not correspond to the time sequence encoding.

[0126]

[0127] Table 1

[0128] In one embodiment, as shown in formula (4), whether the pixel coordinates (x, y) are projected by the structured light pattern of the logical channel j, that is, whether the pixel (x, y) matches the logical channel j, can be determined by Hamming distance.

[0129] Wherein, represents a pixel; can be used to represent the observation encoding sequence of the pixel, which is the binary encoding after restoration encoding, wherein the event sequence in Table 1 above is [0, +1, 0, -1, +1, -1], which is reset before encoding, and the event change starts with the light spot being in an off state, so the binary encoding [011010] can be restored from the event sequence. For simplicity, this formula can refer to the restored binary encoding [011010]. Similarly, the event sequence of is [0, 0, +1, -1, +1, -1], which can be restored to binary encoding [001010].

[0130] In addition, in formula (4), represents the preset time series encoding of the i-th logical channel; represents the Hamming distance calculation function; represents the calculation of the Hamming distance for the i-th logical channel, and taking the channel number with the smallest distance as the optimal matching result.

[0131] As an option, the Hamming distance between the observation encoding sequence of pixel (x, y) and the time series encoding of the logical channel can be compared one by one to determine the time series encoding corresponding to each pixel, or the logical channel matched with each pixel. For example, there are a total of N logical channels, where the logical channel number j = 1, 2, ..., N, and N is a natural number greater than 1, and the Hamming distance is calculated one by one. The value of the logical channel number j when the Hamming distance value is the smallest is taken, and it is determined as the logical channel matched with pixel (x, y). In addition, the Hamming distance calculation process can be expressed as, where the Hamming distance between two encoding sequences of length M and is equal to the sum of the absolute values ​​of the differences between their code values ​​at each corresponding bit. Subtract the value of the m-th bit of the sequence from the value of the m-th bit of the sequence and take the absolute value. Then, sum the calculation results of all M bits to determine the Hamming distance between the two encoded sequences.

[0132] The above is completely the same as the time series encoding of logical channel 1, so the Hamming distance between the two is 1. Page 16 / 30 of the specification 20 CN 121614037 A 0. It can be confirmed that the pixel (a, b) matches logical channel 1. The above is different from the time series encoding of logical channel 1 by one bit, so the Hamming distance between the two is 1.

[0133] In addition, in order to suppress noise and enhance anti-interference ability, a decision threshold can be set: (5) Where, represents the Hamming distance calculation function; represents the Hamming distance decision threshold; represents the observed encoding sequence of the pixel; represents the preset time series encoding of the optimal matching logical channel.

[0134] In other words, when using the above method, for any pixel in the acquisition unit set, by determining the Hamming distance between its corresponding observation coding sequence and the time series coding of each logical channel, and determining the minimum distance value for channel matching, in order to suppress the influence of noise, a threshold can be set to ensure that the matching result is valid.

[0135] In response to the Hamming distance between the observation coding sequence corresponding to the pixel and the time series coding of the logical channel with index j being less than or equal to the threshold, it can be determined that the pixel matches the logical channel with index j.

[0136] For example, after calculating the Hamming distance between the observation coding sequence corresponding to pixel (c, d) and the time series coding of other logical channels, in response to, for example, the Hamming distance between the time series coding of logical channel 3 and the observation coding sequence corresponding to pixel (c, d) being 0, it can be determined that pixel (c, d) matches channel 3. In addition, in response to the Hamming distance between the observation coding sequence corresponding to pixel (c, d) and the time series coding of other logical channels, it can be determined that pixel (c, d) matches channel 3.d) After calculating the Hamming distance between the corresponding observation coding sequence and the time series coding of all logical channels, the Hamming distance 1 between the time series coding of logical channel 1 and the observation coding sequence corresponding to the pixel (c, d) is the minimum value among all Hamming distances, and it can be determined that the pixel (c, d) matches logical channel 1; in response to the Hamming distance 1 between the time series coding of logical channel 1 and the observation coding sequence corresponding to the pixel (c, d) being the minimum value among all Hamming distances, it can be determined that the pixel (c, d) does not match logical channel 1.

[0137] In one embodiment, whether the pixel coordinates (x, y) are projected by the structured light pattern of the logical channel with sequence number j, that is, whether the pixel (x, y) matches the logical channel with sequence number j, can be determined by binary decoding. Binary decoding is to directly interpret a binary sequence into an integer value and then compare these integer values. For example, the decimal integer value of [011010] is 26, and the decimal integer value of [001010] is 10. When using binary decoding for determination, the decimal values ​​after decoding may differ greatly when the difference occurs at different positions. Generally, the two values ​​need to be completely equal to determine whether they match. For example, the two codes mentioned above differ only in the second bit from the left, and the two decimal values ​​differ by 16.

[0138] In one embodiment, whether the pixel coordinates (x, y) are projected by the structured light pattern of the logical channel with sequence number j, that is, whether the pixel (x, y) matches the logical channel with sequence number j, can be determined by Gray code decoding. If the decoded Gray code sequence belongs to a certain channel, then the pixel (x, y) can be determined to match the logical channel with sequence number j.

[0139] Figure 7 is a schematic diagram of the principle of spot aggregation and center positioning provided according to an exemplary embodiment of the present disclosure.

[0140] As shown in Figures 1 and 7, the processing component 400 can also perform spot aggregation, which aggregates multiple acquisition units that are determined to match the same logical channel and are spatially adjacent into a spot area. The center coordinates of each spot area are calculated by center positioning.

[0141] Specifically, taking the acquisition component 300 including an event camera and the time-stamped acquisition data sequence as an event sequence as an example, the light-emitting unit projected by the projection component 200 will affect multiple related pixels in the light-emitting area to generate events. For example, a speckle may cover a diameter of 0.5 mm, and the event camera may record dozens of pixels that cause the event by the speckle flicker.

[0142] After filtering, decoding and channel matching, a logical channel matching each pixel has been obtained. Multiple acquisition units that are determined to be matched with the same logical channel and are spatially adjacent, as specified on page 17 / 30 of the specification 21 CN 121614037 A, can be aggregated into a light spot area.Spatially adjacent pixels can be determined using an aggregation distance threshold. For example, the aggregation distance threshold can be 10 pixels. It should be noted that this aggregation distance threshold is only an example and can be adjusted according to actual needs.

[0143] In other words, in the aggregated pixel set, except for the condition of matching the same logical channel, the difference between the horizontal and vertical coordinates of any two pixels does not exceed the aggregation distance threshold. In addition, Euclidean distance can also be used, where the distance between any two pixels does not exceed the aggregation distance threshold.

[0144] Specifically, for the set of acquisition units of the same channel j, the i-th spot region can be obtained by methods such as connected component clustering.

[0145] Taking the connected component clustering method as an example, aggregating multiple acquisition units that are determined to match the same logical channel and are spatially adjacent into a spot region may include: determining the neighboring pixels of each pixel; and clustering based on the neighborhood relationship between pixels to form a cluster.

[0146] In some embodiments, two adjacent pixels can be identified as belonging to the neighboring pixels. For example, the difference between the horizontal or vertical coordinates of two pixels is 1; or, in the case that pixel (x, y) is a non-boundary pixel, its neighboring pixels are... Furthermore, the range of neighboring pixels can be expanded; for example, the eight pixels surrounding a non-boundary pixel can be considered as neighboring pixels. Alternatively, the Euclidean distance between pixels can be used as a measure; in response to the Euclidean distance between two pixels being less than a set threshold, the two pixels are considered to be neighboring pixels.

[0147] After determining the neighboring pixels of each pixel, clustering can be performed based on the neighborhood relationships between pixels to form clusters, wherein the connectivity between pixels can be determined.

[0148] For example, clustering can be performed based on the neighborhood relationships between pixels using methods such as flood filling (flood algorithm) and Union-Find algorithm. The following explanation uses flood filling as an example. This process may include: an initialization process; a process of traversing unvisited pixels; a process of expanding neighboring pixels; and a process of generating clusters.

[0149] During initialization, all pixels can be marked as "unvisited" to create an empty cluster set C={}; During the traversal of unvisited pixels, an unvisited pixel can be selected as a seed point to initialize a temporary cluster, and the pixel can be marked as "visited"; During the expansion of neighboring pixels, each pixel in the temporary cluster can be traversed, and unvisited pixels in its neighboring point set N(p) can be added to c and marked as "visited"; This step is repeated until no new pixels can be added to the temporary cluster c; During the generation of clusters, the temporary cluster c can be added to the cluster set C, and the process of "traversing unvisited pixels" can be returned until all pixels are visited, and the spot area is obtained through connected component clustering.

[0150] Multiple acquisition units that are determined to be matched with the same logical channel and are spatially adjacent are aggregated intoAfter a spot region, the center coordinates of each spot region can be calculated by center positioning.

[0151] For example, calculating the coordinates of the center pixel of the spot region may include: defining the weight using formula (6): .

[0152] Wherein, is the event count (or event rate) of the pixel in one encoding cycle; is the weight of the pixel.

[0153] According to formula (7), the center coordinates (centroid) of the spot region can be calculated based on the weight: Specification 18 / 30 Page 22 CN 121614037 A (7) Wherein, is the center coordinate of the first spot region of the first logical channel; is the x-axis coordinate of the non-homogeneous spot center in the image coordinate system; is the y-axis coordinate of the non-homogeneous spot center in the image coordinate system; is the set of pixels corresponding to the first spot region of the first logical channel; is the weight of the pixel; is the pixel coordinate in the spot region; is the sum of the weights × coordinates of all pixels in the spot region; is the sum of the weights of all pixels in the spot region.

[0154] In other words, based on the event quantity of each pixel within the spot area, the weight of the event quantity of each pixel, and the total number of events within the spot area, the coordinates of the center pixel of the spot area can be determined. The pixel with the most events indicates that the flickering events of the projected spot are collected most fully, and the encoded information can be preserved to the greatest extent. In this calculation method, the pixel with the most events is given a higher weight, and the center will be biased towards the pixel that retains the most encoded information.

[0155] Figure 8 is a schematic diagram of the principle of three-dimensional reconstruction based on spatial intersection according to an exemplary embodiment of the present disclosure.

[0156] As shown in Figures 1 and 8, in some embodiments of the present disclosure, the processing component 400 can also compare the center coordinates of the spot areas corresponding to the same logical channel with the preset projection coordinates of the channel in the projection component 200 through parallax calculation to obtain parallax information. Using the principle of spatial intersection, the coordinates of the center of each spot area in three-dimensional space are calculated, wherein the centers of all spot areas together constitute the three-dimensional shape of one side of the contact component 100 on the first surface. In other words, based on the fixed position parameters between the projection component 200 and the acquisition component 300, the time-stamped acquisition data sequence can be processed by a spatial intersection algorithm to reconstruct the above-mentioned three-dimensional morphology.

[0157] Specifically, the center coordinates of the spot areas corresponding to the same logical channel and the original projection coordinates are used to form camera rays and projection rays in the camera coordinate system, and the intersection is obtained in three-dimensional space to obtain three-dimensional points.

[0158] The center coordinates of the spot areas obtained above correspond to the coordinates in the camera coordinate system, which can be expressed by formula (8), (8) where, is the coordinate of the center of the first spot of the first logical channel on the z=1 plane in the event camera coordinate system; is the intrinsic parameter matrix of the event camera; is the center coordinate of the first spot area of ​​the first logical channel; isThe pixel coordinates of the center of the spot; the superscript T is the transpose of the matrix / vector; is the image coordinate representation of the center coordinates of the spot area. It means that the image coordinates are projected onto the plane of z=1. For the center image coordinates of the spot area after projection, they are converted into points in the camera coordinate system through the camera intrinsic parameters.

[0159] The camera ray can be represented by the point method using formula (9), (9) where, is the coordinate of any point on the event camera ray (point method). is the camera optical center, which is generally (0, 0, 0). is a point on the plane of z=1 in the event camera coordinate system, which is obtained by converting the center coordinates of the spot area through the event camera's intrinsic parameters. is a scaling factor, which can be used to generate a series of points in the event camera coordinate system based on the coordinates of the center coordinates of the spot area corresponding to the coordinates in the camera coordinate system. This formula (9) means that the camera ray is formed by a line leading from the camera optical center to these points.

[0160] Similarly, structured light can be represented by formula (10) in the event camera coordinate system, (10) where, is the coordinate of any point on the structured light projection ray (point normal); is the representation of the projection center in the event camera coordinate system, and the two can be obtained by calibration parameters through the fixed positional relationship between the acquisition component 300 and the projection component 200. is the representation of the projection coordinate point in the event camera coordinate system, and its coordinates in the projection component can also be converted to the event camera coordinate system through calibration parameters. is the scaling factor, used to adjust the point position in the direction of the projection ray, where >0.

[0161] As shown in Figure 8, in space, the distance between two rays can be calculated. When different values ​​are taken, the distance between the two rays changes. By finding the shortest distance between two spatial lines (least square closed solution), the three-dimensional point can be obtained: (11) Where, is the optimal scaling factor that minimizes the distance between the projected ray and the camera ray; is the optimal scaling factor that minimizes the distance between the event camera ray and the projected ray; is the coordinate of the optimal position point on the projected ray in the event camera coordinate system; is the coordinate of the optimal position point on the event camera ray in the event camera coordinate system; is the three-dimensional representation of the center of the i-th spot region of the logic channel j. Combined with the encoding time of the spot, all three-dimensional point clouds can be represented as three-dimensional point clouds with time information.

[0162] In some embodiments of this disclosure, the processing component 400 can also construct a deformation field based on the three-dimensional topography of the contact component 100 on one side of the first surface, and quantify the deformation and deformation gradient of the first surface relative to the non-contact state of the contact component 100.

[0163] Specifically, a local coordinate system can be defined on one side of the contact component 100 on the first surface, and the reference surface with zero load on this side is defined as.

[0164] As an alternative, the height field can be obtained by fitting / interpolating the three-dimensional point cloud using formula (12), where is the local planar coordinate system of the first surface of the contact component, which is mapped to the event camera coordinate system; is the current time; is the three-dimensional representation of the center of the i-th spot region of the logical channel j at time t; can be one of Delaunay triangulation, RBF (Radial Basis Function Interpolation), and local planar fitting; is the height field under the local coordinates at time t.

[0165] Furthermore, using formula (13), the deformation of the first surface relative to the non-contact state of the contact component 100 can be the difference between the above-mentioned height field and the reference surface, (13) where, is the local plane coordinate system of the first surface of the contact component, which has a mapping relationship with the event camera coordinate system; is the reference height field when the contact component is non-contact (zero load), which can be pre-stored through calibration; is the height field under the local coordinate at time; is the deformation under the local coordinate at time, which can also be understood as the height change of the first surface of the contact component relative to the non-contact state of the contact component.

[0166] Using formula (14), the contact area is segmented according to the deformation change threshold, (14) where, is the local plane coordinate system of the first surface of the contact component, which has a mapping relationship with the event camera coordinate system; is the deformation under the local coordinate at time, which can also be understood as the height change of the first surface of the contact component relative to the non-contact state of the contact component; is the height change threshold for contact determination; is the contact area at time.

[0167] In addition, the contact area can be segmented according to curvature or deformation gradient, etc., and this disclosure does not limit this.

[0168] Optionally, by using formula (15), the tangential displacement (under the small deformation approximation) can be estimated by the displacement of the spot center on the image plane over time: (15) Wherein, is the local planar coordinate system of the first surface of the contact component, which has a mapping relationship with the event camera coordinate system; is the Jacobian mapping obtained by calibration, which can map the pixel displacement to the tangential displacement field of the first surface of the contact component 100. In other words, is the tangential displacement field under the local coordinates at time t; is the pixel displacement of the spot center at time t.

[0169] After obtaining the tangential displacement field, the slip index can be defined by formula (16): (16) Wherein, is the slip index at time t; represents the area of ​​the contact area between the contact component and the external object; is the L2 norm of the vector; is the area infinitesimal; is the time rate of change of the tangential displacement; the double integral is the inner integral of the contact area at time t.

[0170] Referring to formula (17), in response to the slip indexFurthermore, the contact area boundary change rate is relatively large, which triggers the slip alarm output for use in grasping closed-loop control, etc., where.

[0171] Where, is the contact area at time t; represents the area of ​​the contact area between the contact component 100 and the external object; represents the threshold of the change rate of the contact area; is the time change rate of the contact area area at time t.

[0172] In some embodiments of this disclosure, the processing component 400 can also perform pressure inversion based on the deformation field, convert the deformation information into the pressure / pressure distribution of the external object acting on the contact component 100 through a physical model or a data-driven model, and output tactile data, wherein the pressure inversion can be achieved through data-driven calibration mapping, physical model inversion, and fusion inversion of data-driven calibration mapping and physical model inversion.

[0173] Specifically, in data-driven calibration mapping, calibration data can be collected by a standard force sensor and different pressure heads under different intensities on pages 21 / 30 of the specification 25 CN 121614037 A. For example, the standard force sensor can be a six-dimensional force / torque sensor; the indenter can be any known shape, such as a spherical indenter or a cylindrical indenter.

[0174] The six-dimensional force / torque sensor can collect three-dimensional force and three-dimensional torque. The contact end of the spherical indenter is a standard sphere (e.g., a hard alloy sphere with a diameter of 5mm to 20mm), suitable for pressure calibration of point contact or small area surface contact. The contact end of the cylindrical indenter is a cylindrical plane (e.g., a circular end face with a diameter of 10mm to 30mm), suitable for pressure calibration of large area uniform contact.

[0175] Data acquisition under different forces may include: controlling the indenter to act vertically or obliquely on the first surface with a preset force (e.g., 0N to 300N, gradient interval 10N). The standard force sensor records the force / torque data under each force and converts it into pressure, and collects the deformation characteristic data of the first surface. Deformation feature data may include: Δh, Δh, local curvature, spot displacement vector, contact area, etc., where Δh is the height change and Δh is the gradient of the height change.

[0176] Using formula (18), the pressure distribution is discretized into a grid vector, and the deformation features are discretized into (e.g., local curvature, spot displacement vector, second moment of ellipse, contact area, etc.) to give a linear / nonlinear mapping.

[0177] (18).

[0178] Wherein, is the pressure distribution vector; is the mapping model, which can be at least one of linear regression, piecewise polynomial, lightweight neural network, lookup table interpolation, etc.; is the deformation feature vector.

[0179] In some embodiments, as shown in formula (19), ridge regression can also be used, (19) Wherein, is the regression coefficient matrix, each column corresponds to the regression coefficient of one dimension of the pressure grid vector; is the Frobenius norm, which characterizes the overall error of the matrix elements; is the ridge regression regularization coefficient, used to levelTo balance fitting accuracy and parameter complexity; is the pressure vector predicted by regression; is the deformation feature vector; is the minimum value of the regression coefficient matrix; is the pressure distribution vector.

[0180] As shown in formula (20), by taking the derivative of the above objective function with respect to W and setting the derivative to 0, a closed solution can be obtained, (20) where is the optimal regression coefficient matrix of ridge regression; is the feature matrix of multiple calibration samples; is the transpose of the feature matrix of multiple calibration samples; is the corresponding true pressure matrix; is the ridge regression regularization coefficient, used to balance fitting accuracy and parameter complexity; is the identity matrix.

[0181] In addition, in data-driven calibration mapping, the mapping model can also adopt at least one of the following: linear regression, piecewise polynomial, lightweight neural network and lookup table interpolation.

[0182] Optionally, the linear regression model is suitable for scenarios where deformation and pressure have an approximately linear relationship, as shown in formula (21), which can construct a multivariate linear equation, (21) where, is the predicted pressure value; is the number of deformation features; is the parameter of the k-th deformation feature; is the feature weight; is the bias term; is the sum of "weight × feature" for each deformation feature. Formula (21) obtains the optimal weight parameters by fitting the calibration dataset using the least squares method. The advantage of this model is that it has low computational complexity, strong interpretability, and is easy to deploy in real time in engineering. Specification 22 / 30 pages 26 CN 121614037 A

[0183] Optionally, the piecewise polynomial model can divide the calibration force range into multiple sub-intervals for the nonlinear relationship between deformation and pressure, and use a polynomial model to fit within each sub-interval.

[0184] Optionally, the lightweight neural network model is suitable for complex scenarios where deformation features and pressure distribution are highly nonlinear. For example, lightweight network structures such as MobileNet, ShuffleNet, or custom small fully connected networks can be used. The input layer can be multidimensional deformation features, the hidden layer can be 2-3 layers (32-128 neurons), and the output layer can be a pressure distribution matrix. In addition, network parameters can be trained using a calibration dataset, and the loss function (e.g., mean squared error loss) can be optimized using a backpropagation algorithm. The advantage of this model is its strong generalization ability, which can handle high-dimensional, strongly coupled feature data, and its lightweight structure meets the real-time computing requirements of embedded devices.

[0185] Optionally, the lookup table interpolation model is suitable for real-time calibration scenarios with extremely high computational efficiency requirements. For example, the calibration dataset can be pre-stored in a grid according to key feature parameters (e.g., height change, contact area) to form a calibration lookup table. In practical applications, the approximate pressure value is obtained by looking up the table using the input real-time deformation feature parameters, and then the accurate pressure distribution is calculated using linear interpolation or cubic spline interpolation algorithms. The advantage of this model is its fast computation speed, which does not require complex iterative calculations and is suitable for scenarios with limited hardware resources.

[0186] Furthermore, as shown in formula (22), the pressure can be predicted for the input deformation features using the above ridge regression model, (22) where, is the predicted pressure vector based on the new deformation features; is the optimal regression coefficient matrix; is the newly collected deformation feature vector. Optionally, other models can also be used to predict the pressure for the input deformation features.

[0187] Further, based on the obtained pressure distribution, the total normal force output can be obtained according to the following formula (23), (23) where, is the point in the contact area obtained by meeting the contact area segmentation conditions. The total normal force can be obtained by performing multiple integrations or accumulations on the pressure. In other words, is the total normal force exerted by the external object on the contact component at time t; is the pressure distribution function in the local coordinate system at time t; is the area element; is the double integral of the pressure on the contact area; is the pressure of the i-th grid in the contact area; is the area of ​​each grid; indicates that the summation range is all grids in the contact area.

[0188] In some embodiments of this disclosure, the physical model inversion may include: treating the first surface as an elastic body surface, as shown in formula (24), where the pressure and normal displacement satisfy a convolutional relationship, (24) where, is the normal displacement at point r; is the coordinate of the target point of the first surface of the contact component in the local coordinate system; is the integral variable of the first surface of the contact component in the local coordinate system; is the pressure of the target point of the first surface of the contact component in the local coordinate system; is the Green's function; is the total integral of the first surface of the contact component.

[0189] When a unit point force (normal concentrated force) is applied at point, the function of the normal displacement caused at point is the Green's function; which can be determined by "elastic theory / finite element / empirical fitting".

[0190] As shown in formula (25), after discretizing the pressure, the normal displacement vector and the pressure vector satisfy, (25) where, is the system matrix determined by material parameters (e.g., Young's modulus E, Poisson's ratio ν) and geometry (e.g., thickness), which can be obtained by analytical approximation or finite element offline calculation. For example, a half-space model, a thin film model or a finite element approximation can be used, and the material parameters can be corrected by a small amount of calibration. is the normal displacement; is the pressure distribution vector.

[0191] As shown in formula (26), the inversion can be performed using regularized least squares, (26) where, is the gradient operator (encouraging pressure smoothing), and a non-negative constraint is explicitly added (pressure is non-negative, enhancing physical rationality). It can be implemented using NNLS (Non-Negative Least Squares) or projected gradient; α can be understood as a regularization parameter, which can be used to control the weight of the regularization term. is the system matrix determined by material parameters (e.g., Young's modulus E, Poisson's ratio ν) and geometry (e.g., thickness), which can be obtained by analytical approximation or finite element offline calculation.The system matrix determined by Poisson's ratio (ν) and geometry (e.g., thickness d) can be obtained by analytical approximation or finite element offline calculation. is the pressure vector to be determined; is the square of the L2 norm; is the predicted normal displacement vector; is the pressure vector gradient; is the normal displacement amount.

[0192] As shown in formula (27), in some embodiments of this disclosure, the fusion inversion may include: providing prior information based on the above physical model, and then using data-driven correction, (27) where, is the pressure vector obtained by the final inversion, which is consistent with the pressure distribution vector shown in formula (18); is the mapping model, which may be at least one of linear regression, piecewise polynomial, lightweight neural network, lookup table interpolation, etc.; is the deformation feature vector; is the initial pressure vector obtained by the physical model inversion, which can be determined by formula (26).

[0193] Optionally, as shown in formula (28), the material parameters can also be regarded as learnable parameters, and a small number of calibration updates can be used to solve the errors caused by material aging, temperature changes, and batch differences.

[0194]

[0195] Wherein, is a system matrix determined by material parameters (e.g., Young's modulus E, Poisson's ratio ν) and geometry (e.g., thickness), which can be obtained by analytical approximation or finite element offline calculation; is the Young's modulus of the contact component; is the Poisson's ratio of the contact component; is the thickness of the contact component; is the pressure distribution vector; is the normal displacement.

[0196] Therefore, the processing component 400 can output tactile data after the above processing, wherein the tactile data may include at least one of the following data: contact state information between the contact component 100 and the external object; pressure distribution information of the external object acting on the contact component 100; total normal force of the external object acting on the contact component 100; shear force of the external object acting on the contact component 100; and slip information of the external object relative to the contact component 100.

[0197] In addition, this disclosure also provides specific embodiments of the tactile sensing system 1000.

[0198] Example 1

[0199] Referring to FIG1, the contact component 100 can constitute the soft skin of the robot fingertip, wherein the contact component 100 may include an outer layer, a middle layer and an inner layer stacked together, the inner layer may include a first surface projected by the structured light emitted by the projection component 200, and the outer layer may include a second surface of the contact component 100 in contact with an external object.

[0200] As an alternative, the outer layer may be made of silicone with a Shore A hardness of 10~20 and a thickness of 2~5mm; the middle layer may be made of silicone with a Shore A hardness of 30~40 and a thickness of 1~3mm, and the inner layer may be a white diffuse reflection layer formed on the middle layer, wherein the white diffuse reflection layer may have microtextures with features of 50~200µm.

[0201] The projection component 200 may be 850nmThe VCSEL array, wherein the projection component 200 can control the output timing of multiple independent "logic channels", the encoded light of each logic channel is projected onto the first surface of the contact component 100, which will illuminate and form a corresponding "pattern area". The number of multiple logic channels is 8 to 32. The number of steps in a time-series encoding within one encoding cycle can be 6. The Hamming distance between the time-series encodings corresponding to adjacent pattern areas can be greater than or equal to a predetermined threshold, wherein the predetermined threshold can be 2. Replicating the encoding cycle allows the light-emitting unit to switch between on / off or bright / dark states according to a repeating time sequence. The time sequence within the encoding cycle can be used as a unique identifier, and redundancy verification is achieved through multi-cycle acquisition to enhance robustness.

[0202] The acquisition component 300 can be fixedly installed in the cavity. The acquisition component 300 is set opposite to the first surface, or a mirror is used to adjust the optical path so that the field of view of the acquisition component 300 completely covers the first surface.

[0203] One flash of the projection component 200 can correspond to a whole frame, and the 6-frame event sequence on the same acquisition unit of the acquisition component 300 can be used for time decoding matching. The time-series encoding of the logical channel with sequence number j is represented as a 6-bit sequence of 0s and 1s.

[0204] To ensure the correctness of the time-series decoding, a known initial reference state needs to be determined for the acquisition unit before structured light projection. This can be achieved by resetting the state of the acquisition unit in hardware or logic; or by presetting a specific starting state for the time-series encoding. For example, all time-series encodings can be set to start with an "off" state.

[0205] For example, multiple time-series encodings may include [011010], [110110], [100100], etc. There can be 64 time-series encodings with a total of 6 bits in the encoding space. Different time-series encodings can be used for different logical channels. 8 to 32 encodings can be selected from the 64 encodings and applied to the above 8 to 32 logical channels.

[0206] Optionally, the processing component 400 may filter the received time-stamped acquisition data sequence before obtaining the observation coding sequence of each acquisition unit. This filtering may include: counting the acquisition data of each acquisition unit within the coding period of the time series coding; and excluding acquisition units whose acquisition data count is lower than the response rate threshold, to form a set of acquisition units for subsequent processing.

[0207] Therefore, the acquisition data of each acquisition unit within one coding period can be counted, and the event response rate can be calculated. After filtering out acquisition units whose event response rate is lower than the response rate threshold, a set of acquisition units can be obtained.

[0208] By performing time-series decoding on the acquisition data of the acquisition units in the acquisition unit set, the observation coding sequence of each acquisition unit can be obtained; the time series coding corresponding to each acquisition unit can be matched according to the observation coding sequence, or in other words,Each acquisition unit can be matched to the corresponding logical channel.

[0209] For example, Hamming distance can be used to determine whether an acquisition unit matches logical channel j. The Hamming distance between the observation coding sequence of each pixel and the time series coding of the logical channel can be compared one by one to determine the time series coding corresponding to each pixel, or the logical channel matched with each pixel.

[0210] Multiple acquisition units that are determined to match the same logical channel and are spatially adjacent can be aggregated into a spot area. For example, multiple acquisition units that match the same logical channel can be clustered according to a distance of not less than 10 pixels between them. Centroid / ellipse fitting is performed on the clustered pixels to obtain the center coordinates of each spot area.

[0211] Based on the center coordinates of each obtained spot area, a disparity map and spatial intersection algorithm are used to obtain the three-dimensional point cloud or height field of the contact component on one side of the first surface. Optionally, a six-dimensional force / torque sensor and a spherical indenter / cylindrical indenter are used to collect the height change Δh and the actual pressure distribution (or the average pressure is calculated using the known indenter area), and a lightweight model is trained to output the pressure vector P(u,v).

[0212] Example 2

[0213] Based on Example 1, a texture with anisotropic optical characteristics can be introduced into the inner layer of the contact component 100, so that the shear displacement causes a change in the shape / direction of the light spot. Therefore, the change in the direction of the principal axis of the ellipse and the major and minor axes is output in the fitting of the light spot area, and the shear force is estimated by combining the planar displacement. Slip detection is achieved based on the change rate of the contact area boundary with time and the change of the displacement vector. In other words, the tactile data that can be output by this embodiment may include the shear force of the external object acting on the contact component 100; and the slip information of the external object relative to the contact component 100.

[0214] Figure 9 is a schematic flowchart of a tactile perception method 2000 provided according to an exemplary embodiment of the present disclosure.

[0215] As shown in FIG9, the embodiments of this disclosure also provide a tactile sensing method 2000, which includes: Step S1, projecting an active light field with time modulation characteristics onto a first surface of a contact component by a projection component, the active light field being projected onto the first surface to form multiple patterned regions, the light intensity of adjacent patterned regions changing based on different time sequence codes, the first surface facing away from a second surface of the contact component that is in contact with an external object.

[0216] Step S2, acquiring multiple patterned regions by a collection component and outputting a time-stamped acquisition data sequence, wherein the collection component includes multiple acquisition units arranged in an array.

[0217] Step S3, based on the received time-stamped acquisition data sequence, performing at least part of the following: obtaining an observation code sequence for each acquisition unit based on the received time-stamped acquisition data sequence; identifying the multiple observation code sequences.The time series encoding to which the observation encoding sequence belongs is determined; based on the time series encoding corresponding to each acquisition unit, the three-dimensional morphology of the contact component on one side of the first surface is reconstructed; based on the reconstructed three-dimensional morphology, the pressure distribution of the external object acting on the contact component is inverted; and tactile data is output based on the pressure distribution.

[0218] The tactile sensing method 2000 proposed in the embodiments of this disclosure is based on the design of a projection component and an acquisition component, wherein the acquisition component includes multiple acquisition units arranged in an array, and generates time-stamped acquisition data sequences only based on light intensity changes, thus being suitable for tactile sensing such as high-speed sliding or vibration. In addition, since the acquisition component generates time-stamped acquisition data sequences only based on light intensity changes, the detected redundant data is small, which can significantly reduce the operating power consumption of the tactile sensing method 2000.

[0219] Furthermore, the contact component includes a first surface and a second surface arranged opposite to each other, wherein an external object can contact the second surface, and the projection component can project an active light field onto the first surface, wherein the active light field is projected onto the first surface to form multiple patterned regions, and the light intensity of adjacent patterned regions changes based on different time-series encoded changes. The acquisition component can acquire the light intensity changes of the reflected light from the active light field on the first surface and output a time-stamped acquisition data sequence.

[0220] When the contact component does not deform, the reflected light is affected by the light intensity changes of the active light field, resulting in corresponding light intensity changes. The acquisition component can output a time-stamped acquisition data sequence based on the light intensity changes of the reflected light. When the contact component deforms, the reflected light is affected not only by the intensity change of the active light field but also by the deformation of the first surface. In this case, the acquisition component can still output a time-stamped acquisition data sequence based on the intensity change of the reflected light. The output time-stamped acquisition data sequence includes the deformation information of the contact component. Therefore, the tactile sensing method 2000 can reconstruct the three-dimensional shape of one side of the first surface of the contact component, the pressure distribution of the external object acting on the contact component, or the tactile data of the external object acting on the contact component based on the time-stamped acquisition data sequence.

[0221] Regardless of whether the contact component deforms, the acquisition component can output a time-stamped acquisition data sequence based on the intensity change of the reflected light. This always effective data acquisition mechanism provides a reliable foundation for subsequent stable and high-precision deformation field reconstruction and pressure distribution inversion.

[0222] The time-stamped acquisition data sequence may include a data stream acquired by an event camera or other photoelectric detection array that generates synchronous or asynchronous acquisition data based on changes in light intensity. The acquired data in the time-stamped acquisition data sequence includes at least: time information, location information, and light intensity change information characterizing the direction or magnitude of the change in the intensity of the reflected light. When the acquisition unit receives the light intensity change caused by the reflected light, and the change amplitude exceeds a predetermined threshold...Given a light intensity threshold, one data acquisition can be generated.

[0223] Taking an event camera as an example, the time-stamped acquisition data sequence acquired by the event camera can be described as an event sequence. The event sequence may include multiple events, and the multiple events in the event sequence are asynchronous events that are independent of each other in terms of time, spatial location, and event polarity. Specifically, the event camera may include multiple event pixels (hereinafter referred to as pixels) with different pixel positions, and these pixels may form a pixel array. Reflected light irradiating the pixel array can form a light spot area. Based on the reflected light acquired by one or more pixels in the light spot area, an event sequence characterizing the spatiotemporal information of the reflected light can be generated. The events in the event sequence may include the time information, position information, and polarity information of the change in light intensity of the reflected light. When a pixel receives a brightness change caused by the reflected light, and the change amplitude exceeds a preset threshold, an event can be generated.

[0224] The tactile sensing method 2000 proposed in the embodiments of this disclosure projects a time-series encoded active light field onto the first surface of the contact component through a projection component, and uses an acquisition component for detection. The acquisition component 300 includes multiple acquisition units arranged in an array, and generates a time-stamped acquisition data sequence only based on the change in light intensity. Since the acquisition unit only responds to changes in light intensity and outputs acquisition data with precise timestamps, there is no exposure delay or fixed frame rate limitation. It can capture microsecond-level deformation signals caused by high-speed sliding or vibration in near real-time, thereby realizing real-time dynamic tactile perception of dynamic contact processes.

[0225] In some embodiments of this disclosure, based on the time series encoding corresponding to each acquisition unit, reconstructing the three-dimensional morphology of one side of the contact component on the first surface may include: processing the time-stamped acquisition data sequence through a spatial intersection algorithm based on fixed position parameters between the projection component and the acquisition component to reconstruct the three-dimensional morphology.

[0226] In some embodiments of this disclosure, an active light field is projected from the projection component onto the first surface of the contact component, wherein the active light field light is projected onto the first surface to form multiple pattern areas. The light intensity of adjacent pattern areas changes based on different time series encodings, which may include: changing the light intensity of two pattern areas that are spatially greater than or equal to a predetermined distance according to the same time series encoding.

[0227] The multi-channel time series encoding of the projection component is to drive the light spots / spots / stripes of the projected active light field to present bright / dark, bright / dark changes according to the encoded pulses. For ease of description, the light spot / spot / stripe will be referred to as a luminous unit in the following text. One or more luminous units can be programmed into a logical channel. The luminous units in the same logical channel can be encoded according to the same time sequence to present the same luminous rhythm.

[0228] The light intensity of two pattern areas that are spatially greater than or equal to a predetermined distance will be programmed according to the same time sequence.The encoding is changed; in other words, multiple logical channels are divided into multiple groups. A unified control strategy is adopted within each group, while a differentiated strategy is adopted between groups. For example, the same time-series encoding is assigned to the logical channels corresponding to two pattern areas that are spatially far apart and greater than or equal to a predetermined distance, and they are grouped together for unified driving. Different time-series encodings are assigned to the logical channels corresponding to spatially adjacent or close pattern areas. This spatial distance-based grouping method significantly optimizes the system's encoding resources and control complexity, achieving a balance between performance and efficiency.

[0229] In some embodiments of this disclosure, an active light field is projected from the projection component onto the first surface of the contact component. The active light field is projected onto the first surface to form multiple pattern areas. The light intensity of adjacent pattern areas varies based on different time-series encodings. This may also include setting the Hamming distance between the time-series encodings corresponding to adjacent pattern areas to be greater than or equal to a predetermined threshold. Maintaining a sufficient Hamming distance can effectively prevent different logical channels from being confused during decoding due to signal interference.

[0230] In some embodiments of this disclosure, the tactile sensing method 2000 may further include: before obtaining the observation encoding sequence of each acquisition unit, filtering the received time-stamped acquisition data sequence, including: counting the acquisition data of each acquisition unit within the encoding period of the time series encoding; and excluding acquisition units whose acquisition data count is lower than the response rate threshold, so as to form a set of acquisition units for subsequent processing. By setting the response rate threshold, the data corresponding to acquisition units with an average data volume lower than the threshold are excluded from subsequent calculations, thereby obtaining a set of acquisition units. This directly sparsifies the data to be processed and can significantly reduce the complexity of subsequent operations.

[0231] In some embodiments of this disclosure, the material of the contact component includes a light-transmitting material, wherein the tactile sensing method 2000 may further include: based on the time-stamped acquisition data sequence, detecting the change in light intensity transmitted through the light-transmitting material to the side where the first surface is located; and in response to the change in light intensity satisfying a preset condition, triggering the projection component to project an active light field onto the first surface. In other words, the above-mentioned response rate threshold can be dynamically adjusted according to the change in light intensity of reflected light caused by the contact component contacting an external object, which can take into account both the sensitivity and false detection rate of the tactile sensing method 2000.

[0232] In some embodiments of this disclosure, the tactile sensing method 2000 may further include: aligning the timing of the time-stamped data sequence with the timing of the time-series encoding.

[0233] Specifically, the processing component can reconstruct the light intensity state (on or off) observed by each acquisition unit of the acquisition component within the encoding time window defined by the projection component based on the timestamp in the time-stamped data sequence, so as to form an observation.Encoding sequence. If there is a deviation between the timing reference of the projection component and the acquisition component, the acquired data cannot be accurately mapped to its corresponding encoding time window, resulting in state judgment and decoding errors. Therefore, in order to make the acquired data accurately reflect the encoding characteristics of structured light, the timestamp of the time-marked acquisition data sequence can be aligned with the timing of the encoding time sequence of the projection component. Optionally, one or a combination of the following methods can be used: on the one hand, it can be achieved through hardware timing alignment, for example, by using a global clock signal, hardware trigger line or precision time protocol to unify the clock reference of the projection component and the acquisition component; on the other hand, it can be achieved through software calibration, by recording the known reference timing signal emitted by the projection component through the acquisition component, calculating the time deviation and performing software compensation, thereby establishing a time alignment relationship at the data processing level.

[0234] According to the embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0235] FIG10 shows a schematic block diagram of an example electronic device 3000 that can be used to implement the embodiments of this disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobility methods, such as personal digital processing, cellular phones, smartphones, wearable devices, and other similar computing methods. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0236] As shown in FIG10, device 3000 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 302 or a computer program loaded from storage unit 308 into random access memory (RAM) 303. Various programs and data required for the operation of device 3000 may also be stored in RAM 303. The computing unit 301, ROM 302, and RAM 303 are interconnected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0237] Multiple components in device 3000 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; andCommunication unit 309, such as a network interface card (NIC), modem, wireless transceiver, etc., allows device 3000 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunications networks. Computing unit 301 can be various general-purpose and / or dedicated processing components with processing and computing capabilities. Some examples of computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 301 performs the various means and processes described above, such as tactile sensing methods. For example, in some embodiments, the tactile sensing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on device 3000 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by computing unit 301, one or more steps of the tactile sensing method described above can be performed. Alternatively, in other embodiments, computing unit 301 can be configured to perform the tactile sensing method by any other suitable means (e.g., by means of firmware).

[0238] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implementation in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input method, and at least one output method, and transferring data and instructions to the storage system, the at least one input method, and the at least one output method.

[0239] The program code for implementing the apparatus of this disclosure may employ any combination of one or more programming languages.The program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing method, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a standalone software package, or entirely on a remote machine or server.

[0240] In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, method, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, methods, or apparatuses, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0241] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display method for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing method (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of methods can also be used to provide interaction with a user; for example, feedback provided to the user on pages 29 / 30 of the specification 33 CN 121614037 A can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be received from the user in any form (including sound input, voice input, or tactile input).

[0242] The systems and techniques described herein can be implemented in computing systems that include back-end components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or computing systems that include any combination of such back-end components, middleware components, or front-end components. Data can be transmitted in any form or medium.Digital data communication (e.g., communication networks) is used to interconnect components of the system. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0243] A computer system may include clients and servers. Clients and servers are generally geographically distant from each other and typically interact via communication networks. Client-server relationships are created by computer programs running on respective computers and having client-server relationships with each other.

[0244] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, apparatuses, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0245] Various embodiments of the present disclosure have been described above. The above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the market of the various embodiments, or to enable other those skilled in the art to understand the embodiments disclosed herein.

[0246] It should be understood that steps can be rearranged, added, or deleted using the various forms of processes shown above. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed herein can be achieved, and this is not limited herein.

[0247] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure. Specification 30 / 30 pages 34 CN 121614037 A Figure 1 Figure 2 Specification Drawings 1 / 8 pages 35 CN 121614037 A Figure 3Figure 4 (Page 2 / 8, CN 121614037 A) Figure 5 (Page 3 / 8, CN 121614037 A) Figure 6 (Page 38, CN 121614037 A) Figure 7 (Page 5 / 8, CN 121614037 A) Figure 8 (Page 6 / 8, CN 121614037 A) Figure 9 (Page 7 / 8, CN 121614037 A) Figure 10 (Page 8 / 8, CN 121614037 A) Abstract Embodiments of the present disclosure provide a tactile sensing system and method, an electronic device, a storage medium, and a program product. The tactile sensing system includes a contact assembly, a projection assembly, an acquisition assembly, and a processing assembly. The contact assembly is configured to come into contact with an external object and generate deformation. The projection assembly is configured to project an active light field with... time-modulation characteristics onto a first surface of the contact assembly. The active light field is projected on the first surface to form a plurality of pattern regions, wherein the light intensities of adjacent pattern regions vary according to differenttime-sequence codes. The first surface faces away from a second surface of the contact assembly that is in contact with the external object. The acquisition assembly includes a plurality of acquisition units arranged in an array, and is configured to collect the plurality of pattern regions and output time-marked acquisition data sequences.

Claims

1. A tactile sensing system, characterized by, The method comprises: contacting a component configured to be capable of contacting an external object and deforming; projecting, by a projection component, an active light field having time-modulated properties to a first surface of the contacting component, the active light field forming a plurality of pattern regions on the first surface, light intensity of adjacent pattern regions varying based on different time sequence encodings, the first surface being opposite to a second surface of the contacting component that contacts the external object; capturing, by a capturing component, a plurality of the pattern regions and outputting a time-stamped data sequence; and processing, by a processing component, at least part of: obtaining an observed encoding sequence of each of the capturing units based on the received time-stamped data sequence; identifying a plurality of the observed encoding sequences to determine a time sequence encoding to which the observed encoding sequence belongs; reconstructing a three-dimensional profile of a side of the contacting component opposite to the first surface based on the time sequence encoding corresponding to each of the capturing units; inverting a pressure distribution of the external object acting on the contacting component based on the reconstructed three-dimensional profile; and outputting haptic data based on the pressure distribution.

2. The system of claim 1, wherein, The haptic data comprises at least one of: contact state information of the contacting component and the external object; pressure distribution information of the external object acting on the contacting component; total normal force of the external object acting on the contacting component; shear force of the external object acting on the contacting component; and slip information of the external object relative to the contacting component.

3. The system of claim 1, wherein, The processing component is further configured to: reconstruct the three-dimensional profile by processing the time-stamped data sequence through a spatial intersection algorithm based on fixed positional parameters between the projection component and the capturing component.

4. The system of claim 1, wherein, The projection component is further configured to: vary light intensity of two pattern regions that are spatially greater than or equal to a predetermined distance according to the same time sequence encoding.

5. The system of claim 1, wherein, The projection component is further configured to: set a Hamming distance between the time sequence encodings corresponding to adjacent pattern regions to be greater than or equal to a predetermined threshold.

6. The system of claim 1, wherein, The processing component is further configured to: before obtaining the observed encoding sequence of each of the capturing units, screen the received time-stamped data sequence, comprising: counting capturing data of each of the capturing units within an encoding period of the time sequence encoding; and excluding the capturing unit whose count of the capturing data is lower than a response rate threshold to form a capturing unit set for subsequent processing.

7. The system of claim 1, wherein, The contacting component comprises: an outer layer for receiving the contact of the external object and comprising the second surface; and an inner layer stacked with the outer layer and comprising the first surface, wherein the inner layer comprises at least one of a diffuse reflection layer, a scattering layer having a microstructure, a deformable speckle layer, and a texture having anisotropic optical features.

8. The system of claim 7, wherein, The contacting component further comprises: an intermediate layer between the outer layer and the inner layer, The hardness of the intermediate layer close to the outer layer portion is less than the hardness of the intermediate layer close to the inner layer portion. The hardness of the intermediate layer increases in a gradient.

9. The system of claim 1, wherein, The system has a first working mode and a second working mode, and the material of the contact component includes a light-transmitting material, In the first working mode, the processing component is further configured to detect a change in light intensity transmitted to the side where the first surface is located via the light-transmitting material based on the time-labeled acquisition data sequence; and In response to the change in light intensity satisfying a preset condition, triggering the system to switch to the second working mode, so that the projection component projects the active light field to the first surface.

10. The system of claim 1, wherein, The active light field includes: The pattern of the structured light is selected from at least one of a speckle pattern, a stripe pattern, a Gray code pattern, and a binary dot matrix pattern.

11. The system of claim 1, wherein, The system further includes: A timing control module integrated in the projection component, the acquisition component, or the processing component, and configured to align the timing of the time-labeled acquisition data sequence with the timing of the time sequence encoding.

12. A method of haptically perceiving, characterized in that It includes: Projecting, by a projection component, an active light field with time modulation characteristics to a first surface of a contact component, the active light field forming a plurality of pattern regions on the first surface, the light intensity of adjacent pattern regions varying based on different time sequence encodings, the first surface facing away from a second surface of the contact component that contacts an external object; Acquiring, by an acquisition component, a plurality of the pattern regions and outputting a time-labeled acquisition data sequence, wherein the acquisition component includes a plurality of acquisition units arranged in an array; And Based on the received time-labeled acquisition data sequence, at least part of the following is performed: Based on the received time-labeled acquisition data sequence, obtaining an observed encoding sequence for each acquisition unit; Identifying a plurality of the observed encoding sequences to determine the time sequence encoding to which the observed encoding sequence belongs; Based on the time sequence encoding corresponding to each acquisition unit, reconstructing a three-dimensional topography of the contact component on the side of the first surface; Based on the reconstructed three-dimensional topography, inverting a pressure distribution of the external object acting on the contact component; and Outputting haptic data based on the pressure distribution.

13. The method of claim 12, wherein, The based on the time sequence encoding corresponding to each acquisition unit, reconstructing a three-dimensional topography of the contact component on the side of the first surface, includes: Based on the fixed position parameters between the projection component and the acquisition component, processing the time-labeled acquisition data sequence through a spatial intersection algorithm to reconstruct the three-dimensional topography.

14. The method of claim 12, wherein, The active light field projected by the projection component to the first surface of the contact component has time modulation characteristics, and the active light field forms a plurality of pattern regions on the first surface, the light intensity of adjacent pattern regions varying based on different time sequence encodings, which includes: The light intensity of two pattern regions that are greater than or equal to a predetermined distance apart in space varies according to the same time sequence encoding.

15. The method of claim 12, wherein, The active light field with time modulation characteristics is projected by the projection assembly to the first surface of the contact assembly, and the active light field forms a plurality of pattern regions on the first surface. The light intensity of adjacent pattern regions varies based on different time sequence encodings. The method further comprises: Setting a Hamming distance between the time sequence encodings corresponding to adjacent pattern regions to be greater than or equal to a predetermined threshold.

16. The method of claim 12, wherein, The method further comprises: Before obtaining the observation encoding sequence of each acquisition unit, the received time-stamped acquisition data sequence is screened, comprising: Counting the acquisition data of each acquisition unit within the encoding period of the time sequence encoding; and Excluding the acquisition unit whose count of acquisition data is lower than a response rate threshold to form a set of acquisition units for subsequent processing.

17. The method of claim 12, wherein, The material of the contact assembly comprises a light-transmitting material, and the method further comprises: Based on the time-stamped acquisition data sequence, detecting the change in light intensity transmitted to the side where the first surface is located via the light-transmitting material; and In response to the change in light intensity satisfying a preset condition, triggering the projection assembly to project the active light field to the first surface.

18. The method of claim 12, wherein, The method further comprises: Aligning the time sequence of the time-stamped acquisition data sequence with the time sequence of the time sequence encoding.

19. An electronic device, comprising: Comprise: A processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement at least part of the haptic perception method as claimed in any one of claims 12 to 18.

20. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement at least part of the haptic perception method as claimed in any one of claims 12 to 18.

21. A computer program product, characterised in that, Comprise a computer program, the computer program being executed by the processor to implement at least part of the steps of the haptic perception method as claimed in any one of claims 12 to 18.

Images