A multimodal bionic tactile perception method and system based on bionic tactile fingers

Abstract

The present invention discloses a multimodal bionic tactile perception method and system based on a bionic tactile finger. The method comprises: obtaining a charge signal generated by the bionic tactile finger, preprocessing the charge signal to obtain a piezoelectric digital signal; obtaining an image sequence captured by the bionic tactile finger, performing signal processing on the image sequence to obtain a visual tactile signal; fusing the piezoelectric digital signal and the visual tactile signal to obtain a spatiotemporal tactile signal, and performing surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signal to obtain multimodal tactile information. Based on the bionic characteristics of the finger, the present invention integrates piezoelectric sensors and visual tactile sensors to design a bionic tactile finger. Data fusion is performed in the temporal and spatial dimensions to achieve multimodal tactile perception capabilities. Through structural and functional bionics, the robot's tactile sense acquires the ability to recognize and locate different vibration stimuli in different time and space.

Description

Technical Field

[0001] The present invention relates to the field of robot tactile perception technology, and in particular to a multimodal bionic tactile perception method, system, terminal and computer-readable storage medium based on bionic tactile fingers. Background Art

[0002] A robot's tactile perception plays a crucial role in machine grasping, manipulation, and human-machine interaction. By designing anthropomorphic tactile fingers based on the structural, material, and functional characteristics of human fingertips and combining various sensing principles, robots can achieve highly sensitive tactile perception. Tactile recognition of multiple attributes, including shape, force, and texture, plays a crucial role in machine tactile perception and places high demands on tactile sensing.

[0003] Currently, a variety of sensing principles are available for acquiring tactile signals, primarily including piezoelectric and optical. However, single-type sensors are limited by their respective principles and have limited application scenarios. Combining two or more sensing principles can leverage the strengths of different sensing principles and achieve complementary advantages between different sensing modalities. Compared to other sensor principles, optical sensors offer a simple and compact structure, high spatial resolution, and high sensitivity to static pressure and low-frequency contact. They can also be used in conjunction with magnetic resonance imaging for tactile sensing in surgical robots. However, camera-based optical sensors, also known as visual tactile sensors, have a temporal response speed that depends on the camera frame rate, further limiting sensor size and resulting in low temporal resolution at the scale of a tactile finger. A key advantage of piezoelectric sensors is that they require no power supply and offer high sensitivity, reliability, and fast dynamic response. Their wide response range of 1Hz-1kHz makes them suitable for vibration measurement. However, because the voltage decreases with sustained static force, piezoelectric sensors are not suitable for static force measurement. They also suffer from low spatial resolution and poor temperature stability.

[0004] Current visual-tactile sensing methods have the problem of time response speed being limited by frame rate. Piezoelectric sensing methods have problems such as limited spatial resolution and poor static response. In addition, existing multimodal tactile sensors lack a data fusion process. For example, the vibration frequency information obtained by the piezoelectric sensor array is not fused with the information obtained by visual-tactile sensing, resulting in the failure to resolve local area frequency information at a certain spatial resolution.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0006] The main purpose of the present invention is to provide a multimodal bionic tactile perception method based on bionic tactile fingers, aiming to solve the problems of the existing technology that the time response speed of a single sensor is limited by the frame rate, the spatial resolution is limited, the static response is poor, and the multimodal tactile sensor lacks a data fusion process.

[0007] To achieve the above object, the present invention provides a multimodal bionic tactile perception method and system based on a bionic tactile finger, wherein the multimodal bionic tactile perception method based on a bionic tactile finger comprises the following steps:

[0008] Acquiring a charge signal generated by the bionic tactile finger, and preprocessing the charge signal to obtain a piezoelectric digital signal;

[0009] Acquiring an image sequence captured by the bionic tactile finger, and performing signal processing on the image sequence to obtain a visual tactile signal;

[0010] The piezoelectric digital signal and the visual tactile signal are fused to obtain a spatiotemporal tactile signal, and the spatiotemporal tactile signal is subjected to surface three-dimensional reconstruction and spectrum analysis to obtain multimodal tactile information.

[0011] Optionally, the multimodal bionic tactile perception method based on the bionic tactile finger, wherein the step of acquiring the charge signal generated by the bionic tactile finger and preprocessing the charge signal to obtain the piezoelectric digital signal, specifically includes:

[0012] When it is detected that the bionic tactile finger generates a charge signal due to external stimulation, the charge signal of the sensing area is acquired, the charge signal is amplified to obtain a target charge signal, and the target charge signal is converted to obtain a voltage signal;

[0013] The voltage signal is filtered to obtain a target voltage signal, and the target voltage signal is digitally converted to obtain a piezoelectric digital signal.

[0014] Optionally, the multimodal bionic tactile perception method based on the bionic tactile finger, wherein the acquiring of the image sequence captured by the bionic tactile finger and the signal processing of the image sequence to obtain the visual tactile signal, specifically includes:

[0015] When it is detected that the bionic tactile finger is stimulated by an external environment and an image sequence of corresponding marking points is captured, an image sequence of the sensing area is acquired, and the image sequence is separated to obtain a piezoelectric film image and a marking point image;

[0016] Performing differentiation processing on the piezoelectric film image according to the relative relationship between the area and arrangement of the piezoelectric film to obtain corresponding spatial position information, and obtaining first image sequence information according to the spatial position information and the image sequence;

[0017] The position information of each marking point is recorded according to the marking point image, second image sequence information is obtained according to all the position information and the image sequence, and a corresponding visual-tactile signal is constructed according to the first image sequence information and the second image sequence information.

[0018] Optionally, the multimodal bionic tactile perception method based on bionic tactile fingers, wherein the fusing processing of the piezoelectric digital signal and the visual tactile signal to obtain the spatiotemporal tactile signal, specifically includes:

[0019] Comparing the first image sequence information and the second image sequence information to obtain spatial coordinate information of the sensing area;

[0020] Aligning the time starting point of the piezoelectric digital signal with the time starting point of the visual-tactile signal to obtain a timing signal of the perception area;

[0021] A spatiotemporal tactile signal is obtained according to the spatial coordinate information and the time sequence signal.

[0022] Optionally, in the multimodal bionic tactile perception method based on bionic tactile fingers, the spatiotemporal tactile signal includes a tactile spatial signal and a tactile temporal sequence signal;

[0023] The performing surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signals to obtain multimodal tactile information specifically includes:

[0024] Performing three-dimensional surface reconstruction on the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result;

[0025] Setting a target time interval, and performing spectrum conversion on the tactile timing signal according to the target time interval to obtain vibration main frequency information;

[0026] Multimodal tactile information is obtained according to the three-dimensional reconstruction result and the vibration main frequency information.

[0027] Optionally, in the multimodal bionic tactile perception method based on bionic tactile fingers, the types of the tactile spatial signals include single mirror reflection and multi-mirror reflection;

[0028] The performing three-dimensional reconstruction on the surface of the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result specifically includes:

[0029] If the type of the tactile spatial signal is single mirror reflection, performing three-dimensional reconstruction on the two-dimensional coordinates of the sensing area using a preset method to obtain a three-dimensional reconstruction result, wherein the preset method includes a gradient integration of the coordinate point and a scaling ratio of the area of ​​the marker point;

[0030] If the type of the tactile spatial signal is multi-mirror reflection, a virtual binocular vision system is constructed, and the three-dimensional coordinates of each position in the perception area are calculated by the virtual binocular vision system, and all three-dimensional coordinates are interpolated to obtain a three-dimensional reconstruction result.

[0031] Optionally, the multimodal bionic tactile perception method based on bionic tactile fingers, wherein the spatiotemporal tactile signal is obtained according to the spatial coordinate information and the timing signal, further comprises:

[0032] The target information of the perception area is identified and perceived according to the spatial coordinate information and the timing signal to obtain a perception result, wherein the target information includes shape information, texture information, static force information and dynamic force information.

[0033] Optionally, the multimodal bionic tactile perception method based on bionic tactile fingers, wherein the multimodal bionic tactile perception system based on bionic tactile fingers comprises: a bionic tactile finger, a signal processing module and a fusion analysis module;

[0034] The bionic tactile finger includes a bionic fingerprint module, a piezoelectric sensor module, a visual tactile sensor module and a piezoelectric signal modulation circuit module;

[0035] The bionic fingerprint module is used to come into contact with an object;

[0036] The piezoelectric sensing module is configured to generate a corresponding charge signal when the bionic fingerprint module contacts the object;

[0037] The visual-tactile sensing module is configured to capture a corresponding image sequence when the bionic fingerprint module contacts the object;

[0038] A piezoelectric signal modulation circuit module is used to pre-process the charge signal to obtain a piezoelectric digital signal;

[0039] a signal processing module, configured to perform signal processing on the image sequence to obtain visual and tactile signals;

[0040] The fusion analysis module is used to fuse the piezoelectric digital signal and the visual tactile signal to obtain a spatiotemporal tactile signal, and perform surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signal to obtain multimodal tactile information.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multimodal bionic tactile perception program based on bionic tactile fingers stored in the memory and runnable on the processor, and when the multimodal bionic tactile perception program based on bionic tactile fingers is executed by the processor, the steps of the multimodal bionic tactile perception method based on bionic tactile fingers as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal bionic tactile perception program based on bionic tactile fingers, and when the multimodal bionic tactile perception program based on bionic tactile fingers is executed by a processor, the steps of the multimodal bionic tactile perception method based on bionic tactile fingers as described above are implemented.

[0043] In the present invention, the charge signal generated by the bionic tactile finger is obtained, and the charge signal is pre-processed to obtain a piezoelectric digital signal; the image sequence captured by the bionic tactile finger is obtained, and the image sequence is signal processed to obtain a visual tactile signal; the piezoelectric digital signal and the visual tactile signal are fused to obtain a spatiotemporal tactile signal, and the spatiotemporal tactile signal is subjected to surface three-dimensional reconstruction and spectrum analysis to obtain multimodal tactile information. Based on the bionic characteristics of the finger, the present invention integrates piezoelectric sensors and visual tactile sensors to design a bionic tactile finger, performs data fusion in the time and space dimensions, and realizes multimodal tactile perception capabilities. Through structural bionics and functional bionics, the robot's tactile sense has good comprehensive performance in terms of spatial sensitivity, time sensitivity, static force and dynamic force perception, so that the robot's tactile sense can obtain the ability to recognize and locate different vibration stimuli in different time and space. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a preferred embodiment of the multimodal bionic tactile perception method based on bionic tactile fingers in the present invention;

[0045] Figure 2 This is an overall flow chart of the multimodal bionic tactile perception method based on bionic tactile fingers in the present invention;

[0046] Figure 3 This is a schematic diagram of the structural design of the bionic tactile finger in the present invention;

[0047] Figure 4 This is a schematic diagram of the overall process of a preferred embodiment of the multimodal bionic tactile perception method based on bionic tactile fingers in the present invention;

[0048] Figure 5 1 is a structural diagram of a preferred embodiment of a multimodal bionic tactile perception system based on bionic tactile fingers in the present invention;

[0049] Figure 6 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0052] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features specified as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0053] The multimodal bionic tactile perception method based on bionic tactile fingers described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the multimodal bionic tactile perception method based on bionic tactile fingers includes the following steps:

[0054] Step S10: Acquire the charge signal generated by the bionic tactile finger, and pre-process the charge signal to obtain a piezoelectric digital signal.

[0055] The step S10 includes:

[0056] Step S11: when it is detected that the bionic tactile finger generates a charge signal due to external stimulation, acquiring the charge signal of the sensing area, amplifying the charge signal to obtain a target charge signal, and converting the target charge signal to obtain a voltage signal;

[0057] Step S12: filtering the voltage signal to obtain a target voltage signal, and digitally converting the target voltage signal to obtain a piezoelectric digital signal.

[0058] Specifically, in an embodiment of the present invention, a multimodal bionic tactile perception method based on a bionic tactile finger is proposed, such as Figure 2 As shown in the figure, this method is based on the bionic features of fingertips, combines piezoelectric sensing and visual tactile sensing, fuses electrical signals and image signals, realizes bionic tactile perception, and completes the acquisition of multimodal tactile information such as static force, dynamic force, shape, texture, etc. In order to better realize bionic tactile perception, the structure of the bionic tactile finger is designed, as shown in the figure. Figure 3 As shown, the bionic tactile finger includes a piezoelectric signal modulation circuit module 1, a piezoelectric sensor module 6, a bionic fingerprint module 7, and a visual-tactile sensor module. The visual-tactile sensor module includes a light source module 2, a camera module 3, an optical path reflection group 4, and a marking point layer 5. The piezoelectric signal modulation circuit module 1 is required to promptly capture weak electric charges and amplify them into voltage output, filter signals in the interference frequency band, and then convert the corresponding analog signals into digital signals for subsequent signal analysis. The piezoelectric sensor module 6 can mimic fast-adapting receptors (FA receptors). It uses PVDF (Polyvinylidene fluoride) film as a sensing element to simulate FA receptors in human skin. Its frequency sensing range is greater than or equal to 1 kHz, covering the perception threshold of human skin. The piezoelectric film (i.e., PVDF film) is in the shape of a long arc and is arranged to conform to the annular structure of a fingerprint. Because the normal direction of a fingerprint ridge is more sensitive to sliding stimuli than the tangential direction, the piezoelectric film's annular arrangement that conforms to the fingerprint's design helps increase sensitivity to sliding stimuli in all directions. In the direction perpendicular to the contact surface, the piezoelectric film is close to the contact surface, which is consistent with the relative position of FA-I in human skin. The piezoelectric film is located in a transparent flexible material substrate (e.g., silicone), and the flexible layer containing the piezoelectric film is overlaid on the fingerprint flexible layer. The bionic fingerprint on the bionic fingerprint module 7 has a multi-circle curved ridge structure with a wavy papillary cross-section. The purpose of setting it up is because the wavy papillary cross-section of the fingerprint has the function of amplifying the contact signal, and the corresponding fingerprint structure is attached to the outermost flexible material (for example, silicone, etc.) and can be prepared by various methods such as mold forming, and the fingerprint flexible layer has a dome-shaped arc surface, which is similar to the outer contour of the human fingertip.

[0059] The visual-tactile sensing module uses a camera to capture the changes of the marker points under external influences, which is used to calculate the magnitude of the force (for example, torque), obtain texture information or judge slippage, etc.; due to the limited frame rate of commonly used cameras, the visual-tactile sensing design is used to simulate the SA receptors (Slow-adapting receptors) in the human skin to capture low-frequency, static external stimuli. Specifically, the light source module 2 is used to provide a uniform light source to the marker points. In an environment with poor external lighting, the effective measurement of the visual-tactile sensing module can be guaranteed, and the number, color, and position of the light groups can be configured according to the specific algorithm and size, which are not limited here; the camera module 3 is used to record the changes of the marker points, and the axis of the camera lens is angled to the tangent plane of the contact surface at a certain angle. The size of the angle is determined by the design and simulation of the optical path; the optical path reflection group 4 (for example, a mirror) can play a role in changing the optical path. In order to make the shape of the bionic tactile finger more similar to the shape of a real human finger, The dimension perpendicular to the contact surface is compressed in one step, and a mirror can be placed above the marking point. The position of the mirror is calculated and simulated, and the light reflected by the marking point can be effectively projected into the camera. On the other hand, the mirror can imitate the shape of the finger nail, which increases the rigidity of the bionic finger while making the appearance more similar to the real finger. The marking points in the marking point layer 5 can be designed into a variety of shapes, such as circle, square, etc.; and the interval between the marking points is less than or equal to 1mm, which is consistent with the spatial resolution of the human fingertip. The marking points can be located on the surface of a transparent flexible substrate (such as silicone, etc.) or dispersed therein, and the flexible layer containing the marking points is covered on the flexible layer containing the piezoelectric film.

[0060] This invention utilizes a bionic tactile finger design, including wavy, papillary, annular projections to simulate fingerprints. The flexible layer where the fingerprint resides is a domed surface, simulating the fingertips. Piezoelectric sensors simulate human skin's FA-type sensors for high-frequency stimulation, arranged in a circular pattern similar to fingerprints. Visually tactile SA-type sensors simulate human skin for low-frequency and static force sensing. A mirror simulates the shape of a nail, enhancing bionic similarity and rigidity. Through the rational arrangement of the mirror and camera, reflection is used to reduce the distance perpendicular to the finger's contact surface, achieving a compact design.

[0061] After completing the design of the bionic tactile finger, bionic tactile perception is achieved through the bionic tactile finger. Specifically, when the piezoelectric sensing module of the bionic tactile finger is stimulated by the outside world, an electric charge is generated on the piezoelectric film. At the same time, the electric charge is only generated under dynamic stimulation, which is very similar to a pulse signal. Then, the charge signal of the corresponding sensing area is obtained, and the charge signal is amplified by the piezoelectric signal modulation circuit module to obtain a target charge signal. The target charge signal is converted and processed to obtain a voltage signal. Thereafter, the voltage signal is filtered to obtain a target voltage signal in order to remove the signal in the interference frequency band, and the target voltage signal is digitally converted to obtain a piezoelectric digital signal, wherein the output of each piezoelectric film corresponds to a digital signal channel output , Indicates the Piezoelectric film over time Changing piezoelectric digital signal.

[0062] Step S20: Acquire an image sequence captured by the bionic tactile finger, and perform signal processing on the image sequence to obtain a visual tactile signal.

[0063] The step S20 includes:

[0064] Step S21: when it is detected that the bionic tactile finger is stimulated by an external stimulus and an image sequence of corresponding marking points is captured, an image sequence of the sensing area is acquired, and the image sequence is separated to obtain a piezoelectric film image and a marking point image;

[0065] Step S22: performing differentiation processing on the piezoelectric film image according to the relative relationship between the area and arrangement of the piezoelectric film to obtain corresponding spatial position information, and obtaining first image sequence information according to the spatial position information and the image sequence;

[0066] Step S23: Record the position information of each marking point according to the marking point image, obtain second image sequence information according to all the position information and the image sequence, and construct a corresponding visual-tactile signal according to the first image sequence information and the second image sequence information.

[0067] Specifically, in an embodiment of the present invention, when the piezoelectric sensing module of the bionic tactile finger is stimulated by the outside world and the visual tactile sensing module captures an image sequence of the corresponding marker points, the image sequence needs to be preprocessed and separated to obtain the corresponding piezoelectric film image and marker point image. Specifically, since the marker points and piezoelectric films of the image sequence have different colors and are located in transparent substrates at different depth layers, the camera module can capture the images of the marker points and the piezoelectric film at the same time; for each frame of the image, the piezoelectric film image and the marker point image can be separated based on the differences in the color, depth and other information of the piezoelectric film and the marker point; thereafter, for a single frame of the piezoelectric film image, based on the consistency of the relative relationship between the area and arrangement of each piece of piezoelectric film, it can be distinguished which piezoelectric film the relevant pixel points in the piezoelectric film image belong to, thereby obtaining the spatial position of the piezoelectric film, and based on the image sequence, the image sequence information of the piezoelectric film is comprehensively obtained, that is, the first image sequence information , Indicates the Piezoelectric film over time Changing spatial position information. For a single-frame marker image, record the position information of each marker, and track the position of each marker according to the image sequence to obtain the image sequence information of the marker, that is, the second image sequence information , Indicates the Markers over time Then, a corresponding visual-tactile signal is constructed according to the first image sequence information and the second image sequence information.

[0068] Step S30: fusing the piezoelectric digital signal and the visual tactile signal to obtain a spatiotemporal tactile signal, and performing surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signal to obtain multimodal tactile information.

[0069] The step S30 includes:

[0070] Step S31: Compare the first image sequence information and the second image sequence information to obtain spatial coordinate information of the sensing area;

[0071] Step S32: aligning the time starting point of the piezoelectric digital signal with the time starting point of the visual-tactile signal to obtain a timing signal of the sensing area;

[0072] Step S33, performing three-dimensional surface reconstruction on the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result;

[0073] Step S34: setting a target time interval, and performing spectrum conversion on the tactile timing signal according to the target time interval to obtain vibration main frequency information;

[0074] Step S35 : obtaining multimodal tactile information according to the three-dimensional reconstruction result and the main vibration frequency information.

[0075] Specifically, after obtaining the piezoelectric digital signal and the visual tactile signal, it is necessary to perform fusion processing on the piezoelectric digital signal and the visual tactile signal. Specifically, in an embodiment of the present invention, the piezoelectric digital signal and the visual tactile signal can be fused in two dimensions, time and space. In the spatial dimension, there are two relationships between the piezoelectric film and the marking point, namely, inclusion and separation. and , you can get the first Marking point and The relationship between the piezoelectric film areas is established to establish a spatial match between the marking point layer and the piezoelectric layer, thereby obtaining the spatial coordinate information of the sensing area; and in the time dimension, since the time resolution of the piezoelectric digital signal is higher than that of the visual-tactile signal, the piezoelectric digital signal can be aligned with the time starting point of the visual-tactile signal acquisition to synchronize the time series, and the piezoelectric digital signal sampling sequence can be inserted into two adjacent frames to obtain the first The purpose of the piezoelectric signal changes at each marker point is to improve the temporal resolution of each marker point, compensate for the low frame rate of the camera, and obtain the timing signal of the sensing area. Thus, the spatiotemporal tactile signal is obtained based on the spatial coordinate information and the timing signal, wherein the spatiotemporal tactile signal includes a tactile spatial signal and a tactile timing signal.

[0076] For the tactile spatial signal obtained after the above fusion, it is necessary to perform three-dimensional surface reconstruction of the perception area of ​​the tactile spatial signal. Specifically, because the types of the tactile spatial signal include single mirror reflection and multi-mirror reflection, different processing methods are adopted for different mirror reflections. If it is a single mirror reflection, the obtained two-dimensional coordinate on the image plane can be used to complete the three-dimensional surface reconstruction through methods such as coordinate point gradient integration and marker point area scaling. If it is multi-mirror reflection, a virtual binocular vision system can be constructed to directly calculate the three-dimensional coordinates of each position through the virtual binocular vision system, and the surface three-dimensional reconstruction can be completed by combining the interpolation method. For tactile timing signals, the tactile timing signals can be spectrally converted at appropriate time intervals (i.e., target time intervals), for example, FFT (Fast Fourier Transform) can be used to obtain the main vibration frequency information; multimodal tactile information is obtained based on the three-dimensional reconstruction results and the main vibration frequency information. In this way, the coordinates of the tactile points can be expanded from three-dimensional data to four-dimensional data, and the changes in the position and vibration frequency of each tactile point over time can be tracked, thereby giving the bionic finger the ability to recognize and locate different vibration stimuli in different time and space.

[0077] Furthermore, the spatial coordinate information and the timing signal are obtained, and target information within the sensing area can be identified and sensed based on the spatial coordinate information and the timing signal. The target information includes shape information, texture information, static force information, and dynamic force information. Shape information primarily reflects spatial geometric relationships. A three-dimensional reconstruction of the contact surface is performed before and after contact. By comparing the differences in the surface reconstructions before and after contact, shape recognition can be achieved. Texture information, based on the complex spatial layout of the textured surface, can be analyzed using the timing signals generated by the sliding contact. This requires high temporal resolution of the tactile signal, and extracting vibration frequency is crucial for texture analysis. Force measurement is based on the existing spatial geometric relationships and incorporates the material properties of the embedded flexible substrate. This allows mapping changes in geometric space to changes in force space, involving both linear and nonlinear stages. For each static contact moment, a static force is obtained. When dynamic contact occurs, dynamic force is obtained by unfolding the contact surface along the time series. The dynamic force at each contact moment is affected by the previous moment.

[0078] Furthermore, the overall process of the multimodal bionic tactile perception method based on bionic tactile fingers in the present invention is as follows: Figure 3As shown, first, the piezoelectric signal is preprocessed. Specifically, when the piezoelectric sensing module of the bionic tactile finger is stimulated by the external environment to generate a charge signal, the charge signal of the sensing area is obtained, the charge signal is amplified to obtain a target charge signal, and the target charge signal is converted to obtain a voltage signal; the voltage signal is filtered to obtain a target voltage signal, and the target voltage signal is digitally converted to obtain a piezoelectric digital signal.

[0079] Secondly, the visual-tactile signal is preprocessed. When the piezoelectric sensing module is stimulated by the external environment and the camera module of the visual-tactile sensing module of the bionic tactile finger captures an image sequence of the corresponding marking points, an image sequence of the sensing area is obtained, and the image sequence is separated and processed to obtain a piezoelectric film image and a marking point image; the piezoelectric film image is distinguished and processed according to the relative relationship between the area and arrangement of the piezoelectric film to obtain corresponding spatial position information, and first image sequence information is obtained based on the spatial position information and the image sequence; the position information of each marking point is recorded according to the marking point image, and second image sequence information is obtained based on all the position information and the image sequence, and a corresponding visual-tactile signal is constructed based on the first image sequence information and the second image sequence information.

[0080] Subsequently, the first image sequence information and the second image sequence information are compared to obtain the spatial coordinate information of the perception area; the time starting point of the piezoelectric digital signal is aligned with the time starting point of the visual-tactile signal to obtain the timing signal of the perception area; and a spatiotemporal tactile signal is obtained based on the spatial coordinate information and the timing signal.

[0081] Finally, the spatiotemporal tactile signal is subjected to surface three-dimensional reconstruction and spectrum analysis respectively to obtain corresponding spatiotemporal information, namely, a three-dimensional reconstruction result and the vibration main frequency information. Specifically, the tactile space signal of the spatiotemporal tactile signal is subjected to surface three-dimensional reconstruction of the perception area to obtain a three-dimensional reconstruction result; and the tactile timing signal of the spatiotemporal tactile signal is subjected to spectrum conversion according to a set target time interval to obtain vibration main frequency information; and multimodal tactile information is obtained based on the spatiotemporal information.

[0082] Furthermore, if Figure 5 As shown, based on the above-mentioned multimodal bionic tactile perception method based on bionic tactile fingers, the present invention also provides a multimodal bionic tactile perception system based on bionic tactile fingers, and the multimodal bionic tactile perception system based on bionic tactile fingers includes: a bionic tactile finger 51, a signal processing module 52 and a fusion analysis module 53;

[0083] The bionic tactile finger 51 includes a bionic fingerprint module, a piezoelectric sensor module, a visual tactile sensor module and a piezoelectric signal modulation circuit module;

[0084] The bionic fingerprint module is used to come into contact with an object;

[0085] The piezoelectric sensing module is configured to generate a corresponding charge signal when the bionic fingerprint module contacts the object;

[0086] The visual-tactile sensing module is configured to capture a corresponding image sequence when the bionic fingerprint module contacts the object;

[0087] A piezoelectric signal modulation circuit module is used to pre-process the charge signal to obtain a piezoelectric digital signal;

[0088] a signal processing module 52 for performing signal processing on the image sequence to obtain visual and tactile signals;

[0089] The fusion analysis module 53 is used to fuse the piezoelectric digital signal and the visual tactile signal to obtain a spatiotemporal tactile signal, and perform surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signal to obtain multimodal tactile information.

[0090] Furthermore, if Figure 6 As shown, based on the above-mentioned multimodal bionic tactile perception method based on bionic tactile fingers, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0091] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as the terminal's hard drive or memory. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card, etc. equipped on the terminal. Furthermore, the memory 20 may include both the terminal's internal storage unit and an external storage device. The memory 20 is used to store application software installed on the terminal and various types of data, such as program code installed on the terminal. The memory 20 may also be used to temporarily store data that has been output or is about to be output. In one embodiment, the memory 20 stores a multimodal bionic tactile perception program 40 based on a bionic tactile finger. The multimodal bionic tactile perception program 40 based on a bionic tactile finger can be executed by the processor 10, thereby implementing the multimodal bionic tactile perception method based on a bionic tactile finger in this application.

[0092] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the multimodal bionic tactile perception method based on a bionic tactile finger.

[0093] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0094] In one embodiment, when the processor 10 executes the program 40 for multimodal bionic tactile perception based on a bionic tactile finger in the memory 20 , the following steps are implemented:

[0095] Acquiring a charge signal generated by the bionic tactile finger, and preprocessing the charge signal to obtain a piezoelectric digital signal;

[0096] Acquiring an image sequence captured by the bionic tactile finger, and performing signal processing on the image sequence to obtain a visual tactile signal;

[0097] The piezoelectric digital signal and the visual tactile signal are fused to obtain a spatiotemporal tactile signal, and the spatiotemporal tactile signal is subjected to surface three-dimensional reconstruction and spectrum analysis to obtain multimodal tactile information.

[0098] The step of acquiring the charge signal generated by the bionic tactile finger and preprocessing the charge signal to obtain a piezoelectric digital signal specifically includes:

[0099] When it is detected that the bionic tactile finger generates a charge signal due to external stimulation, the charge signal of the sensing area is acquired, the charge signal is amplified to obtain a target charge signal, and the target charge signal is converted to obtain a voltage signal;

[0100] The voltage signal is filtered to obtain a target voltage signal, and the target voltage signal is digitally converted to obtain a piezoelectric digital signal.

[0101] The acquiring of the image sequence captured by the bionic tactile finger and performing signal processing on the image sequence to obtain the visual tactile signal specifically includes:

[0102] When it is detected that the bionic tactile finger is stimulated by an external environment and an image sequence of corresponding marking points is captured, an image sequence of the sensing area is acquired, and the image sequence is separated to obtain a piezoelectric film image and a marking point image;

[0103] Performing differentiation processing on the piezoelectric film image according to the relative relationship between the area and arrangement of the piezoelectric film to obtain corresponding spatial position information, and obtaining first image sequence information according to the spatial position information and the image sequence;

[0104] The position information of each marking point is recorded according to the marking point image, second image sequence information is obtained according to all the position information and the image sequence, and a corresponding visual-tactile signal is constructed according to the first image sequence information and the second image sequence information.

[0105] The fusing of the piezoelectric digital signal and the visual-tactile signal to obtain a spatiotemporal tactile signal specifically includes:

[0106] Comparing the first image sequence information and the second image sequence information to obtain spatial coordinate information of the sensing area;

[0107] Aligning the time starting point of the piezoelectric digital signal with the time starting point of the visual-tactile signal to obtain a timing signal of the perception area;

[0108] A spatiotemporal tactile signal is obtained according to the spatial coordinate information and the time sequence signal.

[0109] Wherein, the spatiotemporal tactile signal includes a tactile spatial signal and a tactile temporal signal;

[0110] The performing surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signals to obtain multimodal tactile information specifically includes:

[0111] Performing three-dimensional surface reconstruction on the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result;

[0112] Setting a target time interval, and performing spectrum conversion on the tactile timing signal according to the target time interval to obtain vibration main frequency information;

[0113] Multimodal tactile information is obtained according to the three-dimensional reconstruction result and the vibration main frequency information.

[0114] Wherein, the types of the tactile spatial signals include single mirror reflection and multi-mirror reflection;

[0115] The performing three-dimensional surface reconstruction of the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result specifically includes:

[0116] If the type of the tactile spatial signal is single mirror reflection, performing three-dimensional reconstruction on the two-dimensional coordinates of the sensing area using a preset method to obtain a three-dimensional reconstruction result, wherein the preset method includes a gradient integration of the coordinate point and a scaling ratio of the area of ​​the marker point;

[0117] If the type of the tactile spatial signal is multi-mirror reflection, a virtual binocular vision system is constructed, and the three-dimensional coordinates of each position in the perception area are calculated by the virtual binocular vision system, and all three-dimensional coordinates are interpolated to obtain a three-dimensional reconstruction result.

[0118] The method of obtaining a spatiotemporal tactile signal according to the spatial coordinate information and the timing signal further includes:

[0119] The target information of the perception area is identified and perceived according to the spatial coordinate information and the timing signal to obtain a perception result, wherein the target information includes shape information, texture information, static force information and dynamic force information.

[0120] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal bionic tactile perception program based on bionic tactile fingers, and when the multimodal bionic tactile perception program based on bionic tactile fingers is executed by a processor, the steps of the multimodal bionic tactile perception method based on bionic tactile fingers as described above are implemented.

[0121] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0122] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When executed, the program can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0123] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal bionic tactile perception method based on bionic tactile fingers, characterized in that: The multimodal bionic tactile perception method based on bionic tactile fingers includes: Acquiring a charge signal generated by the bionic tactile finger, and preprocessing the charge signal to obtain a piezoelectric digital signal; Acquiring an image sequence captured by the bionic tactile finger, and performing signal processing on the image sequence to obtain a visual tactile signal; The piezoelectric digital signal and the visual tactile signal are fused to obtain a spatiotemporal tactile signal, and the spatiotemporal tactile signal is subjected to surface three-dimensional reconstruction and spectrum analysis to obtain multimodal tactile information.

2. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 1, characterized in that: The acquiring of the charge signal generated by the bionic tactile finger and preprocessing the charge signal to obtain a piezoelectric digital signal specifically includes: When it is detected that the bionic tactile finger generates a charge signal due to external stimulation, the charge signal of the sensing area is acquired, the charge signal is amplified to obtain a target charge signal, and the target charge signal is converted to obtain a voltage signal; The voltage signal is filtered to obtain a target voltage signal, and the target voltage signal is digitally converted to obtain a piezoelectric digital signal.

3. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 1, characterized in that: The acquiring of the image sequence captured by the bionic tactile finger and performing signal processing on the image sequence to obtain the visual tactile signal specifically includes: When it is detected that the bionic tactile finger is stimulated by an external environment and an image sequence of corresponding marking points is captured, an image sequence of the sensing area is acquired, and the image sequence is separated to obtain a piezoelectric film image and a marking point image; Performing differentiation processing on the piezoelectric film image according to the relative relationship between the area and arrangement of the piezoelectric film to obtain corresponding spatial position information, and obtaining first image sequence information according to the spatial position information and the image sequence; The position information of each marking point is recorded according to the marking point image, second image sequence information is obtained according to all the position information and the image sequence, and a corresponding visual-tactile signal is constructed according to the first image sequence information and the second image sequence information.

4. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 3, characterized in that: The fusing the piezoelectric digital signal and the visual tactile signal to obtain a spatiotemporal tactile signal specifically includes: Comparing the first image sequence information and the second image sequence information to obtain spatial coordinate information of the sensing area; Aligning the time starting point of the piezoelectric digital signal with the time starting point of the visual-tactile signal to obtain a timing signal of the perception area; A spatiotemporal tactile signal is obtained according to the spatial coordinate information and the time sequence signal.

5. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 1, characterized in that: The spatiotemporal tactile signal includes a tactile spatial signal and a tactile temporal signal; The performing surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signals to obtain multimodal tactile information specifically includes: Performing three-dimensional surface reconstruction on the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result; Setting a target time interval, and performing spectrum conversion on the tactile timing signal according to the target time interval to obtain vibration main frequency information; Multimodal tactile information is obtained according to the three-dimensional reconstruction result and the vibration main frequency information.

6. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 5, characterized in that: The types of the tactile spatial signals include single mirror reflection and multi-mirror reflection; The performing three-dimensional reconstruction on the surface of the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result specifically includes: If the type of the tactile spatial signal is single mirror reflection, performing three-dimensional reconstruction on the two-dimensional coordinates of the sensing area using a preset method to obtain a three-dimensional reconstruction result, wherein the preset method includes a gradient integration of the coordinate point and a scaling ratio of the area of ​​the marker point; If the type of the tactile spatial signal is multi-mirror reflection, a virtual binocular vision system is constructed, and the three-dimensional coordinates of each position in the perception area are calculated by the virtual binocular vision system, and all three-dimensional coordinates are interpolated to obtain a three-dimensional reconstruction result.

7. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 4, characterized in that: The step of obtaining a spatiotemporal tactile signal according to the spatial coordinate information and the timing signal further includes: The target information of the perception area is identified and perceived according to the spatial coordinate information and the timing signal to obtain a perception result, wherein the target information includes shape information, texture information, static force information and dynamic force information.

8. A multimodal bionic tactile perception system based on bionic tactile fingers, characterized in that: The multimodal bionic tactile perception system based on bionic tactile fingers includes: bionic tactile fingers, a signal processing module and a fusion analysis module; The bionic tactile finger includes a bionic fingerprint module, a piezoelectric sensor module, a visual tactile sensor module and a piezoelectric signal modulation circuit module; The bionic fingerprint module is used to come into contact with an object; The piezoelectric sensing module is configured to generate a corresponding charge signal when the bionic fingerprint module contacts the object; The visual-tactile sensing module is configured to capture a corresponding image sequence when the bionic fingerprint module contacts the object; A piezoelectric signal modulation circuit module is used to pre-process the charge signal to obtain a piezoelectric digital signal; a signal processing module, configured to perform signal processing on the image sequence to obtain visual and tactile signals; The fusion analysis module is used to fuse the piezoelectric digital signal and the visual tactile signal to obtain a spatiotemporal tactile signal, and perform surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signal to obtain multimodal tactile information.

9. A terminal, characterized in that: The terminal includes a memory, a processor, and a multimodal bionic tactile perception program based on a bionic tactile finger stored in the memory and runnable on the processor. When the multimodal bionic tactile perception program based on a bionic tactile finger is executed by the processor, the steps of the multimodal bionic tactile perception method based on a bionic tactile finger as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer-readable storage medium stores a multimodal bionic tactile perception program based on bionic tactile fingers. When the multimodal bionic tactile perception program based on bionic tactile fingers is executed by a processor, the steps of the multimodal bionic tactile perception method based on bionic tactile fingers as described in any one of claims 1 to 7 are implemented.

Images