Multi-modal bionic tactile perception method and system based on bionic tactile fingers

Abstract

The invention discloses a multi-mode bionic tactile sensing method and system based on a bionic tactile finger, and the method comprises the steps: obtaining a charge signal generated by the bionic tactile finger, and carrying out the preprocessing of the charge signal, and obtaining a piezoelectric digital signal; acquiring an image sequence shot by the bionic tactile finger, and performing signal processing on the image sequence to obtain a visual tactile signal; and performing fusion processing on the piezoelectric digital signal and the visual tactile signal to obtain a space-time tactile signal, and performing surface three-dimensional reconstruction and spectral analysis on the space-time tactile signal to obtain multi-modal tactile information. On the basis of finger bionic characteristics, a piezoelectric sensor and a visual tactile sensor are integrated, bionic tactile fingers are designed, data fusion is carried out in time and space dimensions, the multi-mode tactile sensing ability is achieved, and through structural bionic and functional bionic, the robot tactile sense obtains the recognition and positioning ability of different vibration stimuli in different time and space.

Description

Technical Field

[0001] The present invention relates to the field of robot tactile perception technology, and in particular to a multimodal bionic tactile perception method, system, terminal and computer-readable storage medium based on bionic tactile fingers. Background Art

[0002] The robot's tactile perception plays a very important role in machine grasping, operation, and human-machine interaction. Based on the guidance of the structure, material properties, and functions of human fingertips, combined with different sensing principles, the design of anthropomorphic tactile fingers can enable the robot to have highly sensitive tactile perception. In the robot's tactile perception process, there are many different attributes, including shape, force, texture, etc. Tactile recognition of multiple attributes plays an important role in machine tactile perception, and it has high requirements for tactile sensing.

[0003] At present, there are many sensing principles that can realize the acquisition of tactile signals, mainly including piezoelectric, optical, etc., and a single type of sensor is limited by its own principle and has limited application scenarios. Based on two or more sensing principles, the advantages of different sensing principles can be brought into play and the advantages of different sensing modalities can be complemented. Compared with other sensor principles, optical sensing has a simple and compact structure, high spatial resolution, high sensitivity to static pressing and low-frequency contact, and can also be used with magnetic resonance imaging technology for tactile sensing of surgical robots. However, the camera-based optical sensor, also called visual tactile sensor, has a time response speed that depends on the camera frame rate, which further limits the size of the sensor, and has low time resolution at the scale of tactile fingers. One of the outstanding advantages of piezoelectric sensors is that they do not require power supply, and have the characteristics of high sensitivity, high reliability, and fast dynamic response. Their wide response range of 1Hz-1kHz can be used for vibration measurement. However, since the voltage will continue to decrease when the static force continues to act, piezoelectric sensors are not suitable for static force measurement. At the same time, they have low spatial resolution and poor temperature stability.

[0004] The current visual-tactile sensing method has the problem that the time response speed is limited by the frame rate. The piezoelectric sensing method has the problems of limited spatial resolution and poor static response. In addition, the existing multimodal tactile sensors lack a data fusion process. For example, the vibration frequency information obtained by the piezoelectric sensor array is not fused with the information obtained by visual-tactile, resulting in the failure to resolve the local area frequency information at a certain spatial resolution.

[0005] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0006] The main purpose of the present invention is to provide a multimodal bionic tactile perception method based on bionic tactile fingers, aiming to solve the problems of the single sensor in the prior art that the time response speed is limited by the frame rate, the spatial resolution is limited, the static response is poor, and the multimodal tactile sensor lacks a data fusion process.

[0007] To achieve the above object, the present invention provides a multimodal bionic tactile perception method and system based on bionic tactile fingers, wherein the multimodal bionic tactile perception method based on bionic tactile fingers comprises the following steps: Acquiring a charge signal generated by the bionic tactile finger, and preprocessing the charge signal to obtain a piezoelectric digital signal; Acquiring an image sequence captured by the bionic tactile finger, and performing signal processing on the image sequence to obtain a visual tactile signal; The piezoelectric digital signal and the visual tactile signal are fused to obtain a spatiotemporal tactile signal, and the spatiotemporal tactile signal is subjected to surface three-dimensional reconstruction and spectrum analysis to obtain multimodal tactile information.

[0008] Optionally, the multimodal bionic tactile perception method based on bionic tactile fingers, wherein the step of acquiring the charge signal generated by the bionic tactile finger and preprocessing the charge signal to obtain the piezoelectric digital signal, specifically includes: When it is detected that the bionic tactile finger generates a charge signal due to external stimulation, the charge signal of the sensing area is acquired, the charge signal is amplified to obtain a target charge signal, and the target charge signal is converted to obtain a voltage signal; The voltage signal is filtered to obtain a target voltage signal, and the target voltage signal is digitally converted to obtain a piezoelectric digital signal.

[0009] Optionally, the multimodal bionic tactile perception method based on bionic tactile fingers, wherein the step of acquiring an image sequence captured by the bionic tactile fingers and performing signal processing on the image sequence to obtain visual tactile signals, specifically comprises: When it is detected that the bionic tactile finger is stimulated by the outside world and an image sequence of corresponding marking points is captured, an image sequence of the sensing area is acquired, and the image sequence is separated to obtain a piezoelectric film image and a marking point image; The piezoelectric film image is distinguished and processed according to the relative relationship between the area and arrangement of the piezoelectric film to obtain corresponding spatial position information, and first image sequence information is obtained according to the spatial position information and the image sequence; The position information of each marking point is recorded according to the marking point image, the second image sequence information is obtained according to all the position information and the image sequence, and the corresponding visual-tactile signal is constructed according to the first image sequence information and the second image sequence information.

[0010] Optionally, the multimodal bionic tactile perception method based on bionic tactile fingers, wherein the piezoelectric digital signal and the visual tactile signal are fused to obtain a spatiotemporal tactile signal, specifically includes: Comparing the first image sequence information with the second image sequence information to obtain spatial coordinate information of the sensing area; Aligning the time starting point of the piezoelectric digital signal with the time starting point of the visual-tactile signal to obtain a timing signal of the perception area; A spatiotemporal tactile signal is obtained according to the spatial coordinate information and the timing signal.

[0011] Optionally, in the multimodal bionic tactile perception method based on bionic tactile fingers, the spatiotemporal tactile signal includes a tactile spatial signal and a tactile temporal sequence signal; The performing surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signals to obtain multimodal tactile information specifically includes: Performing three-dimensional surface reconstruction on the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result; Setting a target time interval, and performing spectrum conversion on the tactile timing signal according to the target time interval to obtain vibration main frequency information; Multimodal tactile information is obtained according to the three-dimensional reconstruction result and the main vibration frequency information.

[0012] Optionally, in the multimodal bionic tactile perception method based on bionic tactile fingers, the types of the tactile spatial signals include single mirror reflection and multi-mirror reflection; The performing three-dimensional reconstruction on the surface of the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result specifically includes: If the type of the tactile spatial signal is single mirror reflection, three-dimensionally reconstructing the two-dimensional coordinates of the sensing area in a preset manner to obtain a three-dimensional reconstruction result, wherein the preset manner includes a coordinate point gradient integral and a marking point area scaling ratio; If the type of the tactile spatial signal is multi-mirror reflection, a virtual binocular vision system is constructed, the three-dimensional coordinates of each position of the perception area are calculated by the virtual binocular vision system, and all the three-dimensional coordinates are interpolated to obtain a three-dimensional reconstruction result.

[0013] Optionally, the multimodal bionic tactile perception method based on bionic tactile fingers, wherein the spatiotemporal tactile signal is obtained according to the spatial coordinate information and the timing signal, and then further comprises: The target information of the perception area is identified and perceived according to the spatial coordinate information and the timing signal to obtain a perception result, wherein the target information includes shape information, texture information, static force information and dynamic force information.

[0014] Optionally, the multimodal bionic tactile perception method based on bionic tactile fingers, wherein the multimodal bionic tactile perception system based on bionic tactile fingers comprises: a bionic tactile finger, a signal processing module and a fusion analysis module; The bionic tactile finger comprises a bionic fingerprint module, a piezoelectric sensor module, a visual tactile sensor module and a piezoelectric signal modulation circuit module; The bionic fingerprint module is used to come into contact with an object; The piezoelectric sensor module is used to generate a corresponding charge signal when the bionic fingerprint module contacts the object; The visual-tactile sensing module is used to capture a corresponding image sequence when the bionic fingerprint module contacts the object; A piezoelectric signal modulation circuit module is used to pre-process the charge signal to obtain a piezoelectric digital signal; A signal processing module, used for performing signal processing on the image sequence to obtain visual and tactile signals; The fusion analysis module is used to fuse the piezoelectric digital signal and the visual tactile signal to obtain a spatiotemporal tactile signal, and to perform surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signal to obtain multimodal tactile information.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multimodal bionic tactile perception program based on a bionic tactile finger stored in the memory and executable on the processor, wherein the multimodal bionic tactile perception program based on a bionic tactile finger implements the steps of the multimodal bionic tactile perception method based on a bionic tactile finger as described above when the multimodal bionic tactile perception program based on a bionic tactile finger is executed by the processor.

[0016] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal bionic tactile perception program based on bionic tactile fingers, and when the multimodal bionic tactile perception program based on bionic tactile fingers is executed by a processor, the steps of the multimodal bionic tactile perception method based on bionic tactile fingers as described above are implemented.

[0017] In the present invention, the charge signal generated by the bionic tactile finger is obtained, and the charge signal is preprocessed to obtain a piezoelectric digital signal; the image sequence captured by the bionic tactile finger is obtained, and the image sequence is signal processed to obtain a visual tactile signal; the piezoelectric digital signal and the visual tactile signal are fused to obtain a spatiotemporal tactile signal, and the spatiotemporal tactile signal is respectively subjected to surface three-dimensional reconstruction and spectrum analysis to obtain multimodal tactile information. Based on the bionic features of fingers, the present invention integrates piezoelectric sensors and visual tactile sensors, designs bionic tactile fingers, performs data fusion in time and space dimensions, and realizes multimodal tactile perception capabilities. Through structural bionics and functional bionics, the robot tactile sense has good comprehensive performance in terms of spatial sensitivity, time sensitivity, static force and dynamic force perception, so that the robot tactile sense can obtain the recognition and positioning capabilities of different vibration stimuli in different time and space. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a preferred embodiment of the multi-modal bionic tactile perception method based on bionic tactile fingers in the present invention; Figure 2 It is an overall flow chart of the multimodal bionic tactile perception method based on bionic tactile fingers in the present invention; Figure 3 It is a schematic diagram of the structural design of the bionic tactile finger in the present invention; Figure 4 It is a schematic diagram of the overall process of a preferred embodiment of the multi-modal bionic tactile perception method based on bionic tactile fingers in the present invention; Figure 5 It is a structural diagram of a preferred embodiment of a multi-modal bionic tactile perception system based on a bionic tactile finger in the present invention; Figure 6 It is a schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0021] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0022] The multimodal bionic tactile perception method based on bionic tactile fingers described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the multimodal bionic tactile perception method based on bionic tactile fingers includes the following steps: Step S10: acquiring a charge signal generated by the bionic tactile finger, and preprocessing the charge signal to obtain a piezoelectric digital signal.

[0023] The step S10 comprises: Step S11, when it is detected that the bionic tactile finger generates a charge signal due to external stimulation, the charge signal of the sensing area is acquired, the charge signal is amplified to obtain a target charge signal, and the target charge signal is converted to obtain a voltage signal; Step S12: filtering the voltage signal to obtain a target voltage signal, and digitally converting the target voltage signal to obtain a piezoelectric digital signal.

[0024] Specifically, in an embodiment of the present invention, a multimodal bionic tactile perception method based on a bionic tactile finger is proposed, such as Figure 2 As shown in the figure, the method is based on the bionic features of fingertips, combined with piezoelectric sensing and visual tactile sensing, to fuse the electrical signal and the image signal, to achieve bionic tactile perception, and to obtain multimodal tactile information such as static force, dynamic force, shape, texture, etc. In order to better achieve bionic tactile perception, the structure of the bionic tactile finger is designed, as shown in the figure. Figure 3As shown, the bionic tactile finger includes a piezoelectric signal modulation circuit module 1, a piezoelectric sensor module 6, a bionic fingerprint module 7 and a visual tactile sensor module, and the visual tactile sensor module includes a light source module 2, a camera module 3, an optical path reflection group 4 and a marking point layer 5. Among them, the piezoelectric signal modulation circuit module 1 needs to capture weak charges in a timely manner and amplify them into voltage output, filter the signal of the interference frequency band, and then convert the corresponding analog signal into a digital signal, so as to facilitate the subsequent analysis and use of the signal. The piezoelectric sensor module 6 can realize the bionics of FA type receptors (Fast-adapting receptors). The piezoelectric sensor module 6 uses PVDF (Polyvinylidene fluoride) film as a sensing element to simulate the FA receptor in human skin. The frequency sensing range is greater than or equal to 1kHz, covering the sensing threshold of human skin. The piezoelectric film (i.e., PVDF film) adopts a long arc shape and conforms to the annular structure of the fingerprint. Because the normal direction of the fingerprint ridge is more sensitive to sliding stimulation than the tangential direction, the piezoelectric film conforms to the annular arrangement design of the fingerprint, which is conducive to increasing the sensitivity to sliding stimulation in all directions; in the direction perpendicular to the contact surface, the piezoelectric film is close to the contact surface, which is consistent with the relative position of FA-I in human skin, and the piezoelectric film is located in a transparent flexible material substrate (for example, silicone, etc.), and the flexible layer containing the piezoelectric film is covered on the fingerprint flexible layer. The bionic fingerprint on the bionic fingerprint module 7 is a multi-circle curved ridge structure with a wavy papillary cross-section. The purpose of setting is because the wavy papillary cross-section of the fingerprint has the function of amplifying the contact signal, and the corresponding fingerprint structure is attached to the outermost flexible material (for example, silicone, etc.), which can be prepared by various methods such as mold molding, and the fingerprint flexible layer is a dome-shaped arc surface, which is similar to the outer contour of the human fingertips.

[0025] The visual-tactile sensing module uses a camera to capture the changes of the marker point under the influence of the external environment, so as to calculate the magnitude of the force (for example, torque), obtain texture information or judge slippage, etc.; since the frame rate of commonly used cameras is limited, the visual-tactile sensing design is used to simulate the SA receptors (Slow-adapting receptors) in the human skin to capture low-frequency, static external stimuli. Specifically, the light source module 2 is used to provide a uniform light source to the marker point. In an environment with poor external lighting, the effective measurement of the visual-tactile sensing module can be guaranteed, and the number, color and position of the light group can be configured according to the specific algorithm and size, which are not limited here; the camera module 3 is used to record the changes of the marker point, and the axis of the camera lens is at a certain angle to the contact surface tangent plane. The size of the angle is determined according to the design and simulation of the optical path; the optical path reflection group 4 (for example, a mirror) can play a role in changing the optical path. In order to make the shape of the bionic tactile finger more similar to the shape of a real human finger, The dimension perpendicular to the contact surface is compressed in one step, and a mirror can be placed above the marking point. The position of the mirror is calculated and simulated, and the light reflected by the marking point can be effectively projected into the camera. On the other hand, the mirror can imitate the shape of the finger nail, which increases the rigidity of the bionic finger while making the appearance more similar to the real finger. The marking points in the marking point layer 5 can be designed into a variety of shapes, such as circle, square, etc.; and the interval between the marking points is less than or equal to 1 mm, which is consistent with the spatial resolution of human fingertips. The marking points can be located on the surface of a transparent flexible substrate (such as silicone, etc.) or dispersed therein, and the flexible layer containing the marking points is covered on the flexible layer containing the piezoelectric film.

[0026] The present invention adopts the design of bionic tactile fingers, including using wavy papillary annular protrusions to simulate fingerprints, the flexible layer where the fingerprint is located is a dome surface to simulate the fingertips; using piezoelectric sensors to simulate the FA type sensors of human skin for high-frequency stimulation perception, and arranging them in an annular pattern in imitation of fingerprints; using visual touch to simulate the SA type sensors of human skin for low-frequency and static external force perception; using mirrors to simulate the shape of nails to improve bionic similarity and rigidity. Through the reasonable layout of mirrors and cameras, the distance in the direction perpendicular to the finger contact surface is reduced by reflection, realizing a miniaturized design.

[0027] After completing the design of the bionic tactile finger, bionic tactile perception is achieved through the bionic tactile finger. Specifically, when the piezoelectric sensing module of the bionic tactile finger is stimulated by the outside world, an electric charge is generated on the piezoelectric film. At the same time, the charge is only generated under dynamic stimulation, which is very similar to a pulse signal. Then, a charge signal of the corresponding sensing area is obtained, and the charge signal is amplified by the piezoelectric signal modulation circuit module to obtain a target charge signal, and the target charge signal is converted to obtain a voltage signal. Thereafter, the voltage signal is filtered to obtain a target voltage signal in order to remove signals in the interfering frequency band, and the target voltage signal is digitally converted to obtain a piezoelectric digital signal, wherein the output of each piezoelectric film corresponds to a digital signal channel output , Indicates Piezoelectric film over time Changing piezoelectric digital signal.

[0028] Step S20: acquiring an image sequence captured by the bionic tactile finger, and performing signal processing on the image sequence to obtain a visual tactile signal.

[0029] The step S20 comprises: Step S21, when it is detected that the bionic tactile finger is stimulated by the outside world and an image sequence of corresponding marking points is captured, an image sequence of the sensing area is acquired, and the image sequence is separated to obtain a piezoelectric film image and a marking point image; Step S22, distinguishing and processing the piezoelectric film image according to the relative relationship between the area and arrangement of the piezoelectric film to obtain corresponding spatial position information, and obtaining first image sequence information according to the spatial position information and the image sequence; Step S23, recording the position information of each marking point according to the marking point image, obtaining second image sequence information according to all the position information and the image sequence, and constructing a corresponding visual-tactile signal according to the first image sequence information and the second image sequence information.

[0030] Specifically, in an embodiment of the present invention, when the piezoelectric sensing module of the bionic tactile finger is stimulated by the outside world and the visual tactile sensing module captures an image sequence of corresponding marking points, the image sequence needs to be preprocessed and separated to obtain the corresponding piezoelectric film image and marking point image. Specifically, since the marking points and piezoelectric films of the image sequence have different colors and are located in transparent substrates at different depth layers, the camera module can capture the images of the marking points and the piezoelectric film at the same time; for each frame of the image, the piezoelectric film image and the marking point image can be separated according to the differences in information such as the color and depth of the piezoelectric film and the marking point; thereafter, for a single frame of the piezoelectric film image, according to the consistency of the relative relationship between the area and arrangement of each piece of piezoelectric film, it can be distinguished which piezoelectric film the relevant pixel points in the piezoelectric film image belong to, thereby obtaining the spatial position of the piezoelectric film, and according to the image sequence, the image sequence information of the piezoelectric film is comprehensively obtained, that is, the first image sequence information , Indicates Piezoelectric film over time For a single-frame marker image, the position information of each marker is recorded, and the position of each marker is tracked according to the image sequence to obtain the image sequence information of the marker, i.e., the second image sequence information. , Indicates Marked points over time Then, construct a corresponding visual-tactile signal according to the first image sequence information and the second image sequence information.

[0031] Step S30: fusing the piezoelectric digital signal and the visual tactile signal to obtain a spatiotemporal tactile signal, and performing surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signal to obtain multimodal tactile information.

[0032] The step S30 comprises: Step S31, comparing the first image sequence information and the second image sequence information to obtain spatial coordinate information of the sensing area; Step S32, aligning the time starting point of the piezoelectric digital signal with the time starting point of the visual-tactile signal to obtain a timing signal of the perception area; Step S33, performing three-dimensional surface reconstruction on the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result; Step S34, setting a target time interval, and performing spectrum conversion on the tactile timing signal according to the target time interval to obtain vibration main frequency information; Step S35 , obtaining multimodal tactile information according to the three-dimensional reconstruction result and the vibration main frequency information.

[0033] Specifically, after obtaining the piezoelectric digital signal and the visual tactile signal, the piezoelectric digital signal and the visual tactile signal need to be fused. Specifically, in an embodiment of the present invention, the piezoelectric digital signal and the visual tactile signal can be fused in two dimensions of time and space. In the spatial dimension, there are two relationships between the piezoelectric film and the marking point: inclusion and separation. and , we can get the first The marking point and The relationship between the piezoelectric film areas is established to establish a spatial match between the marking point layer and the piezoelectric layer, thereby obtaining the spatial coordinate information of the sensing area; and in the time dimension, since the time resolution of the piezoelectric digital signal is higher than that of the visual-tactile signal, the piezoelectric digital signal can be aligned with the time starting point of the visual-tactile signal acquisition, and the time series can be synchronized, and the piezoelectric digital signal sampling sequence can be inserted into two adjacent frames, thereby obtaining the first The purpose is to improve the time resolution of each marking point, make up for the defect of low frame rate of the camera, and obtain the timing signal of the sensing area. Thus, the spatiotemporal tactile signal is obtained according to the spatial coordinate information and the timing signal, wherein the spatiotemporal tactile signal includes a tactile space signal and a tactile timing signal.

[0034] For the tactile space signal obtained after the above fusion, it is necessary to perform surface three-dimensional reconstruction of the perception area of ​​the tactile space signal. Specifically, because the types of the tactile space signal include single mirror reflection and multi-mirror reflection, different processing methods are adopted for different mirror reflections. If it is a single mirror reflection, a two-dimensional coordinate on the image plane is obtained, and the surface three-dimensional reconstruction can be completed by means of coordinate point gradient integration and marking point area scaling ratio; if it is multi-mirror reflection, a virtual binocular vision system can be constructed, and the three-dimensional coordinates of each position are directly calculated by the virtual binocular vision system, and the surface three-dimensional reconstruction is completed by combining the interpolation method. For the tactile timing signal, the tactile timing signal can be converted to a spectrum at an appropriate time interval (i.e., the target time interval), for example, FFT transformation (Fast Fourier Transform) can be used to obtain the main frequency information of vibration; multimodal tactile information is obtained based on the three-dimensional reconstruction result and the main frequency information of vibration. In this way, the coordinates of the tactile points can be expanded from three-dimensional data to four-dimensional data, and the changes in the position and vibration frequency of each tactile point over time can be tracked, thereby giving the bionic finger the ability to recognize and locate different vibration stimuli in different time and space.

[0035] Furthermore, the spatial coordinate information and the timing signal are obtained, and the target information of the sensing area can be identified and sensed according to the spatial coordinate information and the timing signal, wherein the target information includes shape information, texture information, static force information and dynamic force information. For shape information, it is mainly reflected in the spatial geometric relationship. Before and after contact, the surface three-dimensional reconstruction of the contact surface is completed. By comparing the surface reconstruction difference before and after contact, the shape can be recognized. For texture information, the complex spatial layout of the texture surface is analyzed by the timing signal generated by the sliding contact, which requires the tactile signal to have a high time resolution. Among them, the extraction of vibration frequency is crucial for analyzing texture. For force measurement, on the basis of the existing spatial geometric relationship, the material properties of the embedded layer flexible substrate are added, and the changes in the geometric space can be mapped to the changes in the force space, involving linear and nonlinear change stages; for each static contact moment, the static force can be obtained; when dynamic contact occurs, the dynamic force can be obtained along the time series. At this time, the dynamic force at each contact moment will be affected by the previous moment.

[0036] Furthermore, the overall process of the multimodal bionic tactile perception method based on bionic tactile fingers in the present invention is as follows: Figure 3 As shown, first, the piezoelectric signal is preprocessed. Specifically, when the piezoelectric sensing module of the bionic tactile finger is stimulated by the external environment to generate a charge signal, the charge signal of the sensing area is acquired, the charge signal is amplified to obtain a target charge signal, and the target charge signal is converted to obtain a voltage signal; the voltage signal is filtered to obtain a target voltage signal, and the target voltage signal is digitally converted to obtain a piezoelectric digital signal.

[0037] Secondly, the visual-tactile signal is preprocessed. When the piezoelectric sensing module is stimulated by external stimuli and the camera module of the visual-tactile sensing module of the bionic tactile finger captures an image sequence of corresponding marking points, an image sequence of the sensing area is acquired, and the image sequence is separated to obtain a piezoelectric film image and a marking point image; the piezoelectric film image is distinguished according to the relative relationship between the area and arrangement of the piezoelectric film to obtain corresponding spatial position information, and first image sequence information is obtained according to the spatial position information and the image sequence; the position information of each marking point is recorded according to the marking point image, and second image sequence information is obtained according to all the position information and the image sequence, and a corresponding visual-tactile signal is constructed according to the first image sequence information and the second image sequence information.

[0038] Subsequently, the first image sequence information and the second image sequence information are compared to obtain the spatial coordinate information of the perception area; the time starting point of the piezoelectric digital signal is aligned with the time starting point of the visual-tactile signal to obtain the timing signal of the perception area; and the spatiotemporal tactile signal is obtained based on the spatial coordinate information and the timing signal.

[0039] Finally, the spatiotemporal tactile signal is subjected to surface three-dimensional reconstruction and spectrum analysis respectively to obtain corresponding spatiotemporal information, namely, a three-dimensional reconstruction result and the vibration main frequency information. Specifically, the tactile space signal of the spatiotemporal tactile signal performs surface three-dimensional reconstruction on the perception area to obtain a three-dimensional reconstruction result; and the tactile timing signal of the spatiotemporal tactile signal is subjected to spectrum conversion according to a set target time interval to obtain vibration main frequency information; and multimodal tactile information is obtained based on the spatiotemporal information.

[0040] Furthermore, if Figure 5 As shown, based on the above-mentioned multimodal bionic tactile perception method based on bionic tactile fingers, the present invention also provides a multimodal bionic tactile perception system based on bionic tactile fingers, and the multimodal bionic tactile perception system based on bionic tactile fingers includes: a bionic tactile finger 51, a signal processing module 52 and a fusion analysis module 53; The bionic tactile finger 51 includes a bionic fingerprint module, a piezoelectric sensor module, a visual tactile sensor module and a piezoelectric signal modulation circuit module; The bionic fingerprint module is used to come into contact with an object; The piezoelectric sensor module is used to generate a corresponding charge signal when the bionic fingerprint module contacts the object; The visual-tactile sensing module is used to capture a corresponding image sequence when the bionic fingerprint module contacts the object; A piezoelectric signal modulation circuit module is used to pre-process the charge signal to obtain a piezoelectric digital signal; A signal processing module 52, used for performing signal processing on the image sequence to obtain visual tactile signals; The fusion analysis module 53 is used to fuse the piezoelectric digital signal and the visual tactile signal to obtain a spatiotemporal tactile signal, and to perform surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signal to obtain multimodal tactile information.

[0041] Furthermore, if Figure 6 As shown, based on the above-mentioned multimodal bionic tactile perception method based on bionic tactile fingers, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 6Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0042] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a multimodal bionic tactile perception program 40 based on a bionic tactile finger is stored on the memory 20, and the multimodal bionic tactile perception program 40 based on a bionic tactile finger can be executed by the processor 10, thereby realizing the multimodal bionic tactile perception method based on a bionic tactile finger in the present application.

[0043] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the multimodal bionic tactile perception method based on the bionic tactile finger.

[0044] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0045] In one embodiment, when the processor 10 executes the program 40 for multimodal bionic tactile perception based on a bionic tactile finger in the memory 20, the following steps are implemented: Acquiring a charge signal generated by the bionic tactile finger, and preprocessing the charge signal to obtain a piezoelectric digital signal; Acquiring an image sequence captured by the bionic tactile finger, and performing signal processing on the image sequence to obtain a visual tactile signal; The piezoelectric digital signal and the visual tactile signal are fused to obtain a spatiotemporal tactile signal, and the spatiotemporal tactile signal is subjected to surface three-dimensional reconstruction and spectrum analysis to obtain multimodal tactile information.

[0046] The step of acquiring the charge signal generated by the bionic tactile finger and preprocessing the charge signal to obtain a piezoelectric digital signal specifically includes: When it is detected that the bionic tactile finger generates a charge signal due to external stimulation, the charge signal of the sensing area is acquired, the charge signal is amplified to obtain a target charge signal, and the target charge signal is converted to obtain a voltage signal; The voltage signal is filtered to obtain a target voltage signal, and the target voltage signal is digitally converted to obtain a piezoelectric digital signal.

[0047] The step of acquiring the image sequence captured by the bionic tactile finger and performing signal processing on the image sequence to obtain the visual tactile signal specifically includes: When it is detected that the bionic tactile finger is stimulated by the outside world and an image sequence of corresponding marking points is captured, an image sequence of the sensing area is acquired, and the image sequence is separated to obtain a piezoelectric film image and a marking point image; The piezoelectric film image is distinguished and processed according to the relative relationship between the area and arrangement of the piezoelectric film to obtain corresponding spatial position information, and first image sequence information is obtained according to the spatial position information and the image sequence; The position information of each marking point is recorded according to the marking point image, the second image sequence information is obtained according to all the position information and the image sequence, and the corresponding visual-tactile signal is constructed according to the first image sequence information and the second image sequence information.

[0048] The fusing process of the piezoelectric digital signal and the visual tactile signal to obtain the spatiotemporal tactile signal specifically includes: Comparing the first image sequence information with the second image sequence information to obtain spatial coordinate information of the sensing area; Aligning the time starting point of the piezoelectric digital signal with the time starting point of the visual-tactile signal to obtain a timing signal of the perception area; A spatiotemporal tactile signal is obtained according to the spatial coordinate information and the timing signal.

[0049] Wherein, the spatiotemporal tactile signal includes a tactile spatial signal and a tactile temporal signal; The performing surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signals to obtain multimodal tactile information specifically includes: Performing three-dimensional surface reconstruction on the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result; Setting a target time interval, and performing spectrum conversion on the tactile timing signal according to the target time interval to obtain vibration main frequency information; Multimodal tactile information is obtained according to the three-dimensional reconstruction result and the main vibration frequency information.

[0050] Wherein, the types of the tactile spatial signals include single mirror reflection and multi-mirror reflection; The performing three-dimensional reconstruction on the surface of the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result specifically includes: If the type of the tactile spatial signal is single mirror reflection, three-dimensionally reconstructing the two-dimensional coordinates of the sensing area in a preset manner to obtain a three-dimensional reconstruction result, wherein the preset manner includes a coordinate point gradient integral and a marking point area scaling ratio; If the type of the tactile spatial signal is multi-mirror reflection, a virtual binocular vision system is constructed, the three-dimensional coordinates of each position of the perception area are calculated by the virtual binocular vision system, and all the three-dimensional coordinates are interpolated to obtain a three-dimensional reconstruction result.

[0051] The step of obtaining a spatiotemporal tactile signal according to the spatial coordinate information and the timing signal further includes: The target information of the perception area is identified and perceived according to the spatial coordinate information and the timing signal to obtain a perception result, wherein the target information includes shape information, texture information, static force information and dynamic force information.

[0052] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal bionic tactile perception program based on bionic tactile fingers, and when the multimodal bionic tactile perception program based on bionic tactile fingers is executed by a processor, the steps of the multimodal bionic tactile perception method based on bionic tactile fingers as described above are implemented.

[0053] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0054] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0055] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal bionic tactile perception method based on bionic tactile fingers, characterized in that: The multimodal bionic tactile perception method based on bionic tactile fingers comprises: Acquiring a charge signal generated by the bionic tactile finger, and preprocessing the charge signal to obtain a piezoelectric digital signal; Acquiring an image sequence captured by the bionic tactile finger, and performing signal processing on the image sequence to obtain a visual tactile signal; The piezoelectric digital signal and the visual tactile signal are fused to obtain a spatiotemporal tactile signal, and surface three-dimensional reconstruction and spectrum analysis are performed on the spatiotemporal tactile signal to obtain multimodal tactile information.

2. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 1 is characterized in that: The acquiring of the charge signal generated by the bionic tactile finger and preprocessing of the charge signal to obtain a piezoelectric digital signal specifically includes: When it is detected that the bionic tactile finger generates a charge signal due to external stimulation, the charge signal of the sensing area is acquired, the charge signal is amplified to obtain a target charge signal, and the target charge signal is converted to obtain a voltage signal; The voltage signal is filtered to obtain a target voltage signal, and the target voltage signal is digitally converted to obtain a piezoelectric digital signal.

3. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 1 is characterized in that: The acquiring of the image sequence captured by the bionic tactile finger and performing signal processing on the image sequence to obtain the visual tactile signal specifically includes: When it is detected that the bionic tactile finger is stimulated by the outside world and an image sequence of corresponding marking points is captured, an image sequence of the sensing area is acquired, and the image sequence is separated to obtain a piezoelectric film image and a marking point image; The piezoelectric film image is distinguished and processed according to the relative relationship between the area and arrangement of the piezoelectric film to obtain corresponding spatial position information, and first image sequence information is obtained according to the spatial position information and the image sequence; The position information of each marking point is recorded according to the marking point image, the second image sequence information is obtained according to all the position information and the image sequence, and the corresponding visual-tactile signal is constructed according to the first image sequence information and the second image sequence information.

4. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 3 is characterized in that: The fusing process of the piezoelectric digital signal and the visual tactile signal to obtain the spatiotemporal tactile signal specifically includes: Comparing the first image sequence information with the second image sequence information to obtain spatial coordinate information of the sensing area; Aligning the time starting point of the piezoelectric digital signal with the time starting point of the visual-tactile signal to obtain a timing signal of the perception area; A spatiotemporal tactile signal is obtained according to the spatial coordinate information and the timing signal.

5. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 1, characterized in that: The spatiotemporal tactile signal includes a tactile spatial signal and a tactile temporal signal; The performing surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signals to obtain multimodal tactile information specifically includes: Performing three-dimensional surface reconstruction on the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result; Setting a target time interval, and performing spectrum conversion on the tactile timing signal according to the target time interval to obtain vibration main frequency information; Multimodal tactile information is obtained according to the three-dimensional reconstruction result and the main vibration frequency information.

6. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 5, characterized in that: The types of the tactile spatial signals include single mirror reflection and multi-mirror reflection; The performing three-dimensional reconstruction on the surface of the sensing area of ​​the tactile spatial signal to obtain a three-dimensional reconstruction result specifically includes: If the type of the tactile spatial signal is single mirror reflection, three-dimensionally reconstructing the two-dimensional coordinates of the sensing area in a preset manner to obtain a three-dimensional reconstruction result, wherein the preset manner includes a coordinate point gradient integral and a marking point area scaling ratio; If the type of the tactile spatial signal is multi-mirror reflection, a virtual binocular vision system is constructed, the three-dimensional coordinates of each position of the perception area are calculated by the virtual binocular vision system, and all the three-dimensional coordinates are interpolated to obtain a three-dimensional reconstruction result.

7. The multimodal bionic tactile perception method based on bionic tactile fingers according to claim 4 is characterized in that: The step of obtaining a spatiotemporal tactile signal according to the spatial coordinate information and the timing signal further comprises: The target information of the perception area is identified and perceived according to the spatial coordinate information and the timing signal to obtain a perception result, wherein the target information includes shape information, texture information, static force information and dynamic force information.

8. A multimodal bionic tactile perception system based on bionic tactile fingers, characterized in that: The multimodal bionic tactile perception system based on bionic tactile fingers comprises: bionic tactile fingers, a signal processing module and a fusion analysis module; The bionic tactile finger comprises a bionic fingerprint module, a piezoelectric sensor module, a visual tactile sensor module and a piezoelectric signal modulation circuit module; The bionic fingerprint module is used to come into contact with an object; The piezoelectric sensor module is used to generate a corresponding charge signal when the bionic fingerprint module contacts the object; The visual-tactile sensing module is used to capture a corresponding image sequence when the bionic fingerprint module contacts the object; A piezoelectric signal modulation circuit module is used to pre-process the charge signal to obtain a piezoelectric digital signal; A signal processing module, used for performing signal processing on the image sequence to obtain visual and tactile signals; The fusion analysis module is used to fuse the piezoelectric digital signal and the visual tactile signal to obtain a spatiotemporal tactile signal, and to perform surface three-dimensional reconstruction and spectrum analysis on the spatiotemporal tactile signal to obtain multimodal tactile information.

9. A terminal, characterized in that: The terminal includes a memory, a processor, and a multimodal bionic tactile perception program based on a bionic tactile finger stored in the memory and executable on the processor. When the multimodal bionic tactile perception program based on a bionic tactile finger is executed by the processor, the steps of the multimodal bionic tactile perception method based on a bionic tactile finger as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein the computer-readable storage medium stores a multimodal bionic tactile perception program based on a bionic tactile finger, and when the multimodal bionic tactile perception program based on a bionic tactile finger is executed by a processor, the steps of the multimodal bionic tactile perception method based on a bionic tactile finger as described in any one of claims 1-7 are implemented.

Images