Haptic Reproduction Method for Hardness Based on the Bionic Characteristics of Tactile Sensation

Abstract

The present invention discloses a hardness tactile reproduction method based on the bionic characteristics of tactile perception. This method reproduces the subjective feeling of a person perceiving the surface hardness of a real object. First, the interaction data when a human hand touches a real object is collected and input into a force tactile perception model to calculate the intensity of the hardness perceived subjectively by the person. Then, the subjectively perceived hardness is input into a hardness reproduction control model to calculate the control parameters of the hardness reproduction device, and a fused mechanical stimulus is generated through a vibration element and a force feedback device to feedback the reproduced hardness tactile feeling to the user. Compared with the traditional force tactile reproduction method based on "stimulus consistency", this method does not directly reproduce the force stimulus during real interaction to the user. Instead, it first calculates the hardness feeling generated by the real stimulus, and then the force tactile control model controls the reproduction device to generate a force tactile stimulus consistent with the real interaction hardness tactile feeling, so as to achieve a true expression of the hardness touch of a non-uniform hardness object.

Description

Technical Field

[0001] The present invention belongs to the technical field of force tactile reproduction, and in particular relates to a hardness tactile reproduction method based on the bionic characteristics of tactile perception. Background Art

[0002] The force sense and touch sense generated during the interaction between people and the environment are collectively referred to as force touch. Force touch reproduction is to generate physical stimulation acting on the user when contacting the surface of a virtual object through a specific hardware device, thereby simulating the surface texture characteristics of the real object such as roughness, hardness and softness. The present invention performs tactile reproduction for the physical attribute of hardness, specifically when a person contacts a virtual object through a force touch reproduction device, a physical stimulus that can simulate the softness and hardness of the object perceived by a person when touching a real object is generated. Hardness reproduction can make up for the lack of tactile information in human-computer interaction, especially for objects with non-uniform elasticity (human tissue, tumors, fabrics, etc.), reproduction can enhance people's overall cognition of objects. When performing virtual surgery or remote surgery, doctors can obtain the touch of human tissue through the hardness reproduction system, thereby improving the stability of palpation and reducing tissue damage caused by surgical misoperation. In addition, this method can also be applied to scenarios such as teleoperated robots, VR games, and virtual assembly.

[0003] The hardness reproduction process generally includes two parts: the establishment of a hardness reproduction control algorithm and the expression of a force tactile device. Usually, the hardness reproduction algorithm is based on the principle that artificial stimulation is consistent with real stimulation. It directly measures the force tactile stimulation data during real interaction, and then controls the reproduction device to produce artificial stimulation that is consistent with the real stimulation intensity. At the same time, there are many types of force tactile expression devices. From the perspective of stimulation form, they can be divided into force feedback, vibration stimulation, air pressure, etc., and from the perspective of structure, they can be divided into array, pen, embedded, wearable, etc. In the past, in the realization of stimulation form, hardness reproduction often adopted a single artificial stimulation mode. Summary of the invention

[0004] In order to enhance human's perception of the tactile property of object hardness in human-computer interaction, the present invention proposes a tactile hardness reproduction method based on the bionic characteristics of tactile perception.

[0005] The present invention adopts a force / vibration mixed hardness reproduction method based on perceptual consistency under real-time measurement when reproducing hardness. Its notable feature is that it combines a subjective hardness perception model of a person to provide a reproduction stimulus consistent with the subjective hardness perception result. Psychophysical research has found that the intensity of a person's subjective perception stimulus is inconsistent with the objective stimulus, and there is a logarithmic relationship (Fechner's theorem). In addition, there are also complex interactions in the objective force tactile stimulus that affects perception. Taking hardness perception as an example, people's perception of hardness is not only affected by objective factors such as the material properties and material density of the object, but also related to the conditions of tactile interaction, such as the area and duration of tactile stimulation, and the mutual influence of different stimuli acting simultaneously. Therefore, the perception model describing the relationship between subjective tactile perception and objective stimulation is a complex nonlinear multi-input / multi-output relationship. Based on the nonlinear characteristics of tactile perception, in order to achieve true tactile reproduction, the perception model can be "inverse transformed" to seek the optimal inverse solution of artificial stimulation under sensory equivalence. Instead of simply "playing back" the interaction data collected on the surface of the object.

[0006] The present invention proposes a method for tactile reproduction of hardness based on the bionic characteristics of tactile perception, which is based on a relationship model between objective hardness stimulation and subjective perception intensity. According to the correlation between hardness perception and steady-state force stimulation and transient vibration stimulation, a hardness reproduction control model is established, and reproduction is performed in the form of feedback combining force and vibration, so as to overcome the shortcomings of the current force tactile reproduction methods for hardness, such as poor consistency between the reproduced tactile effect and the subjective feeling, and insufficient reproduction realism.

[0007] The present invention proposes a method for reproducing hardness tactile sensation based on the bionic characteristics of tactile perception, which comprises the following main steps:

[0008] Step 1: Collection of interaction data: The remote force tactile sensor measuring device presses the object under the control of the host operator. During the pressing process, the force tactile sensor collects force tactile interaction data, including force, displacement, acceleration, etc., and calculates and extracts the force tactile stimulation characteristic parameters in the hardness perception process.

[0009] Step 2: Obtaining subjective perception intensity: Substitute the characteristic parameters extracted in step 1 into the known objective stimulus characteristic parameters -> the perception model of subjective hardness perception intensity to obtain the subjective hardness perception intensity of the current perceived real object.

[0010] Step 3: Determination of control parameters of the reproduction device: Substitute the subjective hardness perception intensity obtained in step 2 into the force tactile control model to calculate the control parameters of the force tactile reproduction device, including force feedback control parameters and vibration feedback control parameters.

[0011] Step 4: Hardness reproduction: According to the control parameters obtained in step 3, the force feedback and vibration feedback actuators in the reproduction device generate a fused force / vibration tactile stimulation to the main end operator, thereby achieving hardness reproduction of the remote real object.

[0012] Furthermore, the probability distribution function of the peak force and the displacement corresponding to the pressure f applied by the user are calculated based on the displacement x and pressure f collected in step 1, and then substituted into the following two equations to calculate the global stiffness coefficient (GS) and peak force / displacement (PFP) characteristics of the sample.

[0013]

[0014]

[0015] Among them, P i (f) is the probability distribution function of the peak force, is the displacement corresponding to the force f applied by the user.

[0016] Furthermore, the hardness tactile perception model in step 2 is a hardness perception regression relationship model proposed by this laboratory. In this regression relationship model, the hardness perception intensity is related to the logarithm of the sample global stiffness coefficient (GS) and the logarithm of the peak force / displacement (PFP). The specific relationship expression is as follows:

[0017] H=k 1 logGS-k 2 ·logPFP

[0018] Among them, GS is the global stiffness (N / mm), which refers to the average value of the proportionality coefficient of pressure and displacement, that is, the force required to cause unit displacement, which is a characteristic of the overall hardness of the object. PFP is the peak force / displacement (N / mm), which refers to the ratio of the maximum pressure in a period of time to the corresponding displacement at that point, and expresses the transient stiffness characteristics during interaction. GS and PFP are calculated through the force, displacement and other data measured by the sensor during the interaction process. H represents subjective hardness, k 1 , k 2 is the partial regression coefficient, which indicates the degree of influence of the characteristic parameter on the perception result. The larger the value, the greater the weight of the characteristic on the subjective perception result. According to the hardness tactile perception model, the subjective hardness felt by people is mainly affected by two characteristics: the global stiffness coefficient (GS) and the peak force / displacement (PFP). The sign of the partial regression coefficient indicates that the subjectively perceived hardness strength is positively correlated with the logarithm of the global stiffness GS and negatively correlated with the logarithm of the peak force / displacement ratio PFP.

[0019] Furthermore, the present invention uses a combination of force and vibration to reproduce the tactile attribute of hardness. The force tactile control model described in step 3 can be understood as the inverse transformation of the hardness tactile perception model in step 2. The hardness tactile perception model is a model from objective stimulation to subjective perception, while the force tactile control model studies the control relationship from subjective perception to objective stimulation, and converts the subjectively perceived hardness into the control parameters of objective stimulation for expression. Srinivasan et al. have found that the perception of hardness is almost linearly related to the discharge rate (number of pulses per second) response of SA1 skin afferent neurons, and SA1 receptors are more sensitive to continuous pressure and vibration. At the same time, Kuchenbecker et al. found that superimposing vibration signals on contact force can enhance the perception of surface hardness.

[0020] Furthermore, the hardness tactile perception model ultimately realizes the subjective perception of hardness through active force feedback and vibration devices, mainly including but not limited to controlling the size of mechanical force (F) and the size of vibration motor frequency (f v ), the model is expressed as follows:

[0021] F = k f ·k 1 ·GS·x d

[0022]

[0023] Among them, k f represents the force feedback control coefficient, k v Represents the vibration feedback control coefficient, which is determined according to the objective output characteristics of different force feedback devices and vibration elements, and is used to adjust the magnitude of force and vibration to meet the output range of the reproduction device. d It is the normal depth of the probe penetrating the virtual object surface. Since the relationship curve between vibration frequency and vibration intensity tends to be an inverted "U" shape, that is, as the vibration frequency increases, the vibration intensity first increases and then decreases. When the frequency exceeds a certain value (usually 250Hz), the vibration frequency and vibration intensity are negatively correlated. In order to reflect the negative correlation between the PFP characteristics of vibration feedback reproduction and hardness perception intensity in the hardness perception model, f v The represented control frequency output range takes the frequency band where the vibration frequency is negatively correlated with the vibration intensity. is the initial vibration frequency, which is usually the minimum value of the frequency band that is negatively correlated with the vibration intensity. d ) is a function of PFP changing with displacement, indicating the value of PFP corresponding to different stages of the pressing process. cvThe critical PFP value for generating vibration feedback, that is, when the real-time PFP value during pressing is greater than the critical value, vibration feedback is generated, and when it is less than the critical value, there is no vibration feedback, only force feedback. The critical PFP value is obtained by statistical calculation based on the inherent physical properties of the reproduced deformable sample. For the hardness reproduction of different types of objects, such as linear / nonlinear deformable objects, objects with rigid surfaces / surface deformable objects, the PFP critical value is different.

[0024] Furthermore, GS is a relatively stable feature that reflects the global average hardness property of the object, indicating the average force required to cause unit deformation, and is expressed by force feedback during reproduction. PFP is a transient feature that reflects the local hardness property of the object during the pressing process, indicating the time-varying characteristics of force and displacement at each stage of pressing, and is expressed by vibration feedback during reproduction. Using vibration feedback to express the PFP feature is conducive to capturing the transient changes in the hardness of the object during the interaction. When the force change rate in the interaction is large (the transient stiffness change is large), the device generates vibration feedback to reflect the transient changes. Wellman et al. approximated the force response of the probe on the rigid surface as an exponentially decaying sinusoidal signal. In the process of using the probe to press, the subject will feel an obvious force mutation when the probe just contacts the surface of the object. This force mutation can be understood as a vibration feature in the interaction, and then the force felt by the subject tends to stabilize. The PFP and pressure sequence during the pressing process are roughly as follows: Figure 3 As shown, the greater the pressing depth, the greater the local stiffness. At this time, the global stiffness cannot well reflect the overall force and displacement changes. Moreover, as the local stiffness increases, it will cause excessive active force, unstable pressing, and large changes in the transient characteristic PFP. To address this problem, this model uses a stimulation method that combines force and vibration to significantly improve the hardness reproduction effect. The force tactile control model described in step 3 is the "inverse transformation" of the hardness tactile perception model in step 2. It refers to the control method of mechanical force and vibration stimulation to achieve the consistency between the global stiffness and peak force / displacement characteristic parameters transmitted by virtual stimulation and the real feeling effect.

[0025] Furthermore, Zilles et al. found that when the tactile reproduction device interacts with a virtual object, the actual interaction point is located inside the virtual object, but the interaction point seen in the reproduced scene should be located on the surface of the object. The distance from the actual interaction point to the surface interaction point is called the penetration depth. There is an obvious linear relationship between the penetration depth of the probe and the subjectively perceived hardness. GS, as a stable feature, reflects the average force required to cause a unit displacement. Therefore, when reproducing hardness, GS and displacement are used as parameters to control the magnitude of the mechanical force. Interaction algorithm of force feedback Figure 2 shown.

[0026] Furthermore, the vibration intensity can be controlled by the vibration amplitude or the vibration frequency. Relevant studies have found that the vibration amplitude and the vibration intensity have a positively correlated linear relationship, while the relationship curve between the vibration frequency and the vibration intensity tends to be an inverted "U" shape. That is, as the vibration frequency increases, the vibration intensity first increases and then decreases. When a certain frequency value is exceeded, the vibration frequency and the vibration intensity are negatively correlated. According to the perception model, it is known that PFP is negatively correlated with the subjective hardness perception intensity. Therefore, during reproduction control, the vibration of the motor is controlled by a frequency band that is negatively correlated with the vibration intensity. That is, when PFP increases, the vibration frequency increases and the perceived vibration intensity decreases. At the same time, in the initial contact stage of pressing and the stage when the pressing displacement tends to the maximum value, the impact force changes significantly and the vibration characteristics are significant. At this time, PFP is much larger than GS, while in other pressing stages, PFP is stably smaller than GS. Therefore, the critical PFP value P for generating vibration feedback is 1.3847 W / cm2. cv It is basically the same as GS, and is adjusted based on the analysis of specific examples.

[0027] Furthermore, the force tactile reproduction device described in step 4 combines the vibration element with the force feedback device based on the consideration of human tactile perception habits. The present invention connects the vibration element with the force feedback device, and when the pen tip of the handheld device is pressed on the reproduction sample, the reproduction program of the force tactile control model will provide the reproduction device with stimulation feedback that conforms to human perception characteristics.

[0028] Beneficial effects: The present invention reproduces the process of people perceiving the hardness tactile sensation of the surface of real objects. First, the interaction data of human hands touching real objects is collected and input into the force tactile perception model. Secondly, the perceived intensity of subjective hardness is obtained through model analysis. Then, the perceived hardness is input into the force tactile control model. Finally, the vibration tactile generating device and the force feedback device are combined to reproduce the hardness tactile sensation of the surface of real objects. Compared with the traditional force tactile reproduction that focuses on the "consistency of tactile stimulation" with the real environment, and reproduces the force information in the process of interaction between people and real objects with the help of force feedback devices, the present invention emphasizes the "consistency of feeling". When establishing the force tactile control model, the perceptual characteristics of human skin receptors are analyzed, and the influence of human subjective factors on tactile sensation in the process of perceiving the surface of objects is fully considered. At the same time, the tactile information is reproduced by combining the two stimulation forms of force and vibration. This method not only overcomes the limitations of the reproduction of a single stimulation form, but also the reproduction effect is more in line with the tactile perception characteristics of the human body, and improves the realism of the reproduction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flowchart for hardness tactile reproduction.

[0030] Figure 2 Schematic diagram of force feedback algorithm.

[0031] Figure 3PFP and pressure sequence during compressions. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is described in detail below with reference to examples and drawings.

[0033] Embodiment 1: A method for reproducing the tactile sense of hardness based on the bionic characteristics of tactile perception proposed by the present invention is to reproduce the tactile attribute of the hardness of an object. The reproduction process is as follows: Figure 1 The research process of the reproduction method mainly includes four parts: force tactile interaction data collection, force tactile perception model establishment, force tactile control model establishment and reproduction device design.

[0034] This example uses five springs of different stiffness as experimental samples, and reproduces the hardness of the springs through a force tactile reproduction device. In addition, the present invention can also be used to simulate tumors in surgery and reproduce the hardness of various flexible objects in VR games. The basic process of the present invention will be described below in conjunction with specific steps:

[0035] A method for reproducing hardness tactile sensation based on bionic characteristics of tactile sensation, the method comprising the following steps:

[0036] Step 1: Collect force tactile interaction data. In this example, a pressure sensor is placed on a flat panel to record pressure data, and a force feedback device is used to record displacement data. When the subject uses a rigid probe (the pen tip of the force feedback device) to press on the partition, the sensor will capture the force and displacement data during the press, and record the data through the data acquisition card acquisition device. Based on the collected displacement x and pressure f, the probability distribution function of the peak force and the displacement corresponding to the user-applied pressure f are calculated, and substituted into the following two formulas to calculate the global stiffness coefficient (GS) and peak force / displacement (PFP) of the sample.

[0037]

[0038]

[0039] Among them, P i (f) is the probability distribution function of the peak force, is the displacement corresponding to the force f applied by the user.

[0040] Step 2: Calculation of subjectively perceived hardness. Based on the characteristic parameters calculated in step 1, substitute them into the previously established regression relationship model of objective stimulus characteristic parameters / subjective hardness perception intensity. The force tactile perception model finally established in this example is as follows:

[0041] H=4.044log GS-3.077log PFP

[0042] Among them, GS represents global stiffness, reflecting the steady-state force characteristics, PFP represents peak force / displacement, reflecting the transient vibration characteristics, and the partial regression coefficients of 4.044 and -3.077 represent the influence of the two characteristics on the perception results. GS is positively correlated with the perceived hardness strength, while PFP is negatively correlated with the perceived hardness strength.

[0043] Step 3: Establishment of force tactile control model. In this example, a spring sample with uniform elasticity is used as an example for linear fitting. Its PFP varies with the normal displacement x d The function of the change PFP(x d ) roughly conforms to the power exponent, and the fitting results are as follows:

[0044]

[0045] The steady-state characteristics and transient characteristics in the expression of step 2 are expressed by force and vibration respectively. The reproduction device selected in this example is the Geomagic Touch force feedback device and the C-2 tactile motor of the American EAI company. The output force range of the force feedback device is 0-3.3N, and the working voltage of the vibration motor is 0.2-7V. When the output voltage is 7V, the frequency range that is negatively correlated with the vibration intensity is (f v ≥250Hz), and the force control coefficient and vibration control coefficient are estimated accordingly. The final control model expression is as follows:

[0046] F=1.341·GS·x d

[0047]

[0048] Among them, x d is the probe penetration depth, GS is consistent with the perception model and shows a steady-state value, while PFP (x d ) is expressed as a function of displacement. For the sample in this example, the critical PFP value is equal to GS, which is about 0.082N / mm.

[0049] Step 4: Design a force tactile reproduction device and conduct a reproduction experiment. In this example, a vibration motor is embedded in a finger sleeve and connected to a force feedback device to provide both mechanical force and vibration feedback. When a person holds the pen tip of the device and presses on the reproduction sample, the reproduction program of the force tactile control model will control the output of mechanical force and vibration voltage as the pressing process progresses, thereby providing objective stimulation that conforms to human perception characteristics.

[0050] This embodiment is only for the hardness reproduction of springs with different stiffnesses, and the calculation methods of the perception model and the control model are also for this process. When reproducing the hardness tactile sensation of other samples, the corresponding interaction data should be collected, effective features should be selected through correlation analysis, and the corresponding perception model and control model should be calculated, so as to reproduce the hardness tactile sensation through the reproduction device.

[0051] It should be noted that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Equivalent replacements or substitutions made on the basis of the above technical solutions all fall within the protection scope of the present invention.

Claims

1. A hardness tactile reproduction method based on the bionic characteristics of tactile perception, characterized in that: This method includes the following steps: Step 1: Acquisition of interaction data: The distal force tactile sensor measurement device presses an object under the control of the master operator. During the pressing process, force tactile interaction data, including force, displacement, and acceleration, are collected through the force tactile sensor, and the force tactile stimulation characteristic parameters during the hardness perception process are calculated and extracted; Step 2: Obtaining the subjective hardness perception intensity: The characteristic parameters extracted in Step 1 are used as objective stimulation characteristic parameters, and the objective stimulation characteristic parameters are input into the subjective hardness tactile perception model to obtain the subjective hardness perception intensity of the currently perceived real object; Step 3: Determination of the control parameters of the force tactile reproduction device: The subjective hardness perception intensity obtained in Step 2 is substituted into the force tactile control model to calculate the control parameters of the force tactile reproduction device. The control parameters include force feedback control parameters and vibration feedback control parameters; Step 4: Hardness reproduction: According to the control parameters obtained in Step 3, the force feedback and vibration feedback actuators in the force tactile reproduction device generate a combined force / vibration tactile stimulation on the master operator, and the hardness reproduction of the distal real object is realized through the hardness reproduction control model; The subjective hardness tactile perception model in Step 2 refers to a multi-input / single-output regression model established for the interaction characteristic parameters and the subjective hardness perception intensity. This model describes the quantitative relationship between the characteristics related to force and vibration in the objective interaction data and the subjective hardness perception intensity. The model is expressed as the following formula: H = k 1 ·logGS - k 2 ·log PFP Among them, GS is the global stiffness (N / mm), which refers to the average value of the ratio of force to displacement during the complete pressing process, that is, the force required to cause a unit displacement, and represents the characteristic of the overall hardness of the object. PFP is the peak force / displacement (N / mm), which refers to the ratio of the peak value of the pressing force within a period of time to the displacement corresponding to the peak point, that is, the local stiffness corresponding to different stages of the pressing process, and reflects the transient characteristics of the force and displacement changing with time. GS and PFP are calculated from the force and displacement data measured by the sensor during the interaction. H represents the subjective hardness, and k 1 , k 2 are partial regression coefficients, indicating the influence degree of the characteristic parameters on the perception result. The larger the value, the greater the influence weight of the characteristic on the subjective perception result. The sign of the partial regression coefficient indicates that the hardness intensity of the subjective perception is positively correlated with the logarithm of the global stiffness GS and negatively correlated with the logarithm of the peak force / displacement PFP; The hardness reproduction control model finally realizes the subjective perceived hardness through the force and vibration feedback devices. The control forms include controlling the magnitude of the mechanical force F and the magnitude of the vibration frequency f v , and the weights of the force feedback and vibration feedback in the control are consistent with the partial regression coefficients of the perception model. The model is expressed as follows: F = k f ·k 1 ·GS·x d Among them, k f represents the force feedback control coefficient, and k v represents the vibration feedback control coefficient, which are determined according to the objective output characteristics of different force feedback devices and vibration elements respectively, and are used to adjust the magnitudes of the force and vibration to satisfy the output range of the force tactile rendering device. x d is the normal depth at which the probe of the force tactile rendering device penetrates the surface of the virtual object. Since the relationship curve between the vibration frequency and the vibration intensity tends to be an inverted "u" shape, that is, as the vibration frequency increases, the vibration intensity first increases and then decreases. When it exceeds 250 Hz, the vibration frequency and the vibration intensity are negatively correlated. The control frequency output range represented by f v takes the frequency band where the vibration frequency and the vibration intensity are negatively correlated. is the initial vibration frequency, taking the minimum value of the frequency band where it is negatively correlated with the vibration intensity; PFP(x d ) is a function of PFP varying with displacement, representing the values of PFP corresponding to different stages of the pressing process; P cv is the critical PFP value for generating vibration feedback, that is, when the real-time PFP value during the pressing process is greater than this critical value, vibration feedback is generated, and when it is less than the critical value, there is no vibration feedback and only force feedback. The critical PFP value is statistically calculated based on the inherent physical properties of the deformable samples to be reproduced, for the hardness reproduction of different types of objects.

2. The hardness tactile reproduction method based on the bionic characteristics of tactile perception according to claim 1, characterized in that: The force tactile control model described in Step 3 is the "inverse transformation" of the subjective hardness tactile perception model in Step 2, which refers to the control method of mechanical force and vibration stimulation when the global stiffness and peak force / displacement ratio transmitted by the virtual stimulation are consistent with the real sensory effect.

3. The hardness tactile reproduction method based on the bionic characteristics of tactile perception according to claim 1, characterized in that: The force tactile reproduction device described in Step 4 adopts a combination of vibration stimulation and force stimulation. The vibration execution element is combined with the force feedback device. When a person holds the end of the force feedback device and makes a pressing action on the virtual reproduction sample, the force tactile reproduction device provides force and vibration stimulation feedback that conforms to the human perception characteristics.

Images