Visual and electromagnetic fusion type position detection method and force tactile reproduction system

Abstract

The application discloses a visual and electromagnetic fusion type position detection method and a force tactile reproduction system, and the system comprises a background electromagnet module, a fingertip electromagnet module, a camera module, a voltage sensing module, an electromagnet driving module, a power module and a central control module. The relationship between the 6D pose of the fingertip electromagnet module and the theoretical induction voltage of the background electromagnet module is obtained based on the electromagnetic induction principle; the 3D position and 3D attitude relationship of the fingertip relative to the center of the palm are obtained based on the forward kinematics of the human hand; the cost function is established according to the measured induction voltage and the theoretical induction voltage; the parameters of the cost function are reduced by using the optical method and the fingertip pose data table; the cost function is calculated by the central control module in an iterative mode, the 6D pose of the fingertip electromagnet based on the 6D pose of the fingertip electromagnet module is obtained, the interaction force received by the virtual hand in the virtual interaction environment is calculated by the central control module, and a control signal is output to the electromagnet driving module, so that the force tactile reproduction is realized.

Description

Technical Field

[0001] This invention belongs to the field of position detection technology in human-computer interaction, and particularly relates to a visual and electromagnetic fusion position detection system and method for force-tactile human-computer interaction. Background Technology

[0002] Whether in daily life or in human-computer interaction, hand movements are the most frequent and effective means by which humans communicate with the outside world. In multimodal human-computer interaction, hand movements are not only a necessary input for visual reproduction, but also the first step in multimodal human-computer interaction. Researchers have developed various mature methods for tracking hand movements based on different physical laws. Hand movement tracking technologies can be categorized based on the physical laws they are based on, such as optical tracking methods, inertial tracking methods, magnetic tracking methods, and acoustic tracking methods, among others.

[0003] Among them, optical tracking methods can achieve millimeter-level accuracy and have a wide detection range, but they suffer from two problems: ambient light pollution and light occlusion, with the latter being particularly critical. Inertial tracking methods are compact and simple in structure and can be installed in any location, but due to the nature of their time-integration calculations, they suffer from drift problems, and the error increases over time. Acoustic tracking methods also suffer from object occlusion problems and energy dispersion, resulting in low accuracy. Magnetic tracking methods have advantages such as high accuracy, no object occlusion problems, and suitability for natural environments, but the magnetic field strength at a point in space decreases rapidly with distance from the magnetic source, thus limiting the detection range of magnetic tracking methods.

[0004] Optical tracking, as a mature technology, is widely used in the field of hand tracking. Leap Motion adopted a novel posture and position tracking system [8] to achieve high-precision hand position tracking, with a detection accuracy of about 0.01 mm for each fingertip position and a frame rate of up to 300 fps. Kuma et al. from India used Kinect and Leap Motion sensors to capture the positions of fingers and palms, and improved the accuracy of hand position detection through data fusion of the two sensors. Qin et al. from Taiwan, China, used a light-emitting diode light source, two lens arrays and two light sensors to design and implement a finger positioning device called "virtual mouse". Its system is compact and has high resolution, and can replace touchpad, touch screen and even mouse. Li et al. developed a bare-finger three-dimensional air touch interaction technology for portable devices. They used directional optical design on the camera module, and the interaction range was 1.5 cm to 50 cm above the entire display surface. They also proposed a visual computing algorithm different from skin color detection to determine the position of the fingertip, and achieved an air touch function with an average position error of less than 1 cm.

[0005] In recent years, researchers have developed many magnetic-based hand tracking methods, which can be categorized into DC magnetic field tracking methods and AC magnetic field tracking methods. Because permanent magnets generate a relatively stronger magnetic field than electromagnets of the same volume and do not produce eddy current effects, existing force-sensing devices generally use permanent magnets as force transmission terminals. Motion tracking methods based on DC magnetic fields generated by permanent magnets mainly involve deploying magnetic sensor arrays (such as Hall effect sensors, magnetoresistive sensors, etc.) to track the permanent magnet used as a marker.

[0006] McIntosh and colleagues at the University of Copenhagen, Denmark, designed a smartwatch that allows touch input from the air. It consists of a watch, a permanent magnet at the fingertip, a magnetic sensor array beneath the watch, and an excitation coil. The magnetic sensor array obtains the magnetic field strength of the permanent magnet at that location and calculates it using a nonlinear optimization algorithm, which is then used to convert it into the fingertip's position. This allows the user to tap a specific area of ​​the watch. After tapping, the excitation coil is alternately driven, causing the permanent magnet to vibrate and achieving a force feedback effect. Using more sensors allows for the tracking of more permanent magnets.

[0007] Song et al. from Harbin Institute of Technology designed a three-dimensional magnetic sensor array comprising 36 sensors. By collecting data from each sensor and using a nonlinear optimization algorithm, they were able to track three permanent magnets with millimeter-level precision within an area of ​​30cm*50cm*40cm. Compared to permanent magnets, electromagnets offer a wider range of controllable parameters and a higher upper limit for magnetic field strength, making them highly promising for multi-point tracking and applications requiring tracking distances exceeding 1 meter. Unlike permanent magnets, electromagnets typically require a large power supply module, making them unsuitable for wearable devices. Therefore, in tracking systems, electromagnets are generally used as a base fixed to the ground to track one or more sensors worn by the user.

[0008] Currently, most mature commercial motion tracking products based on magnetism utilize electromagnets, such as those from companies like Polhemus, Stem, and Razer. Polhemus' Viper series, in particular, can track up to 16 sensors across all attitudes within a distance of 3 to 6 meters, with a refresh rate of up to 960Hz. At the algorithm level, regardless of whether it's a permanent magnet or electromagnet system, the magnet is generally treated as a magnetic dipole, and optimization algorithms or other algorithms are designed based on this model. For situations where the magnet cannot be considered a magnetic dipole, such as when the magnet is too close to the sensor, or when the magnet's shape is incompatible with the magnetic dipole model, lookup table methods or neural network algorithms can be used.

[0009] The main drawback of optimization algorithms based on the magnetic dipole model is their long computation time, which is primarily related to the number of parameters to be solved in the cost function, the optimization method, and the choice of initial guesses. Constructing constraint relationships between the fingertips and the center of the palm based on hand kinematics can reduce the number of parameters to be solved in the cost function and decrease the computation time. Ma et al. developed a wearable gesture recognition system. This system consists of a wristband containing a magnetic field sensor worn on the wrist and permanent magnets installed at the fingertips. Based on hand kinematics and the output of the magnetic field sensor, the system can obtain the 6D pose of the fingertips and reconstruct the bending angle of each joint of the fingers, thereby completing gesture recognition. Summary of the Invention

[0010] To address the aforementioned technical problems, the present invention aims to propose a visual and electromagnetic fusion position detection system and method for force-tactile human-computer interaction, which enables reusable, natural, unobstructed, faster and more accurate position detection in force-tactile applications.

[0011] The present invention proposes a visual and electromagnetic fusion-based position detection method for force-tactile human-computer interaction, the method comprising:

[0012] Step 1: Establish a virtual scene including virtual hand and virtual object models, and establish the spatial mapping relationship between the real operating space inside the background electromagnet module and the virtual scene, as well as the three-dimensional positional mapping relationship between the fingertip electromagnet module and the virtual hand.

[0013] Step 2: Based on the principle of electromagnetic induction, obtain the relationship between the 6D pose of the fingertip electromagnet submodule and the induced voltage of each background electromagnet submodule; based on the forward kinematics of the human hand, obtain the constraint relationship between the 3D position and 3D posture of the fingertip relative to the center of the palm, and construct the fingertip pose data table.

[0014] Step 3: Simulate the mapping relationship between the 6D pose, driving current, and background electromagnet module driving current of the fingertip electromagnet module and the force of each magnetic moment component of the fingertip electromagnet module, and obtain its offline simulation data.

[0015] Step 4: Obtain the number of fingertips to be detected and the initial 6D pose using optical methods;

[0016] Step 5: The fingertip electromagnet module and the background electromagnet module are switched to position detection mode. The voltage sensing module obtains the voltage at both ends of the background electromagnet and obtains the induced voltage of each fingertip electromagnet sub-module after filtering. The 6D pose of the center of the palm is obtained according to the optical method. Combined with the previous or initial 3D position of the fingertip electromagnet module, the 3D pose of the fingertip electromagnet module is obtained by querying the fingertip pose data table.

[0017] Based on the 3D pose of the fingertip electromagnet module and the induced voltage of the background electromagnet, a cost function is established and solved using an iterative method to finally obtain the 6D pose of the fingertip electromagnet module.

[0018] Step 6: Based on the force-tactile interaction model between the fingertip and the virtual object in the virtual scene, calculate the target 3D feedback force of the fingertip electromagnet module at the current moment; switch the fingertip electromagnet module and the background electromagnet module to the force-tactile reproduction mode, and obtain the driving current required by the fingertip electromagnet module and the background electromagnet module according to the offline simulation data and the given 6D pose of the fingertip electromagnet module and the target 3D feedback force.

[0019] Furthermore, the background electromagnet module is composed of three pairs of mutually parallel hollow cylindrical electromagnet sub-modules placed orthogonally. The spacing between each pair of electromagnet sub-modules is the same and equal to their respective outer diameters, and the center points of the central axes of each pair coincide. Taking the geometric center of the background electromagnet module as the origin, the central axis of any sub-module is set as the X-axis, and the central axis of any other sub-module is set as the Y-axis. The Z-axis is determined according to the right-hand rule, and a spatial coordinate system {G} is established.

[0020] The background electromagnet submodule is a hollow cylindrical coil with N1 turns, formed by multiple layers of uniformly and densely wound circular coils. The physical parameters of the six submodules EMx1, EMx2, EMy1, EMy2, EMz1, and EMz2 are completely identical. Each background electromagnet submodule is connected to a voltage sensing module at both ends.

[0021] The fingertip electromagnet module employs a spherical structure with three fingertip electromagnet sub-modules symmetrically placed in three orthogonal directions. For all fingertip electromagnet modules, with the geometric center of the fingertip electromagnet module as the origin, the central axis of any one sub-module is set as the X-axis, and the central axis of any other sub-module in any other direction is set as the Y-axis. The Z-axis is determined according to the right-hand rule, establishing a spatial coordinate system {S}. i}; The 6D pose of the fingertip electromagnet module numbered i is (x i y i , z i α i ,β i γ i ), where (x i y i , z i Let α be the position coordinates of the fingertip electromagnet module in the spatial coordinate system {G}. i ,β i γ i ) are respectively the fingertip electromagnet module orbiting the coordinate system {S i The rotation angles along the X, Y, and Z axes; the fingertip electromagnet module numbered i is named EM. FTiIts fingertip electromagnet submodules in the x, y, and z directions are respectively named EM FTix EM FTiy EM FTiz Each fingertip electromagnet submodule has identical physical parameters and has N2 turns. The fingertip electromagnet submodule is composed of two circular coils connected in series in the same direction, which overlap in space.

[0022] Furthermore, based on the forward kinematics of the human hand, the human hand skeleton is treated as a linkage structure. When the palm is placed horizontally, the relationship between the fingertip position and posture is determined, and a fingertip pose data table is constructed. The expression for the relationship between the fingertip position and posture is as follows:

[0023] [α i ,β i γ i ]=f(x i y i , z i L)

[0024] Where L = [l1, l2, l3, ... l n [ ] is the collection of the lengths of the phalanges and metacarpals of the hand.

[0025] Furthermore, step four specifically involves placing the human hand horizontally in the operating space with all fingers extended when the system is turned on. At this time, the first position detection is performed to obtain the number of fingertips to be detected, the corresponding set of lengths L of each phalanx and metacarpal bone, and the initial 6D pose of each fingertip using an optical method.

[0026] Furthermore, step five includes:

[0027] Step 5.1: The fingertip electromagnet module and the background electromagnet module are switched to position detection mode. The position detection mode uses an amplitude of U... sigmax A sinusoidal driving current excites all fingertip electromagnet submodules, and the frequencies of each fingertip electromagnet submodule are fi and fj respectively. 1x f 1y f 1z f 2x , ...f nz Minimum signal frequency f min and maximum signal frequency f max They exhibit an equal and increasing relationship;

[0028] Close all drive channels of the background electromagnet submodules and turn on the voltage sensing module; the duration T of the position detection mode. s The time constant τ of the fingertip electromagnet submodule is greater than ft ;

[0029] Step 5.2, to locate EM FT1 For example, based on the previous positioning results, select the distance EM. FT1 The three more distant background electromagnet sub-modules are designated EMx1, EMy1, and EMz1. The voltage sensing module obtains the corresponding voltage U across the two ends of the background electromagnet. x1 U y1 U z1 The measured induced voltage is digitally filtered to separate the voltage components belonging to each fingertip electromagnet submodule.

[0030] For EMx1, a cost function is established based on the theoretically calculated and actually measured values ​​of the induced voltage:

[0031]

[0032] Where M x1 =(O x1x O x1y O x1z M x1 N x1 P x1 () represents the position and orientation of the geometric center of the background electromagnet submodule EMx1; the position of the fingertip electromagnet solved by this cost function is (x 1_x1 y 1_x1 , z 1_x1 );

[0033] Construct EMY1 and EMz1 in the same way regarding EM FT1 The cost function, the solved positions of the fingertip electromagnets are (x 1_y1 y 1_y1 , z 1_y1 ), (x 1_z1 y 1_z1 , z 1_z1 ); Finger electromagnet submodule EM FT1 The position is represented as:

[0034]

[0035]

[0036]

[0037] Construct the cost function required for the positioning of the remaining fingertip electromagnets using the same method;

[0038] Step 5.3: Use optical methods to obtain the 6D pose of the palm. Combine the 6D pose of the palm with the previous fingertip position, query the fingertip pose data table, and obtain the 3D pose of the fingertip to be detected.

[0039] Furthermore, by combining the 3D pose of the fingertips with the established cost function, an optimization algorithm is used to obtain the numerical solution of the three position variables of each cost function, thereby obtaining the 6D pose of all fingertip electromagnet modules.

[0040] This application also provides a non-contact force tactile reproduction system, which includes a background electromagnet module, a fingertip electromagnet module, a camera module, a voltage sensing module, an electromagnet driving module, a power supply module, and a central control module.

[0041] The background electromagnet module comprises 3 pairs of spatially orthogonal 6 background electromagnet sub-modules, with two electromagnet sub-modules in each direction;

[0042] The fingertip electromagnet module adopts a spherical structure in which three fingertip electromagnet sub-modules are symmetrically placed in three orthogonal directions;

[0043] The voltage sensing module includes a voltage limiting module and a voltage sensor. The voltage limiting module limits the input signal of the voltage sensor to no more than the maximum value. The voltage sensor converts the analog signal into a digital signal.

[0044] The electromagnet drive module includes an electromagnet control submodule and an electromagnet drive submodule. The electromagnet control submodule is used to receive control information from the central control module and generate PWM signals to control the electromagnet drive submodule to output the required electrical signals.

[0045] The central control module obtains the 6D position and pose of the fingertip according to the position detection method as described in claim 1, calculates the interaction force of the fingertip agent in the virtual environment, and outputs a drive signal.

[0046] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0047] (1) The visual and electromagnetic fusion position detection method can speed up the magnetic field position detection algorithm, thereby improving dynamic accuracy and overall immersion.

[0048] (2) The position detection method based on the fusion of vision and electromagnetic can greatly reduce light occlusion and light pollution.

[0049] Interference from factors such as these improves the accuracy and robustness of position detection.

[0050] (3) The reuse of background electromagnet modules and fingertip electromagnet modules makes the force-touch human-computer interaction system more compact, reduces the overall cost, and expands the operating space. Attached Figure Description

[0051] Figure 1 This is a structural diagram of the background electromagnet module and the fingertip electromagnet module;

[0052] Figure 2 This is a schematic diagram of a visual and electromagnetic fusion position detection system for force and tactile reproduction. Detailed Implementation

[0053] The control method of the present invention will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0054] Example 1

[0055] This embodiment provides a non-contact force-tactile reproduction system, wherein the vision and electromagnetic fusion position detection system for force-tactile human-computer interaction is as follows: Figure 2 As shown, the vision and electromagnetic fusion system for human-computer interaction based on force and tactile reproduction includes a background electromagnet module 1, a fingertip electromagnet module 2, a voltage sensing module 3, a camera module 4, an electromagnet drive control module 5, and a central control module 6.

[0056] The background electromagnet module comprises 3 pairs of spatially orthogonal 6 background electromagnet sub-modules, with two electromagnet sub-modules in each direction;

[0057] The fingertip electromagnet module adopts a spherical structure in which three fingertip electromagnet sub-modules are symmetrically placed in three orthogonal directions;

[0058] The voltage sensing module includes a voltage limiting module and a voltage sensor. The voltage limiting module limits the input signal of the voltage sensor to no more than the maximum value, and the voltage sensor converts the analog voltage signal into a digital signal.

[0059] The electromagnet drive module includes an electromagnet submodule and an electromagnet drive submodule. The electromagnet submodule is used to receive control information from the central control module and generate a PWM signal to control the electromagnet drive submodule to output the required electrical signal.

[0060] The central control module obtains three-dimensional feedback force based on the interaction between the fingertip position information and the virtual environment, and realizes the mapping from force to electromagnet excitation current based on the multi-point interactive haptic reproduction control algorithm.

[0061] Furthermore, a specific excitation current is applied to the background electromagnet module to generate the required background electromagnet. The background electromagnet module consists of three pairs of orthogonally placed hollow cylindrical electromagnet sub-modules that are parallel to each other. The spacing between each pair of electromagnet sub-modules is the same and equal to their respective outer diameters, and the center points of the central axes of each pair coincide. Taking the geometric center of the background electromagnet module as the origin, the central axis of any one sub-module is selected as the X-axis, and the central axis of any other sub-module in any other direction is selected as the Y-axis. The Z-axis is determined according to the right-hand rule, and a spatial coordinate system {G} is established.

[0062] The background electromagnet submodule is a hollow cylindrical coil with N1 turns, formed by multiple layers of uniformly and densely wound circular coils. The physical parameters of the six submodules EMx1, EMx2, EMy1, EMy2, EMz1, and EMz2 are completely identical. Each background electromagnet submodule is connected to a voltage sensing module at both ends.

[0063] The fingertip electromagnet module adopts a spherical structure in which three fingertip electromagnet sub-modules are symmetrically placed in three orthogonal directions. The fingertip electromagnet sub-modules in the x, y, and z directions of the fingertip electromagnet module numbered i are named EMFTix, EMFTiy, and EM, respectively. FTiz The driving current I corresponds to the three force tactile reproduction modes respectively. FTix U+I FTiy I FTiz The maximum safe current of the position detection mode drive current submodule is I. FTmax The fingertip electromagnet submodule consists of two circular coils connected in series in the same direction, which overlap in space.

[0064] The voltage sensing module includes a voltage limiting module and a voltage sensor. The voltage limiting module limits the input signal of the voltage sensor to no more than the maximum value, and the voltage sensor converts the analog voltage signal into a digital signal.

[0065] The electromagnet drive module consists of an electromagnet submodule and an electromagnet drive submodule; the electromagnet submodule is used to receive control information from the central control module and generate PWM signals to control the electromagnet drive submodule to output the required electrical signals.

[0066] The central control module calculates the 6D position and pose of the fingertip based on the induced voltage of the background electromagnet, calculates the interaction force of the fingertip agent in the virtual environment, and outputs a drive signal.

[0067] Furthermore, each sub-module of the aforementioned three-dimensional background electromagnet module has inner and outer diameters of R1 and R2, respectively, a height of L, and a maximum safe current of I. max ;

[0068] Furthermore, the fingertip electromagnet module consists of three orthogonally placed cylindrical hollow coil sub-modules with their centers at the same point. Each sub-module is composed of two coils connected in series, and they overlap in space.

[0069] Example 2

[0070] This embodiment describes a visual and electromagnetic fusion-based position detection method for force-tactile human-computer interaction. The force-tactile human-computer interaction is achieved based on the electromagnetic force interaction of a background electromagnet and a fingertip electromagnet module. The method includes:

[0071] Step 1: Establish a virtual scene including virtual hand and virtual object models, and establish the spatial mapping relationship between the real operating space inside the background electromagnet module and the virtual scene, as well as the three-dimensional positional mapping relationship between the fingertip electromagnet module and the virtual hand.

[0072] Step two: Based on the principle of electromagnetic induction, obtain the relationship between the 6D pose of the fingertip electromagnet submodule and the induced voltage of each background electromagnet submodule. Based on the forward kinematics of the human hand, obtain the constraint relationship between the 3D position and 3D pose of the fingertip relative to the center of the palm, and construct a fingertip pose data table;

[0073] Step 3: Using the finite element method, simulate the mapping relationship between the 6D pose of the fingertip electromagnet module, the driving current, and the driving current of the background electromagnet module on the force of each magnetic moment component of the fingertip electromagnet module, and obtain its offline simulation data.

[0074] Step 4: Obtain the number of fingertips to be detected and the initial 6D pose using optical methods;

[0075] Step 5: The fingertip electromagnet module and the background electromagnet module are switched to position detection mode. The voltage sensing module obtains the voltage across the two ends of the background electromagnet and obtains the induced voltage of each fingertip electromagnet sub-module after filtering. The 6D pose of the center of the palm is obtained according to the optical method. Combined with the previous or initial 3D position of the fingertip electromagnet, the 3D pose of the fingertip electromagnet module is obtained by querying the fingertip pose data table.

[0076] Based on the 3D pose of the fingertip electromagnet module and the induced voltage of the background electromagnet, the central control module establishes a cost function and solves it using an iterative method to finally obtain the 6D pose of the fingertip electromagnet module.

[0077] Step six: Based on the force-tactile interaction model between the fingertip and the virtual object in the virtual scene, the central control module calculates the target three-dimensional feedback force of the fingertip electromagnet module at the current moment. The fingertip electromagnet module and the background electromagnet module switch to force-tactile reproduction mode. According to the offline simulation data and the given 6D pose of the fingertip electromagnet module and the target three-dimensional feedback force, the control board outputs the required driving current for the fingertip electromagnet module and the background electromagnet module.

[0078] In step one, the background electromagnet module consists of three pairs of parallel hollow cylindrical electromagnet sub-modules placed orthogonally. The spacing between each pair of electromagnet sub-modules is the same and equal to their respective outer diameters, and the center points of the central axes of each pair coincide. Taking the geometric center of the background electromagnet module as the origin, the central axis of any one sub-module is selected as the X-axis, and the central axis of any other sub-module in any other direction is selected as the Y-axis. The Z-axis is determined according to the right-hand rule, and a spatial coordinate system {G} is established.

[0079] The background electromagnet submodule is a hollow cylindrical coil with N1 turns, formed by multiple layers of uniformly and densely wound circular coils. The physical parameters of the six submodules EMx1, EMx2, EMy1, EMy2, EMz1, and EMz2 are completely identical. Each background electromagnet submodule is connected to a voltage sensing module at both ends.

[0080] The fingertip electromagnet module employs a spherical structure with three fingertip electromagnet sub-modules symmetrically placed in three orthogonal directions. For all fingertip electromagnet modules, with the geometric center of the module as the origin, the central axis of any one sub-module is selected as the X-axis, and the central axis of any other sub-module in any other direction is selected as the Y-axis. The Z-axis is determined according to the right-hand rule, establishing a spatial coordinate system {S}. i The 6D pose of the fingertip electromagnet module numbered i is (x... i y i , z i α i ,β i γ i ), where (x i y i , z i Let α be the position coordinates of the fingertip electromagnet module in the spatial coordinate system {G}. i ,β i γ i ) are respectively the fingertip electromagnet module orbiting the coordinate system {S i The rotation angles along the X, Y, and Z axes in the diagram. The fingertip electromagnet submodules in the x, y, and z directions of the module numbered i are named EM. FTix EM FTiy EM FTiz The submodules have identical physical parameters, with N2 turns, and each corresponds to one of three drive currents I. FTix I FTiy I FTiz The fingertip electromagnet submodule consists of two circular coils connected in series in the same direction, which overlap in space.

[0081] Furthermore, step two includes:

[0082] Step 2.1, based on electromagnetic theory, the fingertip electromagnet submodule EM FTix This can be considered as a magnetic dipole, generating an induced voltage U′ on the background electromagnet submodule EMx1. x1_x It can be represented as

[0083]

[0084] Where, ω 1x For EM FTix The frequency of the sinusoidal current, It is a constant, where μ0 is the permeability of air and m is the magnetic permeability of EM. FTix The magnitude of the magnetic moment; R = f(α) i ,β i γ i ) is EM FTix rotation matrix, It is EM FTix The unit direction vector of the magnetic moment, r represents the direction from EM. FTix A spatial vector pointing to a point on the EMx1 ring.

[0085] Furthermore, the fingertip electromagnet submodule EM can be obtained. FTiy EM FTiz The induced voltage U′ generated on the background electromagnet submodule EMx1 x1_y and U′ x1_y The expression.

[0086] Furthermore, the above expression can be extended to the remaining background electromagnet submodules.

[0087] Step 2.2: Based on the forward kinematics of the human hand, the human hand skeleton is considered as a linkage structure. When the palm is placed horizontally, a unique relationship between the fingertip position and posture can be obtained, and a fingertip pose data table can be constructed. The specific relationship expression is as follows:

[0088] [α i ,β i γ i ]=f(x i y i , z i L)

[0089] Where L = [l1, l2, l3, ... l n [ ] is the set of lengths of the phalanges and metacarpals of the hand.

[0090] Step three includes:

[0091] Step 3.1: Simulate the background electromagnet submodule EMx1, ensuring its current is within -I... max To I max The values ​​increase sequentially, ranging from 0 to L+2R1 on the x-axis, from -R1 to R1 on the y-axis, where R1 is the inner diameter of the background electromagnet, L is the height of the 3D background electromagnet module, and from -R1 to R1 on the z-axis to represent the simulation space, where R1 is the inner diameter length of the sub-module of the back electromagnet module.

[0092] Using the finite element method, the background magnetic field distribution within this space is obtained, and the force situation of the fingertip electromagnet submodule EMft1x is calculated. The specific relationship is expressed as follows:

[0093] F = f(x) i y i , z i α i ,β i γ i I x1 I FT1x I FT1y I FT1z )

[0094] Step 3.2: Extend the obtained offline simulation data of EMx1 on EMft1x to EMft1y and EMft1z to obtain the offline simulation data of EMx1 on the entire fingertip electromagnet EMft1.

[0095] Step 3.3: Extend the obtained offline simulation data of EMx1 to the fingertip electromagnet EMft1 to the other background electromagnets EMx2, Emy1...EMz2, so as to obtain the offline simulation data of the entire three-dimensional background electromagnet to EMft1 and all other background electromagnet sub-modules, thus obtaining the offline simulation data of the entire background electromagnet module.

[0096] Step four specifically involves placing the user's hand horizontally in the operating space with all fingers extended when the system is turned on. At this point, a first position detection is performed to obtain the number of fingertips to be detected, the lengths of the corresponding phalanges and metacarpals, and the initial 6D pose of each fingertip using an optical method.

[0097] Step five includes the following steps:

[0098] Step 5.1: The fingertip electromagnet module and the background electromagnet module are switched to position detection mode. The position detection mode refers to a mode with an amplitude of U. sigmax The sinusoidal drive current excites all fingertip electromagnet submodules EM FT1x EM FT1y EM FT1z EM FT2x …EM FTnz The frequencies are f 1x f 1y f 1z f 2x , ...f nz They are at the minimum signal frequency f min and maximum signal frequency f max The values ​​increase proportionally with each other; simultaneously, all drive channels of the background electromagnet submodules are shut down, and the voltage sensing module is turned on. The duration T of the position detection mode... s The time constant τ of the fingertip electromagnet submodule is greater than ft .

[0099] Step 5.2, to locate EM FT1 For example, to ensure the effectiveness of the magnetic dipole model, based on the previous positioning results, a distance EM is selected. FT1 The three more distant background electromagnet sub-modules, assuming they are EMx1, EMy1, and EMz1, have their voltage sensing modules obtaining the corresponding voltage U across the two ends of the background electromagnets. x1 U y1 U z1 The measured induced voltage is digitally filtered to separate the voltage components belonging to each fingertip electromagnet submodule.

[0100] For EMx1, a cost function is established based on the theoretically calculated and actually measured values ​​of the induced voltage:

[0101]

[0102] Where M x1 =(O x1x O x1y O x1z M x1 N x1 P x1 (x) represents the position and orientation of the geometric center of the background electromagnet submodule EMx1. The cost function solves for the fingertip electromagnet position as (x) 1_x1 y 1_x1 , z 1_x1 )

[0103] Construct EMY1 and EMz1 in the same way regarding EM FT1 The cost function, the solved positions of the fingertip electromagnets are (x 1_y1 y 1_y1 , z 1_y1 ), (x 1_z1 y 1_z1 , z 1_z1 ). Finger electromagnet submodule EM FT1 The position can be represented as:

[0104]

[0105]

[0106]

[0107] Use the same method to construct the cost function required for the positioning of the remaining fingertip electromagnets.

[0108] Step 5.3: Obtain the 6D pose of the palm using optical methods. Based on the relative position and pose relationship of the fingertips, and combining the 6D pose of the palm with the previous fingertip position, obtain the 3D pose of the fingertip to be detected.

[0109] Furthermore, by combining the 3D pose of the fingertips with the established cost function, an optimization algorithm is used to obtain the numerical solution of the three position variables of each cost function, thereby obtaining the 6D pose of all fingertip electromagnet modules.

[0110] Step six includes the following steps:

[0111] Step 6.1: Based on the force-tactile interaction model between a human finger and a virtual object in a virtual scene, the central control module calculates the target three-dimensional feedback force of the fingertip electromagnet module at the current moment. The fingertip electromagnet module and the background electromagnet module switch to force-tactile reproduction mode;

[0112] Step 6.2: Based on the offline simulation data and the given 6D pose of the fingertip electromagnet module and the target 3D feedback force, the central control module calculates the driving current of all background electromagnets and fingertip electromagnets, and outputs a PWM signal to control the drive module to adjust the driving current of the fingertip electromagnet module and the background electromagnet module.

[0113] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A visual and electromagnetic fusion position detection method for force-tactile human-computer interaction, characterized in that, The method includes: Step 1: Establish a virtual scene including virtual hand and virtual object models, and establish the spatial mapping relationship between the real operating space inside the background electromagnet module and the virtual scene, as well as the three-dimensional positional mapping relationship between the fingertip electromagnet module and the virtual hand. Step 2: Based on the principle of electromagnetic induction, obtain the relationship between the 6D pose of the fingertip electromagnet submodule and the induced voltage of each background electromagnet submodule; based on the forward kinematics of the human hand, obtain the constraint relationship between the 3D position and 3D posture of the fingertip relative to the center of the palm, and construct the fingertip pose data table. Step 3: Simulate the mapping relationship between the 6D pose, driving current, and background electromagnet module driving current of the fingertip electromagnet module and the force of each magnetic moment component of the fingertip electromagnet module, and obtain its offline simulation data. Step 4: Obtain the number of fingertips to be detected and the initial 6D pose using optical methods; Step 5: The fingertip electromagnet module and the background electromagnet module are switched to position detection mode. The voltage sensing module obtains the voltage at both ends of the background electromagnet and obtains the induced voltage of each fingertip electromagnet sub-module after filtering. The 6D pose of the center of the palm is obtained according to the optical method. Combined with the previous or initial 3D position of the fingertip electromagnet module, the 3D pose of the fingertip electromagnet module is obtained by querying the fingertip pose data table. Based on the 3D pose of the fingertip electromagnet module and the induced voltage of the background electromagnet, a cost function is established and solved using an iterative method to finally obtain the 6D pose of the fingertip electromagnet module. Step 6: Based on the force-tactile interaction model between the fingertip and the virtual object in the virtual scene, calculate the target 3D feedback force of the fingertip electromagnet module at the current moment. The fingertip electromagnet module and the background electromagnet module are switched to the force tactile reproduction mode. Based on the offline simulation data and the given 6D pose of the fingertip electromagnet module and the target three-dimensional feedback force, the required driving current of the fingertip electromagnet module and the background electromagnet module is obtained. The background electromagnet module consists of three pairs of parallel hollow cylindrical electromagnet sub-modules placed orthogonally. The spacing between each pair of electromagnet sub-modules is the same and equal to their respective outer diameters, and the center points of the central axes of each pair coincide. Taking the geometric center of the background electromagnet module as the origin, the central axis of any one sub-module is set as the X-axis, and the central axis of any other sub-module in any other direction is set as the Y-axis. The Z-axis is determined according to the right-hand rule, and a spatial coordinate system {G} is established. The background electromagnet submodule is a hollow cylindrical coil made of multiple layers of uniformly and densely wound circular coils, with N1 turns. The physical parameters of the six submodules EMx1, EMx2, EMy1, EMy2, EMz1, and EMz2 are completely identical. Each background electromagnet submodule is connected to a voltage sensing module at both ends. The fingertip electromagnet module employs a spherical structure with three fingertip electromagnet sub-modules symmetrically placed in three orthogonal directions. For all fingertip electromagnet modules, with the geometric center of the module as the origin, the central axis of any one sub-module is set as the X-axis, and the central axis of any other sub-module in any other direction is set as the Y-axis. The Z-axis is determined according to the right-hand rule, establishing a spatial coordinate system { }; The 6D pose of the fingertip electromagnet module numbered i is ( ), in Let G be the position coordinates of the fingertip electromagnet module in the spatial coordinate system {G}. The fingertip electromagnet module revolves around the coordinate system { The rotation angles along the X, Y, and Z axes; the fingertip electromagnet module numbered i is named... Its fingertip electromagnet submodules in the x, y, and z directions are respectively named , , Each fingertip electromagnet submodule has identical physical parameters and has N2 turns. The fingertip electromagnet submodule is composed of two circular coils connected in series in the same direction, which overlap in space.

2. The visual and electromagnetic fusion position detection method according to claim 1, characterized in that, Based on the forward kinematics of the human hand, the human hand skeleton is treated as a linkage structure. When the palm is placed horizontally, the relationship between the fingertip position and posture is determined, and a fingertip pose data table is constructed. The expression for the relationship between the fingertip position and posture is as follows: ； in It is the collection of the lengths of the finger bones and metacarpal bones of the hand.

3. The visual and electromagnetic fusion position detection method according to claim 2, characterized in that, Step four specifically involves, when the system is turned on, placing the hand horizontally in the operating space with all fingers extended, performing the first position detection, and obtaining the number of fingertips to be detected and the corresponding set of lengths of each phalanx and metacarpal bone using an optical method. And the initial 6D pose of each fingertip to be tested.

4. The visual and electromagnetic fusion position detection method according to claim 2, characterized in that, Step five includes: Step 5.1: The fingertip electromagnet module and the background electromagnet module are switched to position detection mode. The position detection mode uses an amplitude value... A sinusoidal driving current excites all fingertip electromagnet submodules, and the frequencies of each fingertip electromagnet submodule are respectively... , , , ,… minimum signal frequency and maximum signal frequency They exhibit an equal and increasing relationship; Close all drive channels of the background electromagnet submodules and turn on the voltage sensing module; duration of position detection mode. The time constant of the fingertip electromagnet submodule is greater than that of the fingertip electromagnet submodule. ; Step 5.2, for positioning For example, based on the previous location results, select the distance. The three more distant background electromagnet sub-modules, designated EMx1, EMy1, and EMz1, have their voltage sensing modules obtaining the corresponding voltages across the background electromagnets. , , The measured induced voltage is digitally filtered to separate the voltage components belonging to each fingertip electromagnet submodule. For EMx1, a cost function is established based on the theoretically calculated and actually measured values ​​of the induced voltage: ； in The position and orientation of the geometric center of the background electromagnet submodule EMx1 are represented by the cost function; the position of the fingertip electromagnet is solved as follows: ); Construct EMy1 and EMz1 in the same way regarding The cost function, the solved positions of the fingertip electromagnets are ( ), ( Fingertip electromagnet submodule The position is represented as: ； ； ； Construct the cost function required for the positioning of the remaining fingertip electromagnets using the same method; Step 5.3: Use optical methods to obtain the 6D pose of the palm. Combine the 6D pose of the palm with the previous fingertip position, query the fingertip pose data table, and obtain the 3D pose of the fingertip to be detected.

5. The visual and electromagnetic fusion position detection method according to claim 2, characterized in that, By combining the 3D pose of the fingertip with the established cost function, an optimization algorithm is used to obtain the numerical solution of the three position variables of each cost function, thereby obtaining the 6D pose of all fingertip electromagnet modules.

6. A non-contact force-tactile reproduction system, characterized in that, The tactile reproduction system includes a background electromagnet module, a fingertip electromagnet module, a camera module, a voltage sensing module, an electromagnet drive module, a power supply module, and a central control module. The background electromagnet module comprises 3 pairs of spatially orthogonal 6 background electromagnet sub-modules, with two electromagnet sub-modules in each direction; The fingertip electromagnet module adopts a spherical structure in which three fingertip electromagnet sub-modules are symmetrically placed in three orthogonal directions; The voltage sensing module includes a voltage limiting module and a voltage sensor. The voltage limiting module limits the input signal of the voltage sensor to no more than the maximum value. The voltage sensor converts the analog signal into a digital signal. The electromagnet drive module includes an electromagnet control submodule and an electromagnet drive submodule. The electromagnet control submodule is used to receive control information from the central control module and generate PWM signals to control the electromagnet drive submodule to output the required electrical signals. The central control module obtains the 6D position and pose of the fingertip according to the position detection method as described in claim 1, calculates the interaction force of the virtual hand in the virtual environment, and outputs a drive signal.

Images