A Virtual Hand Grasping Perception and Exoskeleton Force Feedback Mapping Method Based on Virtual-Real Interaction

Abstract

This invention relates to a virtual hand grasping perception and exoskeleton force feedback mapping method based on virtual-real interaction, belonging to the field of virtual reality technology. The method includes: constructing a hand exoskeleton and reading joint data in real time; calculating the pose of each joint coordinate system of the virtual hand relative to the virtual world; driving the virtual hand to achieve real-time synchronized movement with the hand exoskeleton based on the calculated pose of each joint; monitoring collision and contact events between the virtual hand and digital objects in real time, and calculating the virtual interaction force when the virtual hand interacts with digital objects; calculating the required feedback torque output of each exoskeleton motor, and driving the motors to output the corresponding feedback force through the motor controller of the hand exoskeleton, thereby achieving force feedback to the human hand. This invention can enhance the realism of interaction in virtual reality applications, enabling users to interact with the virtual environment more naturally and immersively, thus improving the user experience.

Description

Technical Field

[0001] This invention belongs to the field of virtual reality technology, and in particular relates to a method for virtual hand grasping perception and exoskeleton force feedback mapping based on virtual-real interaction. Background Technology

[0002] With the rapid development of virtual reality technology, people's demand for more realistic and natural interactive experiences in virtual environments is becoming increasingly urgent. In many application scenarios, such as virtual assembly, surgical simulation, and rehabilitation training, users need to interact with digital objects through hand operations and hope to achieve the force feedback of real-world object manipulation.

[0003] Traditional virtual reality interaction methods mostly focus on visual presentation. Although users can see the interaction process between the virtual hand and digital objects, they lack real tactile feedback, making the interactive experience less immersive and realistic.

[0004] As a wearable device, hand exoskeletons have the potential to provide force feedback to users. However, existing technologies have failed to effectively combine the interactive forces in virtual scenes with the driving force of hand exoskeletons, and cannot accurately simulate the various force feedbacks felt when a human hand grasps and manipulates real objects, thus limiting the in-depth application of virtual-real fusion interaction technology in many fields. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a virtual reality technology that can enhance the realism of interaction in virtual reality applications, enabling users to interact with the virtual environment more naturally and immersively, thereby improving the user experience.

[0006] This invention proposes a method for mapping virtual hand grasping perception and exoskeleton force feedback based on virtual-real interaction, including:

[0007] S1, Construct a hand exoskeleton and read the joint data of the hand exoskeleton in real time under the operation of an external person;

[0008] S2, using the robotics forward kinematics algorithm on the collected joint data, and based on the human hand kinematics model, calculates the pose of each joint coordinate system of the virtual hand relative to the virtual world.

[0009] S3, Construct a virtual scene, which includes a virtual hand and digital objects. Based on the calculated poses of each joint of the virtual hand, drive the virtual hand to achieve real-time synchronized movement with the hand exoskeleton, and display the movement status of the virtual hand and digital objects.

[0010] S4, Real-time monitoring of collision and contact events between the virtual hand and digital objects, and calculation of virtual interaction force when the virtual hand interacts with the digital object based on the physical properties of the digital object;

[0011] S5 maps the interactive force received by the virtual hand to each phalanx of the hand exoskeleton. Based on the preset force mapping model, the interactive force received by the virtual hand is mapped to the joint feedback torque of the hand exoskeleton. The required feedback torque output by each hand exoskeleton motor is calculated. The motor controller of the hand exoskeleton drives the motor to output the corresponding feedback force, thereby realizing the force feedback to the hand of the external person.

[0012] Furthermore, the hand exoskeleton includes: a wrist link, a wrist joint motor, a palm link, a thumb metacarpophalangeal joint first degree of freedom joint motor, a thumb proximal phalanx link, a thumb metacarpophalangeal joint motor, a thumb distal metacarpophalangeal phalanx link, an index finger metacarpophalangeal joint motor, a middle finger metacarpophalangeal joint motor, a ring finger metacarpophalangeal joint motor, a little finger metacarpophalangeal joint motor, an index finger proximal phalanx link, a middle finger proximal phalanx link, a ring finger proximal phalanx link, a little finger proximal phalanx link, and an index finger... Proximal interphalangeal joint motor, middle finger proximal interphalangeal joint motor, ring finger proximal interphalangeal joint motor, little finger proximal interphalangeal joint motor, index finger middle phalangeal joint connecting rod, middle finger middle phalangeal joint connecting rod, ring finger middle phalangeal joint connecting rod, little finger middle phalangeal joint connecting rod, index finger distal interphalangeal joint motor, middle finger distal interphalangeal joint motor, ring finger distal interphalangeal joint motor, little finger distal interphalangeal joint motor, index finger distal phalangeal joint connecting rod, middle finger distal phalangeal joint connecting rod, ring finger distal phalangeal joint connecting rod, little finger distal phalangeal joint connecting rod.

[0013] Furthermore, in S1, a motor encoder is installed on each finger joint and key part of the palm of the hand exoskeleton, and a DC motor and a motor controller are configured. The motor encoder measures the joint angle, the DC motor serves as the joint drive source, and the motor controller performs closed-loop control of the speed, position, and torque of the DC motor.

[0014] Furthermore, in S1, the joint data includes: joint angle and torque data of the hand exoskeleton.

[0015] Furthermore, the joint data is updated to the cache at a preset frequency.

[0016] Further, in S2, calculating the pose of each joint coordinate system of the virtual hand relative to the virtual world includes:

[0017]

[0018] In real physical space, a set of pose matrices for all joint coordinate systems of the hand exoskeleton relative to the base system, where i = 1, 2, ... n;

[0019] θ = [θ1, θ2, θ3, ... θ n ]T : is the joint angle vector.

[0020] f DK The robot's positive kinematics function depends on the configuration of the hand exoskeleton;

[0021]

[0022] in,

[0023] In virtual space, a set of pose matrices for all joint coordinate systems relative to the virtual coordinate system, where i = 1, 2, ... n;

[0024] Mapping the joint poses of a hand exoskeleton in the real physical world to a pose transformation matrix in virtual space;

[0025] n: The total number of joints in the exoskeleton of the hand.

[0026] Furthermore, in the virtual scene, the virtual hand and digital object are presented as 3D particles.

[0027] Furthermore, in S4, the physical properties of the digital object include: the object's mass, material properties, and motion state.

[0028] Furthermore, in S4, when monitoring collision and contact events between the virtual hand and digital objects, the virtual engine calculates in real time the pose, shape, and physical state of the particles with physical properties of the digital object at the current moment.

[0029] Furthermore, in S5, the calculation of the required feedback torque output by each hand exoskeleton motor based on the mapped force information includes: using a preset force mapping model:

[0030] f i，real_contact =f virtual_force_projection (x p F p )

[0031] in,

[0032] f i，real_contact : A set of equivalent contact force vectors at specific contact positions of each link of the hand exoskeleton obtained by mapping, i.e., f i，real_contact ∈R 3 , where R is the set of real numbers, R 3 Represents a three-dimensional Euclidean space, where i is the index number of the triggerable contact position of the hand exoskeleton;

[0033] x p The spatial coordinates of the particles at the contact point of the virtual hand, i.e., xp ∈R 3 , where p is the particle index number of the virtual hand contacting the part of the particle;

[0034] F p,virtual : The deformation gradient matrix of the particles in the contact area of ​​the virtual hand, where p is the index number of the particles in the contact area of ​​the virtual hand;

[0035] f virtual_force_projection A function that maps the deformation of a virtual hand to the equivalent contact force of the hand exoskeleton;

[0036] τ=J T f i，real_contact

[0037] in,

[0038] τ=[τ1,τ2,τ3,...τ n ] T : Exoskeleton joint feedback torque vector, where τ1, τ2, τ3, ... τ n These are the feedback torque values ​​for the 1st to nth joints, where n is the total number of joints for which the exoskeleton must provide feedback force.

[0039] J: The Jacobian matrix of the exoskeleton robot's kinematics, determined by the configuration of the exoskeleton;

[0040] f i，real_contact : Equivalent contact force vector of the exoskeleton, where i is the contact point index;

[0041] The required feedback torque output for the motors of each hand exoskeleton is calculated as follows:

[0042]

[0043] Where, τ motor τ: Motor torque of the hand exoskeleton; joint τ: Joint torque of the hand exoskeleton; friction : Frictional torque of the joint transmission mechanism of the hand exoskeleton; I: Equivalent rotational inertia of the joints of the hand exoskeleton; α: Joint angular acceleration.

[0044] The beneficial effects of this invention are as follows:

[0045] The virtual hand grasping perception and exoskeleton force feedback mapping method based on virtual-real interaction of the present invention is an innovative algorithm that can calculate the interaction force between the virtual hand and objects in virtual-real scenes in real time and accurately, and convert it into the force exerted by the exoskeleton on the human hand. This greatly enhances the user's sense of immersion and realism in the virtual reality environment and expands the practical value of related technologies in multiple fields.

[0046] 1. The method of the present invention greatly enhances the realism of interaction in virtual reality applications, enabling users to interact with the virtual environment more naturally and immersively, thereby improving the user experience.

[0047] 2. In professional fields such as virtual assembly and surgical simulation, the method of the present invention allows operators to perform complex operations more accurately with precise force and posture feedback, thereby improving work efficiency and quality and reducing the risk of operational errors.

[0048] 3. The method of the present invention provides a more effective means of rehabilitation training. When patients are undergoing hand rehabilitation training, they can feel real resistance and assistance through interaction with digital objects, which promotes the recovery of hand function.

[0049] 4. Compared with traditional virtual reality interactive devices, the method of this invention combined with a hand exoskeleton expands the interaction dimension, has higher technological added value, and is expected to promote the development of related industries. Attached Figure Description

[0050] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0051] Figure 1 This is a flowchart of a virtual hand grasping perception and exoskeleton force feedback mapping method based on virtual-real interaction according to an embodiment of the present invention;

[0052] Figure 2 This is a flowchart illustrating the virtual-real interaction process of a virtual hand grasping perception and exoskeleton force feedback mapping method based on virtual-real interaction according to an embodiment of the present invention.

[0053] Figure 3 This is a schematic diagram of the hand exoskeleton joint linkage structure according to an embodiment of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0055] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0056] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0058] This invention proposes a virtual hand grasping perception and exoskeleton force feedback mapping method based on virtual-real interaction, involving the fields of virtual reality, physical simulation, human-computer interaction and robot control technology. It is a method that can accurately calculate and convert the interaction force between the virtual hand and the object into the force exerted by the exoskeleton on the human hand under the fusion of virtual and real scenes.

[0059] like Figure 1 and Figure 2 As shown, this invention proposes a method for mapping virtual hand gripping perception and exoskeleton force feedback based on virtual-real interaction, including:

[0060] S1, build a hand exoskeleton and read joint data of the hand exoskeleton in real time under the operation of external personnel.

[0061] First, the hardware used in the method of this invention will be described:

[0062] (1) Hand exoskeleton: The exoskeleton frame is made of lightweight, high-strength materials to ensure comfortable wear and to not hinder natural hand movement. Appropriate range motor encoders (accuracy up to 0.1°) and communication methods for measuring motor torque (accuracy 0.1 N·m) are installed at each finger joint and key parts of the palm. A small, high-torque DC motor is equipped as the joint drive source. The motor controller has fast response and precise control capabilities, and can realize closed-loop control of motor speed, position and torque.

[0063] (2) Build a computer system, selecting a high-performance graphics workstation equipped with a professional graphics card (such as the NVIDIA RTX series) to support real-time rendering of complex virtual scenes. The processor should be a multi-core, high-frequency CPU (such as the Intel Core i9 series) for fast algorithm execution. It should also be equipped with high-speed memory (32GB or more) and a large-capacity solid-state drive (1TB or more) to meet data storage and fast read / write requirements. The workstation should be equipped with an operating system supporting virtual reality development (such as Ubuntu 18.04 or 20.04) and the corresponding development software platform, as well as a hardware interface adapted for communication with the exoskeleton.

[0064] The following is for reference. Figure 3 The structure of the hand exoskeleton is described. Figure 3 The cylinders and cuboids in the diagram represent the joint motors and linkages of the hand exoskeleton, respectively, and correspond to the anatomical structure of the human hand.

[0065] like Figure 3 As shown, the hand exoskeleton includes: a wrist link, a wrist joint motor, a palm link, a thumb metacarpophalangeal joint motor (first degree of freedom), a thumb proximal phalanx link, a thumb metacarpophalangeal joint motor, a thumb distal metacarpophalangeal joint link, an index finger metacarpophalangeal joint motor, a middle finger metacarpophalangeal joint motor, a ring finger metacarpophalangeal joint motor, a little finger metacarpophalangeal joint motor, an index finger proximal phalanx link, a middle finger proximal phalanx link, a ring finger proximal phalanx link, a little finger proximal phalanx link, and a proximal link of the index finger. Knuckle motors, proximal knuckle motors of the middle finger, proximal knuckle motors of the ring finger, proximal knuckle motors of the little finger, middle phalanx link of the index finger, middle phalanx link of the middle finger, middle phalanx link of the ring finger, middle phalanx link of the little finger, distal knuckle motors of the index finger, distal knuckle motors of the middle finger, distal knuckle motors of the ring finger, distal knuckle motors of the little finger, distal phalanx link of the index finger, distal phalanx link of the middle finger, distal phalanx link of the ring finger, distal phalanx link of the little finger.

[0066] Motor encoders are installed on each finger joint and key parts of the palm of the hand exoskeleton, and a DC motor and motor controller are configured. The motor encoders measure the joint angles, the DC motors serve as the joint drive source, and the motor controllers provide closed-loop control of the speed, position, and torque of the DC motors.

[0067] An external computer interacts with the hand exoskeleton sensor (i.e., motor encoder) and motor controller via serial communication or wireless communication protocol stack API, reads joint data in real time, and updates the collected joint data to the buffer at a preset frequency.

[0068] The joint data includes: joint angles and torques of the hand exoskeleton. The joint data is updated to the system cache at a preset frequency (e.g., 100Hz).

[0069] S2 uses a robotic direct kinematics algorithm to calculate the pose of each joint coordinate system of the virtual hand relative to the virtual world, based on the human hand kinematics model, on the collected joint data. The robotic direct kinematics algorithm takes joint angles as input and outputs the pose of the joint system relative to the base system.

[0070] Figure 2 It shows the complete process from the acquisition of hand exoskeleton data, through virtual hand (digital hand) pose calculation, virtual hand-object interaction force calculation, and finally to the calculation and application of hand exoskeleton feedback torque.

[0071] refer to Figure 2 Based on a human hand kinematics database, the collected joint data is used as input, and a robotics forward kinematics algorithm is employed to solve for the target angles of each joint in the virtual hand. Here, a joint angle refers to the relative rotation angle between the two links connecting the front and rear ends of a joint in the mathematical model. The target angle refers to the "final angle" to which the joint is to rotate from its initial angle.

[0072] After receiving joint data (i.e., joint angle and torque data) from the hand exoskeleton, the algorithm in the computer system quickly calculates the pose of each joint of the virtual finger based on the human hand kinematics model. The pose is the position and orientation of the coordinate system attached to the finger link relative to the base coordinate system. The specific method is as follows:

[0073] Specifically, calculating the pose of each joint coordinate system of the virtual hand relative to the virtual world includes:

[0074]

[0075] in,

[0076] In real physical space, a set of pose transformation matrices for all joint coordinate systems of the hand exoskeleton relative to the base system (i.e., SE(3) is short for the three-dimensional special Euclidean group, that is, this matrix It is a group element in Special Euclidean 3Group.

[0077] Where i = 1, 2, ..., n;

[0078] θ = [θ1, θ2, θ3, ... θ n ] T : is the joint angle vector.

[0079] θ1,θ2,θ3,...θ n These are the joint angles.

[0080] n represents the total number of joints in the hand exoskeleton.

[0081] f DK In robotics, forward kinematics functions depend on the specific robot configuration (i.e., the mechanical structure and geometric parameters of the joints and links). Here, forward kinematics is used to calculate the pose of each link coordinate system relative to the base system using the joint angles of the robotic arm and manipulator.

[0082]

[0083] in,

[0084] In virtual space, a set of pose transformation matrices for all joint coordinate systems relative to the virtual coordinate system. That is, ), where i is the index number of the virtual joint, i = 1, 2, ... n; the virtual joints correspond one-to-one with the physical exoskeleton joints.

[0085] Mapping the joint poses of a hand exoskeleton in the real physical world to a pose transformation matrix in virtual space. That is, The purpose of the pose transformation matrix is ​​to cause the applied matrix to undergo rotation or translation. This 4x4 matrix belongs to SE(3), which is the three-dimensional special Euclidean group.

[0086] This process is achieved by applying mathematical methods such as coordinate transformation and robot kinematics to ensure that the virtual hand's movements are highly synchronized with the user's actual hand movements.

[0087] S3. Construct a virtual scene, which includes a virtual hand and digital objects. Based on the calculated poses of each joint of the virtual hand, drive the virtual hand to achieve real-time synchronized movement with the hand exoskeleton, and display the movement status of the virtual hand and digital objects.

[0088] In the virtual scene, the virtual hand and digital objects are presented as 3D particles. An animation system is used to drive the virtual hand model to move in real time in sync with the physical hand exoskeleton.

[0089] S4 monitors collision and contact events between the virtual hand and digital objects in real time, and calculates the virtual interaction force when the virtual hand interacts with the digital object based on the physical properties of the digital object.

[0090] The virtual scene is built using an advanced graphics rendering engine, and the digital objects within it all possess rich and configurable physical properties. Developers or users can assign physical properties such as rigid body, elastic body, plastic body, elasto-plastic material, viscoelastic body, viscoplastic body, granular material, and fluid to the digital objects according to different application needs. These properties will determine the mechanical behavior of the digital objects when interacting with the virtual hand.

[0091] This invention utilizes a specially designed physics engine and a simulation algorithm based on physical principles to configure physical attributes for digital objects, monitor collision and contact events between the virtual hand and digital objects in real time, and calculate the interaction force on the virtual hand based on parameters such as the object's mass, material properties, and motion state.

[0092] In embodiments of the present invention, the physical properties of a digital object include: the object's mass, material properties, and motion state.

[0093] When monitoring collision and contact events between the virtual hand and digital objects, the virtual engine calculates in real time the pose, shape, and physical state of the particles with physical properties of the digital object at the current moment.

[0094] Specifically, when a person wears a hand exoskeleton and moves in physical space, the virtual hand interacts with digital objects in the virtual scene, performing actions such as collision, grasping, holding, twisting, and pinching. At this time, the physics engine in the virtual scene, based on the physical properties of the digital objects and the motion state of the virtual hand, uses principles such as conservation of momentum, collision detection, and contact mechanics to accurately calculate the forces acting on the virtual hand at each moment of interaction. These forces include detailed information such as magnitude, direction, and point of application, which forms the basis for subsequent conversion into feedback forces from the hand exoskeleton.

[0095] S5 maps the interactive force received by the virtual hand to each phalanx of the hand exoskeleton. Based on the preset force mapping model, the interactive force received by the virtual hand is mapped to the joint feedback torque of the hand exoskeleton. The required feedback torque output by each hand exoskeleton motor is calculated. The motor controller of the hand exoskeleton drives the motor to output the corresponding feedback force, thereby realizing the force feedback to the hand of the external person.

[0096] Specifically, based on the mapped force information, the required feedback torque output of each hand exoskeleton motor is calculated, including: through a preset force mapping model:

[0097] f i，real_contact =f virtual_force_projection (x p F p (3)

[0098] in,

[0099] f i，real_contact : A set of three-dimensional equivalent contact force vectors (i.e., f) at specific contact positions of each link of the hand exoskeleton obtained by mapping. i，real_contact ∈R 3 Where R is the set of real numbers, R 3 It represents "3-dimensional Euclidean space";

[0100] Where i is the index number of the triggerable contact position of the hand exoskeleton;

[0101] x p The spatial coordinates of the particles at the contact point of the virtual hand (i.e., x) p ∈R 3 ), where p is the particle index number;

[0102] F p,virtual : The deformation gradient matrix of the particles in the contact portion of the virtual hand, where p is the index number of the particles in the contact portion; this deformation gradient matrix is ​​referenced to the deformation gradient of continuum mechanics.

[0103] f virtual_force_projection The specific steps for mapping the deformation of the virtual hand to a function of the equivalent contact force of the hand exoskeleton are as follows:

[0104] The first step is to filter out all particles within a distance r of the i-th specific contact position (usually several are set on each finger link) xi, and store their indices in the neighborhood list L. i :

[0105] 1. Initialize the neighborhood list

[0106] 2. For each particle p∈P in the contact region, perform the following operation:

[0107] (1) Calculate particle x p With x i Euclidean distance:

[0108] d=||x p -x i ||

[0109] (2) If d < r, add the particle p to the neighborhood list:

[0110] L i ← L i ∪ {p}

[0111] 3. End the loop

[0112] 4. Return the neighborhood list L i

[0113] The second step is to solve the comprehensive deformation ratio vector near the i-th contact position

[0114] 1. Initialize the accumulation vector:

[0115] sum_vec ← [0, 0, 0] T

[0116] 2. Obtain the number of neighborhood force particles:

[0117] N ← L i |(|·| represents the size of the set)

[0118] 3. For each particle j ∈ L i Perform the following operations:

[0119] (1) Extract the diagonal elements of F j to construct a three-dimensional vector:

[0120] v j ← [F j,11 [[ID=5,0]], F j,22 , F j,33 T

[0121] (where F j,mn represents the element in the m-th row and n-th column of F j )

[0122] (2) Accumulate:

[0123] sum_vec ← sum_vec + v j

[0124] 4. Calculate the average vector:

[0125]

[0126] The third step is to calculate the equivalent contact force vector at the i-th contact position:

[0127]

[0128] τ = J T f i，real_contact

[0129] in,

[0130] τ=[τ1,τ2,τ3,...τ n ] T : Exoskeleton joint feedback torque vector;

[0131] Where τ1, τ2, τ3, ..., τ n These are the feedback torque values ​​for the 1st to nth joints, where n is the total number of joints for which the exoskeleton must provide feedback force.

[0132] J: The Jacobian matrix of the exoskeleton robot's kinematics, determined by the configuration of the exoskeleton;

[0133] f i，real_contact : Equivalent contact force vector of the exoskeleton; where i is the contact point index.

[0134] The model maps the interactive forces experienced by the virtual hand to the individual phalanges of the physical hand exoskeleton. It considers the anatomical structure of the human hand, the mechanical structure of the exoskeleton, and the kinematic transmission relationship between the two, ensuring that the force mapping is reasonable and accurate.

[0135] Based on the mapped force information and combined with the exoskeleton's dynamic model, the required feedback torque output of the motors for each hand exoskeleton is calculated as follows:

[0136]

[0137] Where, τ motor Exoskeleton motor torque (unit: N·m);

[0138] τ joint Exoskeleton joint torque (unit: N·m);

[0139] τ friction Frictional torque of exoskeleton joint transmission mechanism (unit: N·m, which can be obtained through experiments or manufacturer data);

[0140] I: Equivalent rotational inertia of exoskeleton joints (unit: kg·m) 2 );

[0141] α: Joint angular acceleration (unit: rad / s) 2 ).

[0142] The motor drive control system adjusts the motor output in real time based on the calculation results, so that the exoskeleton of the hand applies corresponding force to the human hand, allowing the user to truly feel the tactile sensation as if operating a real object.

[0143] Based on the calculated virtual feedback force τ j The corresponding motor input torque τmotor The force is sent to the motor in real time, and people can feel the feedback force in real time. If you grip it tightly, you will feel the feedback force increase, and if you grip it loosely, you will feel the force decrease.

[0144] In general, a virtual hand can be understood as one link in providing a sensory loop. This is essentially a visual interaction that gives users a realistic, immersive experience. Users see that they can manipulate the virtual hand to grasp objects, and based on what they see—the virtual hand grasping an object—the brain also makes a prediction (what kind of tactile sensation the hand should experience upon grasping this object). Then, when they feel the "calculated exoskeleton feedback force," they will believe they have grasped the virtual object, thus forming a sensory loop.

[0145] In summary, this step converts the virtual hand interaction force into target torque values ​​for the motors of each joint of the exoskeleton based on the pre-established force mapping model. Through the torque control mode of the motor controller, the motor is driven to output the corresponding force, thereby realizing force feedback to the human hand.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A method for mapping virtual hand grasping perception and exoskeleton force feedback based on virtual-real interaction, characterized in that, include: S1, Construct a hand exoskeleton and read the joint data of the hand exoskeleton in real time under the operation of an external person; S2, using the robotics forward kinematics algorithm on the collected joint data, and based on the human hand kinematics model, calculates the pose of each joint coordinate system of the virtual hand relative to the virtual world. S3, Construct a virtual scene, which includes a virtual hand and digital objects. Based on the calculated poses of each joint of the virtual hand, drive the virtual hand to achieve real-time synchronized movement with the hand exoskeleton, and display the movement status of the virtual hand and digital objects. In the virtual scene, the virtual hand and digital objects present a 3D particle state. S4, Real-time monitoring of collision and contact events between the virtual hand and digital objects, and calculation of virtual interaction force when the virtual hand interacts with the digital object based on the physical properties of the digital object; S5 maps the interactive force received by the virtual hand to each phalanx of the hand exoskeleton. Based on the preset force mapping model, the interactive force received by the virtual hand is mapped to the joint feedback torque of the hand exoskeleton. The required feedback torque of each hand exoskeleton motor is calculated. The motor controller of the hand exoskeleton drives the motor to output the corresponding feedback force, thereby realizing the force feedback to the hand of the external person. The preset force mapping model includes: in, : A set of equivalent contact force vectors at specific contact positions of each link of the hand exoskeleton obtained by mapping, i.e. Where R is the set of real numbers, Represents three-dimensional Euclidean space, where i is the index number of the triggerable contact position of the hand exoskeleton; The spatial coordinates of the particles in the contact area of ​​the virtual hand, i.e. , where p is the particle index number of the virtual hand contacting the part of the particle; : The deformation gradient matrix of the particles in the contact area of ​​the virtual hand, where p is the index number of the particles in the contact area of ​​the virtual hand; The specific steps for mapping the deformation of the virtual hand to a function of the equivalent contact force of the hand exoskeleton are as follows: The first step is to filter out all particles within a distance range of the i-th specific contact position xi and store their indices in the neighborhood list. ; The second step is to solve for the comprehensive deformation ratio vector near the i-th contact position. ; The third step is to calculate the equivalent contact force vector at the i-th contact position; in, : Exoskeleton joint feedback torque vector These are the feedback torque values ​​for the 1st to nth joints, where n is the total number of joints for which the exoskeleton must provide feedback force. The Jacobian matrix of the exoskeleton robot's kinematics is determined by the configuration of the exoskeleton. : Equivalent contact force vector of the exoskeleton, where i is the contact point index; The required feedback torque output for the motors of each hand exoskeleton is calculated as follows: in, : Motor torque of the hand exoskeleton; Joint torques of the hand exoskeleton; Frictional torque of the joint transmission mechanism of the hand exoskeleton; α: Equivalent rotational inertia of the hand exoskeleton joints; α: Joint angular acceleration.

2. The method for virtual hand grasping perception and exoskeleton force feedback mapping based on virtual-real interaction according to claim 1, characterized in that, The hand exoskeleton includes: a wrist link, a wrist joint motor, a palm link, a thumb metacarpophalangeal joint first degree of freedom joint motor, a thumb proximal phalanx link, a thumb metacarpophalangeal joint motor, a thumb distal metacarpophalangeal joint link, an index finger metacarpophalangeal joint motor, a middle finger metacarpophalangeal joint motor, a ring finger metacarpophalangeal joint motor, a little finger metacarpophalangeal joint motor, an index finger proximal phalanx link, a middle finger proximal phalanx link, a ring finger proximal phalanx link, a little finger proximal phalanx link, and a proximal link of the index finger. Knuckle motors, proximal knuckle motors of the middle finger, proximal knuckle motors of the ring finger, proximal knuckle motors of the little finger, middle phalanx link of the index finger, middle phalanx link of the middle finger, middle phalanx link of the ring finger, middle phalanx link of the little finger, distal knuckle motors of the index finger, distal knuckle motors of the middle finger, distal knuckle motors of the ring finger, distal knuckle motors of the little finger, distal phalanx link of the index finger, distal phalanx link of the middle finger, distal phalanx link of the ring finger, distal phalanx link of the little finger.

3. The method for virtual hand grasping perception and exoskeleton force feedback mapping based on virtual-real interaction according to claim 1, characterized in that, In S1, a motor encoder is installed on each finger joint and key part of the palm of the hand exoskeleton, and a DC motor and a motor controller are configured. The motor encoder measures the joint angle, the DC motor serves as the joint drive source, and the motor controller performs closed-loop control of the speed, position, and torque of the DC motor.

4. The method for virtual hand grasping perception and exoskeleton force feedback mapping based on virtual-real interaction according to claim 1, characterized in that, In S1, the joint data includes: the joint angles and torque data of the hand exoskeleton.

5. The virtual hand grasping perception and exoskeleton force feedback mapping method based on virtual-real interaction according to claim 4, characterized in that, The joint data is updated to the cache at a preset frequency.

6. The method for virtual hand grasping perception and exoskeleton force feedback mapping based on virtual-real interaction according to claim 1, characterized in that, In S2, calculating the pose of each joint coordinate system of the virtual hand relative to the virtual world includes: In real physical space, a set of pose matrices for all joint coordinate systems of the hand exoskeleton relative to the base system, where i = 1, 2, ... n; = : is the joint angle vector; The robot's positive kinematics function depends on the configuration of the hand exoskeleton; in, In virtual space, a set of pose matrices for all joint coordinate systems relative to the virtual coordinate system, where i = 1, 2, ... n; Mapping the joint poses of a hand exoskeleton in the real physical world to a pose transformation matrix in virtual space; n: The total number of joints in the exoskeleton of the hand.

7. The method for virtual hand grasping perception and exoskeleton force feedback mapping based on virtual-real interaction according to claim 1, characterized in that, In S4, the physical properties of the digital object include: the object's mass, material properties, and motion state.

8. The method for virtual hand grasping perception and exoskeleton force feedback mapping based on virtual-real interaction according to claim 1, characterized in that, In S4, when monitoring collision and contact events between the virtual hand and digital objects, the virtual engine calculates in real time the pose, shape, and physical state of the particles with physical properties of the digital object at the current moment.

Images