Wearable fingertip force touch mixed reality system and perceptual motion training evaluation method

Abstract

The invention relates to the technical field of perceptual motion, and provides a wearable fingertip force haptic mixed reality system and a perceptual motion training evaluation method, and the system comprises a mixed reality platform which is used for displaying a grabbed object block in a virtual scene, and rendering the state of the grabbed object block; the depth camera is used for collecting the center position of the distal joint of the finger in the process that the subject completes the grasping task; the upper computer is used for determining feedback force and the state of a grabbed object block according to the center position of the far-end joint of the finger and the position of the grabbed object block in a virtual scene, calculating expected local coordinates of the end effector through the relation between the distance from the end effector to the center point of the far-end knuckle and the feedback force, and calculating the expected local coordinates of the end effector through inverse kinematics analysis. A steering engine rotation angle is obtained; and the haptic feedback device is worn on the finger of the subject and is used for adjusting the position of the end effector by changing the rotation angle of a steering engine and feeding back haptic feeling to the subject. And the fusion of vision and touch in accurate grasping is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of sensorimotor, and particularly to a wearable fingertip force - touch mixed reality system and a sensorimotor training evaluation method. Background Art

[0002] The statements in this part merely provide the background art related to the present invention and do not necessarily constitute the prior art.

[0003] Sensorimotor, as the sensory feedback process of the human body's motion state, involves the input of multiple senses such as vision, touch, and hearing. The tactile system is usually described as being divided into three modes: cutaneous perception, kinesthetic perception, and force - touch. Among them, force - touch is the general term for force sense and touch. Force - touch feedback is crucial for precise grasping. Compared with other tactile modes, force - touch can provide the human body with more physical information about the hardness, weight, and friction of objects.

[0004] However, current sensorimotor training mostly relies on traditional real - world devices and technologies. For the evaluation of hand function, especially the evaluation of precise grasping, it often depends on the clinical experience judgment of physicians. This subjective - judgment - based method has significant deficiencies in quantitative evaluation and quantitative treatment, and it is difficult to provide an objective and unified standard, resulting in a lack of systematicness and repeatability in practical applications. Currently, research has increasingly focused on wearable devices. Among them, force - touch feedback devices can provide rich force - touch information for finger tips, enabling users to enhance the interaction experience in virtual reality and augmented reality environments, so they have received more attention. However, the effective fusion of force - touch information and visual information has become an important challenge restricting refined hand function training.

[0005] The mixed reality technology based on the fusion of vision and touch is expected to provide a new technical platform for cross - modal perception fusion and refined hand function training. However, at present, there is a lack of a precise grasping training system that fuses force - touch and visual perception feedback based on a mixed reality system. Most of the existing mixed reality hand function training systems are aimed at a single training target and lack comprehensive evaluation and training capabilities. Summary of the Invention

[0006] To solve the deficiencies of the prior art, the present invention provides a wearable fingertip force - touch mixed reality system and a sensorimotor training evaluation method, which form tactile feedback at finger tips through wearable devices and combine mixed reality technology to achieve the fusion of vision and touch in precise grasping.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] The first aspect of the present invention provides a wearable fingertip force - touch mixed reality system.

[0009] A wearable fingertip force - tactile mixed reality system, including a host computer, a mixed reality platform, a depth camera, and a force - tactile feedback device;

[0010] The mixed reality platform is used to display the grasped object block in the virtual scene and render the state of the grasped object block;

[0011] The depth camera is used to collect the position of the distal joint center of the finger during the process of the subject completing the grasping task;

[0012] The host computer is used to determine the feedback force and the state of the grasped object block according to the position of the distal joint center of the finger and the position of the grasped object block in the virtual scene. Based on the feedback force, through the relationship between the distance from the end - effector to the center point of the distal phalanx and the feedback force, calculate the expected local coordinates of the end - effector, and through inverse kinematics analysis, obtain the rotation angle of the servo;

[0013] The force - tactile feedback device is worn on the subject's finger and is used to adjust the position of the end - effector by changing the rotation angle of the servo and feedback the force - tactile feeling to the subject.

[0014] Further, the relationship between the distance from the end - effector to the center point of the distal phalanx and the feedback force is:

[0015] y = ax 3 +bx 2 +cx + d

[0016] x = k d ·d + b d

[0017] In the formula, k d and b d are constants adjusted according to experimental data, d is the distance from the end - effector to the center point of the distal phalanx; a, b, c, and d obtain specific values according to experimental results, x is the duty cycle, and y is the feedback force.

[0018] Further, the host computer is also used to obtain the position of the human hand in the real world, map it to the virtual scene through a transformation matrix, and determine the feedback force when the distance between the human hand in the virtual scene and the grasped object block in the virtual scene is less than a certain value.

[0019] Further, there are two force - tactile feedback devices, which are respectively worn on the subject's thumb and index finger. Each force - tactile feedback device includes an end - effector and three servos. The expected local coordinates of each end - effector are calculated based on the position of the distal joint center of the finger on which it is worn.

[0020] Further, the steps for determining the feedback force and the state of the grasped object block include:

[0021] When the distance between the centers of the distal joints of the subject's thumb and index finger and the grasped object block in the virtual scene is greater than d1, it is in the non-tactile stage, the state of the grasped object block is dropped, and the feedback force is 0;

[0022] When the distance between the center of the distal joint of the subject's thumb or index finger and the grasped object block in the virtual scene is less than d1 and greater than d2, it is in the profile stage, the state of the grasped object block is moving with the position of the human hand, and the feedback force is N1;

[0023] When the distance between the center of the distal joint of the subject's thumb or index finger and the grasped object block in the virtual scene is less than d2 and greater than d3, it is in the contact stage, the state of the grasped object block is moving with the position of the human hand, and the feedback force is N2;

[0024] When the distance between the centers of the distal joints of the subject's thumb and index finger and the grasped object block in the virtual scene is less than d3, it is in the penetration stage, the state of the grasped object block is broken, and the feedback force is 0;

[0025] Among them, the distance d1 > d2 > d3, and the feedback force N1 < N2.

[0026] Furthermore, the virtual scene further includes a baffle, a specified placement point on the right, a specified placement point on the left, and a desktop.

[0027] Furthermore, the grasping task is: the subject moves the grasped object block in the virtual scene from the specified placement point on the right, crosses the baffle in the virtual scene, and places it at the specified placement point on the left.

[0028] The second aspect of the present invention provides a method for evaluating perceptual-motor training.

[0029] A method for evaluating perceptual-motor training, based on a wearable fingertip force-tactile mixed reality system described in the first aspect, includes the following steps:

[0030] Obtain the finger kinematic data of the subject during the grasping task;

[0031] Based on the finger kinematic data, through an evaluation model, obtain a motor coordination score.

[0032] Furthermore, the evaluation model uses LSTM to extract time series features from the finger kinematic data, and takes the time series features as the feature input of the nodes of GNN. After combining the output features of LSTM and GNN, the method of mean pooling is used to generate a global feature vector, and principal component analysis is used for dimensionality reduction processing. Finally, a classifier is used to obtain the motor coordination score.

[0033] Further, it also includes that before the finger kinematic data is input into the evaluation model, the finger kinematic data is normalized and filtered to remove noise.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] The present invention forms tactile feedback at the fingertips of the fingers through a wearable device, and combines mixed reality technology to achieve the fusion of vision and touch in precise grasping, constituting a system that can record, evaluate, and train perceptual motor functions in real time.

[0036] The present invention realizes the quantitative evaluation of perceptual movement by constructing a kinematic model of long short-term memory network - graph neural network - support vector regression. Compared with the traditional method that relies on subjective judgment or a single evaluation method, the present invention realizes the quantitative evaluation of precise grasping movement, and can significantly improve the accuracy, real-time performance, and repeatability of the evaluation results.

[0037] The present invention can be used for the evaluation of perceptual motor functions of healthy people and the perceptual motor evaluation of fine hand operations after various neuromuscular diseases, and also has important application value for the early examination of nervous system diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0039] Figure 1 It is a schematic diagram of a wearable fingertip force tactile mixed reality system provided by the present invention;

[0040] Figure 2 It is a coordinate system diagram corresponding to the center points of the corresponding joints of the finger for collecting data and the wearable fingertip force tactile device provided by the present invention;

[0041] Figure 3 It is a human hand interaction flowchart of the mixed reality system and the wearable fingertip force tactile feedback device provided by the present invention;

[0042] Figure 4 It is a tactile feedback diagram for detecting the center point and contact point of the distal phalanx provided by the present invention;

[0043] Figure 5 It is a flowchart for constructing a precise grasping kinematic calculation model provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0047] In the present invention, terms such as "upper", "lower", "left", "right", "front", "rear", "vertical", "horizontal", "side", "bottom", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only relational terms determined for the convenience of describing the structural relationship of each component or element of the present invention and do not specifically refer to any component or element in the present invention and should not be construed as a limitation to the present invention.

[0048] In the present invention, terms such as "fixed connection", "connected", "connected" should be understood in a broad sense, which may mean a fixed connection, an integral connection or a detachable connection; it may be directly connected or indirectly connected through an intermediate medium. For relevant scientific research or technical personnel in this field, the specific meaning of the above terms in the present invention can be determined according to specific circumstances and should not be construed as a limitation to the present invention.

[0049] Without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0050] Embodiment 1

[0051] Embodiment 1 of the present invention provides a wearable fingertip force-tactile mixed reality system, which forms tactile feedback on the finger fingertips through a wearable device, combines mixed reality technology to achieve the integration of vision and touch in precise grasping, and finally constructs a platform capable of real-time recording, evaluating, and training perceptual motor functions.

[0052] A wearable fingertip force-tactile mixed reality system provided by this embodiment includes: a mixed reality platform, a depth camera, a real-scene force-tactile feedback device, and a host computer. It can not only measure the real-time motion trajectory signals of the three-dimensional position space of the distal joint centers of the thumb and index finger during the entire grasping process, but also enable the subject to generate force-tactile sensations when grasping an object during the grasping process.

[0053] The mixed reality platform includes a display device, which displays a baffle 1 in a virtual scene, a grasping object 2 in the virtual scene, a designated placement point of the grasping object in the right virtual scene (referred to as the designated placement point on the right side) 3, a designated placement point of the grasping object in the left virtual scene (referred to as the designated placement point on the left side) 4, and a desktop 5 in the virtual scene, such as Figure 1 shown.

[0054] Among them, the mixed reality platform displays the grasping task in the virtual scene, and adopts a method based on the box-box experimental paradigm to simulate and evaluate the process of interaction between hand movements and objects. The interaction process includes simulating the hand operations of grasping, moving and releasing. The interaction process is to move the grasping block in the virtual scene from the designated placement point of the grasping block in the right virtual scene, across the baffle in the virtual scene, and place it at the designated placement point of the grasping block in the left virtual scene. Specifically, the designated placement point 3 of the grasping block in the right virtual scene will reset a grasping block 2 in the virtual scene on top before each grasping. The grasping block 2 is the block that needs to be grasped when performing the task. The subject needs to pinch the cube-shaped grasping block 2 back and forth to ensure that it is moved smoothly from the designated placement point on the right to the designated placement point on the left without falling or breaking, and finally release the grasping block 2.

[0055] Among them, the depth camera is configured to obtain the initial coordinates of each end effector in the subject's precise grasping movement process (interaction process), the real-time posture of the hand (such as the position of the base of the thumb), the starting point information of the grasping target, and the posture signals of the distal joint center (O1) 10 of the thumb and the distal joint center (O2) 11 of the index finger of the subject's hand 9 in the real world during the precise grasping process, map the real hand information into the virtual scene, detect the distance between the hand in the virtual scene and the grasping object 2 in the virtual scene, and transmit the instructions to the force tactile feedback device through the platform to realize force tactile feedback on the fingers.

[0056] Among them, Figure 1 As shown, the real-scene force tactile feedback device includes a No. 1 force tactile feedback device 6 worn on the thumb in the real world and a No. 2 force tactile feedback device 7 worn on the index finger in the real world, and performs force tactile feedback on a human hand 9 in the real world, allowing the subject to feel the touch of objects 8 in the real world.

[0057] Among them, the tactile feedback of the force tactile feedback device No. 1 6 is realized by the rotation of the first servo 14, the second servo 15 and the third servo 16; the tactile feedback of the force tactile feedback device No. 2 7 is realized by the rotation of the fourth servo 17, the fifth servo 18 and the sixth servo 19. Figure 2As shown in the figure. The rotation is realized by: after the depth camera detects the pose signals of the distal joint center O1 of the thumb and the distal joint center O2 of the index finger, mapping them into the hand in the virtual scene, and detecting the distance standard between the hand in the virtual scene and the grasping block 2 in the virtual scene, and realizing it through the platform feedback to the force haptic feedback device.

[0058] Among them, as Figure 4 shown, the distance standard includes four stages, namely no haptic, profile stage, contact stage and penetration.

[0059] In this embodiment, to realize force haptic feedback, it is necessary to clarify how the force haptic feedback device realizes force haptic feedback in the mixed reality system during the interaction process, as Figure 3 and Figure 4 shown, the host computer is specifically configured as:

[0060] (1) During the interaction between the mixed reality system and the force haptic feedback device, it is necessary to ensure the initialization of the force haptic feedback device and turn on the communication serial port of the host computer and the slave computer before starting.

[0061] (2) Map the pose signals of the human hand 9 in the real world detected by the depth camera to the virtual scene through the transformation matrix. When the human hand in the virtual scene approaches the grasping block 2 in the virtual scene, detect the distance between the grasping block 2 in the virtual scene and the virtual hand. When a certain distance is reached, start collision detection.

[0062] (3) Start collision detection: The purpose of collision detection is to judge whether the virtual finger and the grasping block 2 in the virtual scene have physical contact or collision (in the unity software, the built-in physical engine of the software will automatically perform collision detection and reaction, usually using Rigidbody and Collider to achieve). Specifically: Detect the position coordinates of the distal joint center point corresponding to the distal joint center O1 of the thumb and the distal joint center O2 of the index finger mapped to the virtual finger tip. Detect the distance between the distal phalanx center point (finger tip detection point) in the virtual world and the contact point of the grasping block 2. Judge whether the virtual finger and the object in the virtual scene have physical contact or collision, obtain the stage, and determine the feedback force and the state of the grasping block according to the positions of the distal joint centers of the subject's thumb and index finger and the position of the grasping block in the virtual scene. Based on the feedback force, obtain the expected local coordinates of the end effector through the relationship between the distance from the end effector to the distal phalanx center point and the feedback force, and through inverse kinematics analysis, obtain the rotation angle of the servo motor, and send the signal to the two force haptic feedback devices of the slave computer to give the subject a force haptic feeling.

[0063] In this embodiment, the distance between the center point of the distal phalanx and the contact point, determined by the spatial distance between the fingertip and the surface and edge of the grasped object block 2, determines at what time and what kind of feeling a person can experience, and is defined as four stages, including: no haptic sensation, silhouette stage, contact stage, and penetration.

[0064] In this embodiment, the distance L from the actual contour of the virtual object to the center point of the distal phalanx min can be calculated by the Euclidean distance: In the formula, P finger =(x f , y f , z f ) is the world coordinate of the center point of the distal phalanx in the virtual hand, and P cube =(x c , y c , z c ) is the world coordinate of the point on the virtual hand's distal phalanx center point that is the closest distance to the surface of the virtual object.

[0065] Furthermore, during the collision detection process, it is necessary to continuously determine whether P finger is inside the bounding box of the virtual object. The bounding box can enclose the entire virtual object block and is usually represented by a minimum value and a maximum value to indicate its range on each coordinate. The minimum value is (MinX, MinY, MinZ), and the maximum value is (MaxX, MaxY, MaxZ), satisfying MinX ≤ x f ≤ MaxX and MinY ≤ y f ≤ MaxY and MinZ ≤ z f ≤ MaxZ, then the center point of the distal phalanx in the virtual hand is inside the bounding box; otherwise, there is no haptic sensation, and the virtual object block is rendered as falling.

[0066] In this embodiment, the distance from the actual contour 21 of the virtual object to the center point of the distal phalanx is defined as L min , the boundary distance between the no haptic sensation and silhouette stages is defined as d1 from the object surface, the boundary distance between the silhouette stage and the contact stage is defined as d2 from the object surface, and the boundary distance between the contact stage and the penetration stage is defined as -d3 from the object surface.

[0067] Among them, for no haptic sensation, please refer to (b) in the appendix Figure 4 . The force feedback device has no haptic feedback, and the range of this stage is defined as outside the visual contour 22 of the silhouette stage; specifically, the mixed reality system renders the object block as falling and feeds back a visual sensation to the subject, and the feedback force is 0 at this time.

[0068] Among them, for the silhouette stage, please refer to the appendix Figure 4In (c), the range of this stage is defined as being between the visualization contour 22 of the profiling stage and the visualization contour 23 of the contact stage; specifically, during the profiling stage, the depth camera can quickly locate the center point of the finger distal end and the position of the virtual object, and the haptic effect is rendered by the mixed reality system. The state of the grasped block is to move with the position of the human hand, and visual and haptic sensations are fed back to the subject.

[0069] Among them, for the contact stage, please refer to Appendix Figure 4 In (d), the range of this stage is defined as being between the visualization contour 23 of the contact stage and the visualization contour 24 of the penetration stage; specifically, during the contact stage, the depth camera locates the center point of the finger distal end and the position of the virtual object, and the haptic effect is rendered by the mixed reality system, and visual and haptic sensations are fed back to the subject.

[0070] Among them, for the penetration stage, please refer to Appendix Figure 4 In (e), the range of this stage is within the visualization contour of the penetration stage; specifically, during the penetration stage, the depth camera locates the center point of the finger distal end and the position of the virtual object, and the mixed reality system renders the breaking of the block instead of haptic feedback, and visual sensations are fed back to the subject. At this time, the end effector returns to the initial local coordinates and the feedback force is 0.

[0071] Specifically, when the distance between the center of the distal joint of the subject's thumb and index finger and the grasped block in the virtual scene is greater than d1, it is in the non-tactile stage, the state of the grasped block is to drop, and the feedback force is 0;

[0072] When the distance between the center of the distal joint of the subject's thumb or index finger and the grasped block in the virtual scene is less than d1 and greater than d2, it is in the profiling stage, the state of the grasped block is to move with the position of the human hand, and the feedback force is N1;

[0073] When the distance between the center of the distal joint of the subject's thumb or index finger and the grasped block in the virtual scene is less than d2 and greater than d3, it is in the contact stage, the state of the grasped block is to move with the position of the human hand, and the feedback force is N2;

[0074] When the distance between the center of the distal joints of the subject's thumb and index finger and the grasped block in the virtual scene is less than d3, it is in the penetration stage, the state of the grasped block is to break, and the feedback force is 0;

[0075] Among them, the distance d1 > d2 > d3, and the feedback force N1 < N2.

[0076] In this embodiment, the different stage divisions are completely determined by the distance between the center point of the distal phalanx of the finger and the contact point, that is, the direct center point of the finger is in the corresponding haptic feedback stage according to different spatial distance parameters.

[0077] In this embodiment, the feedback steps of the force and tactile sensations in the profile display stage and the contact stage include:

[0078] (a) For a certain end effector, obtain the initial local coordinates; for example, the initial local coordinates of a certain end effector are (1, 2, 3), and the force at this time is 0 N;

[0079] (b) Obtain the feedback force corresponding to the stage; for example, the feedback force in the contact stage is 2 N, and the feedback force for each stage is different, which is set according to experience during the experiment. The defined distance for each stage is for dividing each stage, and a feedback force value will be defined for each stage;

[0080] (c) Substitute the feedback force into the relational formula between the distance from the end effector to the center point of the distal phalanx of the finger and the normal force (feedback force) to obtain the distance d from the end effector to the center point of the distal phalanx of the finger, and then obtain the expected local coordinates (C x , C y , C z ) of the end effector, such as (2, 2, 3);

[0081] In this embodiment, according to the established kinematic model and experimental verification, the relationship between the duty cycle (servo rotation angle) and the normal force output of the end effector can be obtained: y = ax 3 + bx 2 + cx + d, where the specific values of a, b, c, and d can be obtained according to the experimental results, x is the duty cycle, y is the normal force, and the distance at the contact point between the finger and the object affects the intensity of the force and tactile feedback. Generally speaking, as the contact distance decreases (the contact becomes closer), the intensity of the feedback force will increase; further, a function is defined to represent the relationship between the distance d from the end effector to the center point of the distal phalanx of the finger (the distance between the thumb or index finger and their respective end effectors) and the duty cycle x, x = k d ·d + b d , where k d and b d are constants adjusted according to the experimental data; combining the above two formulas can establish the relationship between the distance from the end effector to the center point of the distal phalanx of the finger and the normal force, and it is used to define the force and tactile feedback for each grasping stage.

[0082] (d) Based on the expected local coordinates (C x , C y , C z ) of the end effector, through inverse kinematic analysis, calculate the rotation angle of the servo, and the formulas are respectively

[0083] Among them, let the rotation axes of the first servo, the second servo, and the third servo be points A, point E, and point G respectively, and the key point of the end effector be point C, and the joint center points be B and C. For details, see Figure 2 . l5 is the distance between AE, l1 and l3 are the distances of BC and CD, and l1 and l4 are the distances of AB and DE. F is a point on the same horizontal line as G and is located at the midpoint of the connection of the connecting mechanism. l6 is the length of HG, l7 is the length of FH, and l0 is the distance of CF on the Z-axis. represents the angle between the connecting rod and the X-axis. θ1 and θ2 represent the angles between the connecting rod and the Z-axis, and θ3 represents the angle between l6 and l7. By solving the simultaneous equations, the rotation angles of the three servos corresponding to the relevant end effector can be obtained and θ1. Further force feedback effects are rendered by the mixed reality system and fed back to the subject for visual and force feedback sensations.

[0084] Among them, in the mixed reality system, the world coordinates of the first force feedback device 6 and the second force feedback device 7 are (X1, Y1, Z1) and (X2, Y2, Z2) respectively. By defining the transformation matrix T local_to_world to convert the world coordinates into local coordinates, the formula can be expressed as: P local = (T loca_to_world ) -1 · P world . In the formula, P local is the local coordinate of the force feedback device in the virtual world, and P world is the world coordinate of the force feedback device in the real world. Through the formula, the local coordinate 12 of the first force feedback device 6 can be obtained as (x1, y1, z1), and the local coordinate 13 of the second force feedback device 7 can be obtained as (x2, y2, z2). Among them, the world coordinate refers to the position of an object in the real world, which is defined in a unified and fixed coordinate system; the local coordinate system takes the position of the object itself in the virtual world as the origin, represents the relative position and direction between objects, and will rotate according to the rotation of the object itself.

[0085] (4) The lower computer receives the instruction signal transmitted from the upper computer and transmits the corresponding angle signal to the servo according to the relevant instructions transmitted from the upper computer. The expected local coordinates (x1, y1, z1) of the end effector of the first force feedback device control the rotation of the first servo 14, the second servo 15, and the third servo 16 to achieve the force feedback of the thumb; the expected local coordinates (x2, y2, z2) of the end effector of the second force feedback device control the rotation of the fourth servo 17, the fifth servo 18, and the sixth servo 19 to achieve the force feedback of the index finger.

[0086] In this embodiment, the position of the object block is variable. During the process of a human hand grasping the object block, tactile feedback detection is performed in real time, that is Figure 4 In the described process, the position changes of the virtual object block and the hand are both defined based on a world coordinate. When it is detected in the world coordinate that the surface of the hand approaches the contour of the virtual object in the silhouette stage, it is equivalent to Figure 3 the distance detection process in Figure 4 the entire process. In the above process, collision detection will determine whether the object block contacts the finger, and update the position and state of the object block in real time to ensure that the relative position between the virtual object block and the finger remains consistent. If the object block is grasped (in the silhouette stage or contact stage), the position of the object block will change with the movement of the hand, and the movement of the object block should be consistent with the movement of the finger, usually achieved through a physics engine or script control for synchronous update; when the finger moves, the position of the object block will be dynamically adjusted with the grasping action of the hand to ensure that the object block remains within a specific area of the finger; the position change of the object block is updated based on the relationship between the world coordinate system and the position of the hand, and during the grasping process, the object block will be affected by collision feedback and force tactile feedback for appropriate fine-tuning; through a physics engine or script control, the position relationship between the hand and the object block can be updated in real time to ensure the naturalness and smoothness of the interaction. If the hand releases or the grasping force decreases, the object block will start to fall, and the physics engine in the virtual reality will update the free movement trajectory of the object block according to the collision detection result until the object block contacts other objects or stays at a certain position. The hand releasing or the physics becoming smaller is Figure 4 no tactile sensation and penetration in

[0087] A wearable fingertip force tactile mixed reality system provided in this embodiment can form tactile feedback at the fingertip by integrating a wearable device, and combine mixed reality technology to achieve the integration of vision and touch in precise grasping, and can record, evaluate, and train the sensorimotor function in real time. Compared with the traditional sensorimotor training evaluation, the wearable fingertip force tactile mixed reality system provided in this embodiment has stronger real-time performance and accuracy, and can provide more comprehensive and accurate evaluation results in sensorimotor training.

[0088] Embodiment 2

[0089] This embodiment provides a perceptual-motor training evaluation method, based on a wearable fingertip force-tactile mixed reality system described in Embodiment 1.

[0090] This embodiment provides a perceptual-motor training evaluation method. By extracting relevant features during the movement process and constructing a kinematic calculation model of Long Short Term Memory - Graph Neural Networks - Support Vector Regression (LSTM - GNN - SVR), it realizes the quantitative evaluation of perceptual-motor, which can be used for the evaluation of perceptual-motor function of healthy people and the perceptual-motor evaluation and training of fine hand operations after various neuromuscular diseases, and also has important application value for the early examination of nervous system diseases.

[0091] This embodiment provides a perceptual-motor training evaluation method, which is evaluated and trained through the data recorded by a wearable fingertip force-tactile mixed reality system described in Embodiment 1: A finger movement model is established. By obtaining the kinematic feature data of the finger during the grasping process, a model is established by combining the time series based on LSTM and the spatial relationship of GNN, and the regression analysis method SVR of SVM is used for calculation to obtain a quantitative score reflecting the spatio-temporal characteristics and movement coordination of the grasping process, so as to scientifically and objectively evaluate and train the kinematic coordination of precise grasping finger movement.

[0092] This embodiment provides a perceptual-motor training evaluation method, as Figure 5 shown, including the following steps:

[0093] Step 1: Collect the finger kinematic data during the process of grasping the grasped object block through the depth camera in a wearable fingertip force-tactile mixed reality system described in Embodiment 1. According to the configured depth camera, it can be obtained that: X = {(X i , Y i , Z i , t i )|i = 1, 2,, n}. In the formula, (X i , Y i , Z i ) is the spatial position of the fingertip at the i-th moment in the world coordinate system, t i is the time point, and n is the number of sampling points;

[0094] Step 2: Preprocess the collected kinematic data, including: normalization processing and filtering denoising;

[0095] Normalized acquisition aims to convert data of different dimensions into the same numerical range, eliminate the feature weight differences caused by different numerical ranges, and avoid excessive influence of data in a certain dimension on the model.

[0096] Specifically, the min-max normalization method is adopted to map the data D of each dimension to the range [0, 1]. The specific formula is as follows: In the formula, D is the input data, D min is the minimum value of the dimension in this dataset, D max is the maximum value of the dimension in this dataset, D normal is the result after normalization.

[0097] Furthermore, high-frequency noise usually mixes in the acquired data, and these noises will interfere with the extraction of actual motion features and model analysis. By filtering and denoising, the main trend information of the motion data can be retained while removing the noises, improving the accuracy of subsequent modeling. A low-pass filter is used to remove high-frequency noise and retain low-frequency signals.

[0098] Step 3: After the above preprocessing, the input features can be obtained. In order to use the speed features of the thumb and index finger during the grasping process as the input basis of time series data, the features are organized in combination with the time dimension to form an input sequence that can be used in the LSTM model as follows: In the formula, F t is the data feature at the t-th time point, (v t x , v t y , v t z ) is the velocity vector at time t; the feature vectors at multiple consecutive time points t = 1, 2, …, n are combined in chronological order to form the input time series F, which is defined as follows: F = {F1, F2,, F n} In the formula, F is the set of time series input features for the entire grasping process.

[0099] Specifically, the speed feature is calculated based on the normalized spatial coordinates, and the speed is calculated according to the position difference between adjacent moments, calculating the derivative of the position with respect to time. The calculation formula is: and

[0100] Step 4: Input the time series F into the LSTM network. The basic computational unit of this network includes the forget gate, input gate, cell state update, and output gate core. The calculation includes the following steps:

[0101] The forget gate is used to determine the historical information to be discarded at the current moment. Its calculation formula is: f t = σ(W f · [ht-1 , x t + b f ). In the formula, x t is the input feature at the current moment, h t-1 is the hidden state at the previous moment, W f and b f are the weight matrix and bias vector of the forget gate respectively, and σ is the activation function;

[0102] The input gate is used to determine the importance of the current input information, and its calculation formula is: i t = σ(W i · [h t-1 , x t + b i ).

[0103] The calculation formula for the updated value of the candidate memory cell is: In the formula, W i is the weight matrix of the input gate, b i is the bias vector of the input gate, W c is the weight matrix of the candidate memory cell, b c is the bias vector of the input gate;

[0104] Cell state update is used to perform weighted update on the cell state at the previous moment and the current input through the forget gate and the input gate, and its calculation formula is: In the formula, C t is the cell state at the current moment, C t-1 is the cell state at the previous moment;

[0105] The output gate, used to control the output information of the hidden state, and its calculation formula is: o t = σ(W o · [h t-1 , x t + b o ).

[0106] The hidden state at the current moment is updated to: h t = o t · tanh(C t ). In the formula, W o is the weight matrix of the output gate, b o is the bias vector of the output gate.

[0107] The LSTM network generates a feature sequence H LSTM = {h1, h2,..., h m} through time series modeling, where H LSTMare the global dynamic features of the time series, which can reflect the speed changes, dynamic dependencies, and coordination between the thumb and index finger during the grasping process.

[0108] Among them, the matrix H LSTM is an m×d LSTM matrix,

[0109] Step 5: Use the time series features output by the LSTM as the feature input of the nodes of the GNN, and the GNN is responsible for constructing the spatial relationship between objects; assume that the graph structure of the GNN is G=(V,E), where V is the set of nodes, E is the set of edges, and the initial feature of each node is the feature vector H output by the LSTM LSTM Update the nodes through graph convolution, and the formula is as follows: In the formula, is the feature of node i at the l-th layer, N(i) is the neighbor nodes of node i, W (l) and b (l) are the weight matrix and bias term of the l-th layer respectively, σ(·) is the activation function, and after l layers of convolution, the output feature is where n is the number of nodes, and d GNN is the feature dimension of each node;

[0110] Step 6: Combine the output features of the LSTM and the GNN. The feature dimension output by the LSTM is d LSTM , and the feature dimension d output by the GNN GNN , then the combined feature vector H concat is: H concat =[H LSTM ,H GNN . Concatenate the feature vectors of the LSTM and the GNN into a long feature vector, and the dimension is: d concat =d LSTM +d GNN .

[0111] Step 7: For the output high-dimensional feature matrix H concat , in order to further extract the global dynamic characteristics and at the same time reduce the computational complexity of the subsequent regression analysis, use a dimensionality reduction method to transform the matrix H concat into a unified high-dimensional time series feature vector h.

[0112] Step 8: In order to capture the global characteristics of the entire time series, integrate the features h concat at each time step in H t , and use the method of mean pooling to generate the global feature vector The formula is: In the formula, the weights satisfy the condition Weight α t It is dynamically adjusted according to the task requirements.

[0113] Step 9. After feature integration, the generated global feature vector may still have a relatively high dimension d. Therefore, further dimensionality reduction processing is required. The present invention provides principal component analysis (PCA) for dimensionality reduction, including but not limited to this method. The formula is as follows: In the formula, W PCA is the PCA projection matrix, and its column vectors are the principal component directions of

[0114] Step 10. After feature integration and dimensionality reduction processing, the generated final high-dimensional time series feature vector h has a relatively low dimension p (usually p << d); the vector h represents the feature vector after feature integration and dimensionality reduction processing, and contains the dynamic characteristics and coordination information of the kinematic signals. The formula expression is:

[0115] Step 11. Use SVR (support vector regression) to perform regression analysis on the time series of the grasping kinematic features, so as to output a quantified motion coordination score Y. The input of SVR is the feature vector h after feature integration and dimensionality reduction processing by the LSTM network and GNN.

[0116] During the training process, by using the labeled motion coordination score Y i and the corresponding feature vector h i to optimize the parameters β and b of the SVR model, and by minimizing the regression error, the optimal prediction function is obtained. The training objective is: In the formula, ξ i is the slack variable, and C is the regularization parameter, which controls the balance between the complexity of the model and the error.

[0117] Furthermore, after the training is completed, the SVR model can predict the new kinematic feature vector h and output the corresponding motion coordination score formula: Y = w T h + b. In the formula, the score value Y can reflect the grasping coordination level corresponding to the input feature vector, and can provide a quantified evaluation for the motion accuracy, stability and coordination during the grasping process.

[0118] This embodiment provides a perceptual motor training evaluation method. By combining the advantages of LSTM in capturing time series features, the spatial extraction ability of GNN, and the ability of SVR in dealing with non-linear problems, especially its generalization ability in small datasets, the dynamic characteristics of the precise grasping process can be effectively and objectively evaluated. Compared with traditional evaluation methods that rely on subjective judgment or single evaluation methods, this method can significantly improve the accuracy, real-time performance, and repeatability of the evaluation results, ensuring quantitative analysis of the precise grasping quality.

[0119] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A wearable fingertip force and tactile mixed reality system, characterized in that: It includes a host computer, a mixed reality platform, a depth camera, and a force - tactile feedback device; The mixed reality platform is used to display the grasped object block in the virtual scene and render the state of the grasped object block; The depth camera is used to collect the central positions of the distal joints of the fingers during the process of the subject completing the grasping task; The host computer is used to determine the feedback force and the state of the grasped object block according to the central positions of the distal joints of the fingers and the position of the grasped object block in the virtual scene. Based on the feedback force, through the relationship between the distance from the end - effector to the center point of the distal phalanx and the feedback force, calculate the expected local coordinates of the end - effector, and through inverse kinematics analysis, obtain the rotation angles of the servos; The force - tactile feedback device is worn on the subject's fingers and is used to adjust the position of the end - effector by changing the rotation angle of the servo and feedback a force - tactile feeling to the subject.

2. The wearable fingertip force-tactile mixed reality system according to claim 1, characterized in that: The relationship between the distance from the end - effector to the center point of the distal phalanx and the feedback force is: y = ax 3 + bx 2 + cx + d x = k d ·d + b d where k d and b d are constants adjusted according to experimental data, d is the distance from the end effector to the center point of the distal phalanx; specific values of a, b, c, and d are obtained from the experimental results, x is the duty cycle, and y is the feedback force.

3. The wearable fingertip force and tactile mixed reality system according to claim 1, characterized in that: The host computer is also used to obtain the position of the human hand in the real world, map it to the virtual scene through a transformation matrix, and determine the feedback force when the distance between the human hand in the virtual scene and the grasped object block in the virtual scene is less than a certain value.

4. The wearable fingertip force-tactile hybrid reality system according to claim 1, characterized in that: There are two force - tactile feedback devices, which are respectively worn on the subject's thumb and index finger. Each force - tactile feedback device includes an end - effector and three servos. The expected local coordinates of each end - effector are calculated based on the central positions of the distal joints of the fingers on which it is worn.

5. The wearable fingertip force-tactile mixed reality system according to claim 4, wherein: The steps for determining the feedback force and the state of the grasped object block include: When the distance between the central positions of the distal joints of the subject's thumb and index finger and the grasped object block in the virtual scene is greater than d1, it is in the non - tactile stage, the state of the grasped object block is dropped, and the feedback force is 0; When the distance between the central position of the distal joint of the subject's thumb or index finger and the grasped object block in the virtual scene is less than d1 and greater than d2, it is in the profiling stage, the state of the grasped object block is moving with the position of the human hand, and the feedback force is N1; When the distance between the central position of the distal joint of the subject's thumb or index finger and the grasped object block in the virtual scene is less than d2 and greater than d3, it is in the contact stage, the state of the grasped object block is moving with the position of the human hand, and the feedback force is N2; When the distance between the central positions of the distal joints of the subject's thumb and index finger and the grasped object block in the virtual scene is less than d3, it is in the penetration stage, the state of the grasped object block is broken, and the feedback force is 0; Among them, the distance d1 > d2 > d3, and the feedback force N1 < N2.

6. The wearable fingertip force-tactile hybrid reality system according to claim 1, characterized in that: The virtual scene also includes a baffle, a right designated placement point, a left designated placement point, and a desktop.

7. The wearable fingertip force and tactile hybrid reality system according to claim 6, wherein: The grasping task is: the subject moves the grasped object block in the virtual scene from the right designated placement point, crosses the baffle in the virtual scene, and places it at the left designated placement point.

8. A method for evaluating sensorimotor training, characterized in that: Based on a wearable fingertip force - tactile mixed reality system according to any one of claims 1 - 7, it includes the following steps: Obtain the finger kinematic data of the subject during the process of completing the grasping task; Based on the finger kinematic data, through an evaluation model, obtain a motion coordination score.

9. The perception-motor training evaluation method according to claim 8, wherein: The evaluation model uses LSTM to extract time series features from finger kinematic data, and takes the time series features as the feature input of the nodes of the GNN. After combining the output features of LSTM and GNN, the mean pooling method is used to generate a global feature vector, and principal component analysis is used for dimensionality reduction processing. Finally, a classifier is used to obtain the motor coordination score.

10. The perception-motor training evaluation method according to claim 8, characterized in that: It also includes that before the finger kinematic data is input into the evaluation model, normalization processing and filtering and denoising processing are performed on the finger kinematic data.

Images