Multi-mode intelligent interactive visual rehabilitation evaluation and treatment glove

Abstract

The invention discloses a multi-mode intelligent interactive visual rehabilitation evaluation and treatment glove, and relates to the field of intelligent medical treatment, the glove comprises a flexible glove body and an integrated multi-source sensing array, sensors are arranged in metacarpophalangeal joints, fingertips, thenar, joints and hand back areas, and pressure, touch, tension, bending and posture data are collected in real time. And the data processing unit carries out fusion processing on the multi-modal signals, constructs a dynamic three-dimensional force field model, and captures the magnitude, direction, speed and path change of applied force. The system is further integrated with an interaction feedback module, multichannel real-time early warning and visual assistance are achieved through an AR visual assembly, a tactile feedback assembly and a voice broadcast assembly, and therefore personalized force application safety and treatment quality are guaranteed.

Description

TECHNICAL FIELD

[0001] The present specification relates to the field of smart healthcare, and more particularly, to a multi-modal intelligent interactive visual rehabilitation assessment and treatment glove. BACKGROUND

[0002] In rehabilitation treatment, medical personnel mainly rely on hand feeling and experience to judge and control the size and direction of force, which cannot accurately quantify the vector information of force, is easily disturbed by individual subjective factors, and has great differences in the operation of the same technology by different medical personnel. Rehabilitation treatment technology cannot be accurately and standardized, so there are significant errors and risks between individuals. Even if the same person operates, there are mechanical errors in each execution, and long-time treatment easily leads to fatigue accumulation, affecting the intervention accuracy and safety.

[0003] Some existing technologies such as Leap Motion optical tracking system can monitor finger movement trajectory, but cannot provide real-time force vector information, and it is difficult to reflect the true force state during treatment. Some other high-sensitivity flexible sensor technologies, such as pressure sensors based on graphene or carbon nanotube materials, have excellent force sensing ability, but are mainly focused on the industrial or wearable field, and have not been deeply integrated with clinical rehabilitation assessment and training scenes. In addition, most current rehabilitation assistance systems still use one-way monitoring architecture, lack of two-way perception and feedback of biological information between medical personnel and patients, and cannot actively warn before force abnormality or risk event occurs.

[0004] Therefore, there is an urgent need for an intelligent device with real-time force vector modeling and interactive feedback capability to assist medical personnel to achieve high-precision, low-risk, and individualized rehabilitation intervention operation, while providing standardized and accurate data management and guidance in various scenes such as medical quality control management and rehabilitation technology teaching. SUMMARY

[0005] A series of simplified concepts are introduced in the summary section, which will be further described in detail in the specific embodiment section. The summary section of the present application does not mean to attempt to limit the key features and necessary technical features of the claimed technical solutions, nor to attempt to determine the protection scope of the claimed technical solutions.

[0006] In a first aspect, the present application provides a multi-modal intelligent interactive visual rehabilitation assessment and treatment glove, comprising:

[0007] a flexible glove body;

[0008] A multi-source sensor array, the multi-source sensor array comprising a microarray triaxial pressure sensor disposed at the metacarpophalangeal joint region, a capacitive tactile sensor disposed at the fingertip and finger pulp, a piezoresistive tension sensor disposed at the thenar eminence / hypothenar region, a flexible bending sensor disposed at the joint, and an inertial measurement unit disposed at the back of the hand;

[0009] A data processing unit configured to perform fusion processing on signals collected by the multi-source sensor array to establish a three-dimensional dynamic force field model containing force magnitude, direction, force application speed, and path;

[0010] An interactive feedback module, the interactive feedback module comprising an AR visualization component, a tactile feedback component, and a voice feedback component; the AR visualization component is configured to project three-dimensional force field information, patient muscle tension heat map, and rhythmic spectrum information into the field of view of medical personnel; the tactile feedback component is configured to provide a vibration prompt to medical personnel when the detected applied force is greater than the individualized safe force application threshold; the voice feedback component is configured to broadcast abnormal force application prompt information.

[0011] In a feasible implementation, the glove further comprises:

[0012] A personalized assessment module configured to determine the individualized safe force application threshold based on bone density, spasm level, tissue elastic modulus, and pain threshold information in the patient's electronic medical record.

[0013] In a feasible implementation, the personalized assessment module comprises a feature vector encoding unit, a threshold inference unit, and a dynamic adjustment module.

[0014] The feature vector encoding unit is configured to extract the bone density, the spasm level, the tissue elastic modulus, and the pain threshold information from the patient's electronic medical record and map them into a multi-dimensional feature vector.

[0015] The threshold inference unit maps the multi-dimensional feature vector into an individualized safe force application threshold through a neural network model.

[0016] The dynamic adjustment module calculates a threshold adjustment factor in real time based on changes in muscle tension, pain feedback, and electromyographic signal changes of the patient during rehabilitation, and updates the safe force application interval threshold.

[0017] In a feasible implementation, the neural network model in the threshold inference unit comprises:

[0018] An input layer configured to receive a multi-dimensional normalized feature vector containing the bone density, the spasm level, the tissue elastic modulus, and the pain threshold information.

[0019] At least two fully connected hidden layers, the first hidden layer includes 16 neurons, the second hidden layer includes 8 neurons, and the activation function is ReLU, which is used for non-linear mapping and feature combination learning of input features;

[0020] An output layer, which is a continuous regression output layer including two neurons, is used to predict the upper and lower limits of the individualized force safety threshold interval of the patient, and a Softplus activation function is used.

[0021] The neural network model is trained by using a composite loss function, and the loss function includes a mean square error term and an interval stability regularization term.

[0022] In a feasible implementation, the three-axis pressure sensor is a microarray sensor composed of MWCNTs / PDMS composite material, which is packaged in a medical silicone micro-dome structure.

[0023] In a feasible implementation, the capacitive tactile sensor adopts an interdigital electrode design and is printed on a palm elastic substrate.

[0024] In a feasible implementation, the inertial measurement unit includes an accelerometer, a gyroscope, and a magnetometer, and is configured to calculate the hand posture and spatial trajectory in real time.

[0025] In a feasible implementation, the specific processing steps of the data processing unit include:

[0026] The spatial position embedding and time synchronization processing are introduced to the multi-source sensing data, and the Kalman filter is used to realize signal alignment and interference suppression;

[0027] A four-dimensional force tensor flow field is constructed, a long short-term memory network is used to model and predict the continuous force vector, and a three-dimensional dynamic force field model is generated, which includes force size, direction, action speed, and path change information.

[0028] In a feasible implementation, the AR visualization component includes an augmented reality glasses, in which the force vector is displayed in the form of an arrow, the length of the arrow represents the force size, and the color represents the safety level.

[0029] In a feasible implementation, the tactile feedback component includes at least four linear resonance motors arranged on the back of the hand, which can output vibration according to the abnormal force level.

[0030] In summary, the multi-source sensing array constructed by the micro-array three-axis pressure sensor, the capacitive tactile sensor and the piezoresistive tension sensor can obtain the mechanical information (such as the size, direction, action position and duration of force) of multiple key areas of the hand in real time and accurately. Cooperating with the fusion posture solving technology of the flexible bending sensor and the IMU, the system can construct a dynamic force field model in the three-dimensional space and visualize the force field model through the AR interface, effectively making up for the technical short board of "can't see the force" and "feeling inaccurate" in the traditional rehabilitation process. The data processing unit can intelligently generate a personalized force application safety threshold interval according to the individual pathological data (such as bone density, muscle tension level, tissue elastic modulus, etc.) imported by the patient, and dynamically adjust it during the treatment process. This adaptive ability significantly enhances the individual matching ability of the system, which is suitable for patients with various neuromuscular diseases and different rehabilitation stages. The tactile feedback component and the voice feedback component can immediately issue an alarm when dangerous force application behavior is monitored, helping medical staff to avoid secondary damage caused by excessive force application, especially for patients who are sensitive to force application or in the early postoperative stage. The multi-channel feedback mechanism significantly improves the response speed and risk identification ability of the rehabilitation intervention. Through data recording and AR guidance, the system can be used for medical staff training and clinical teaching, combined with the expert demonstration database and operation standards to realize the standardization of rehabilitation operation, provide effective assistance for inexperienced medical staff, and improve the overall rehabilitation quality.

[0031] Other advantages, objects, and features of the present application will be apparent to those skilled in the art from the following specification. BRIEF DESCRIPTION OF DRAWINGS

[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The detailed description is made with reference to the accompanying drawings.

[0033] Figure 1 A structural schematic diagram of a multi-modal intelligent interactive visual rehabilitation evaluation and treatment glove provided by an embodiment of the present application;

[0034] Figure 2 Another structural schematic diagram of a multi-modal intelligent interactive visual rehabilitation evaluation and treatment glove provided by an embodiment of the present application. DETAILED DESCRIPTION

[0035] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application and above-described drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed herein is merely for convenience and brevity and does not necessarily imply a serial or chronological order unless expressly so defined by the patentee. In addition, the terms "comprise", "comprising", "have", "having", "include", "including", "contain", "containing", "provide", "provided", "provide for", "provided for" and any variations thereof in the description and in the claims of the present application, if any, are to be construed as specifying the presence of what is stated in the description and claims rather than as specifying the presence of what is stated in the description and claims and any step or element not explicitly stated in the description and claims should be implied. The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application.

[0036] Referring to FIG. Figure 1 and Figure 2 A structural schematic diagram of a multi-modal intelligent interactive visual rehabilitation evaluation and treatment glove 10 provided by the embodiments of the present application, specifically can include:

[0037] A flexible glove body 11;

[0038] A multi-source sensing array 12, the multi-source sensing array 12 includes a microarray triaxial pressure sensor 121 arranged at a metacarpophalangeal joint area, a capacitive tactile sensor 122 arranged at a fingertip and a finger palm, a piezoresistive tension sensor 123 arranged at a thenar eminence / hypothenar eminence area, a flexible bending sensor 124 arranged at a joint, and an inertial measurement unit 125 arranged at a back of a hand area;

[0039] A data processing unit 13 configured to perform fusion processing on signals collected by the multi-source sensing array, and establish a three-dimensional dynamic force field model containing force size, direction, force application speed and path;

[0040] An interactive feedback module 14, the interactive feedback module includes an AR visualization component 141, a tactile feedback component 142, and a voice feedback component 143; the AR visualization component is used to project three-dimensional force field information, patient muscle tension heat map and rhythmic spectrum information to a field of view of a medical staff; the tactile feedback component is used to provide a vibration prompt to the medical staff in the case of detecting that an applied force is greater than an individualized force application safety threshold; and the voice feedback component is used to broadcast abnormal force application prompt information.

[0041] Exemplarily, the embodiment provides a visual rehabilitation evaluation and treatment glove system based on multi-modal intelligent interaction, aiming to realize real-time monitoring, dynamic analysis and multi-channel intelligent feedback of force exertion behavior and physiological feedback of patients during rehabilitation. The system mainly includes a flexible glove body 11, a multi-source sensing array 12, a data processing unit 13 and an interactive feedback module 14.

[0042] The flexible glove body 11 is made of medical silicone material with high adhesion and biocompatibility, has good flexibility and wearability, and is suitable for rehabilitation patients or medical personnel of different hand shapes. A variety of types of flexible sensors are integrated inside the glove, and are connected to the control box at the wrist through wiring to realize signal acquisition and processing.

[0043] The multi-source sensing array 12 is a key component of the present application, which is configured with five types of core sensors, respectively:

[0044] Microarray three-axis pressure sensor 121: arranged at the metacarpophalangeal joint area of the second to fifth fingers, used for high-resolution monitoring of pressure distribution and change of each joint area during gripping;

[0045] Capacitive tactile sensor 122: arranged at the fingertips and finger pads of each finger, used for capturing point pressure and shear force information to realize monitoring of fine touch operation;

[0046] Piezoresistive tension sensor 123: arranged at the thenar eminence and hypothenar region of the palm, monitoring the tension distribution and change trend of the larger area;

[0047] Flexible bending sensor 124: integrated at the finger joint position, used for monitoring the bending angle and flexion amplitude of the joint;

[0048] Inertial measurement unit (IMU) 125: arranged on the back of the hand, used for capturing the motion trajectory of the whole hand in space, including attitude angle, acceleration and angular velocity information.

[0049] The above sensing array is connected to the data processing unit 13 through FPC flexible flat cable. The unit is built-in with high-performance microcontroller (such as STM32H7 series chip), which is responsible for time sequence synchronization, filtering and noise reduction and feature fusion processing of heterogeneous data from different types of sensors. Through Kalman filtering and spatial registration algorithm, the unit can calculate the resultant force vector (including the size, direction and action position of the force) of the current hand in three-dimensional space in real time, and further construct a dynamic three-dimensional force field model. The force field model can also combine with time window to perform path trajectory backtracking and trend prediction, which is used for subsequent feedback control and personalized evaluation.

[0050] The interactive feedback module 14 is the core channel to realize intelligent interaction, including the following three sub-modules:

[0051] AR visualization component 141: preferably using augmented reality devices such as Hololens 2, superimpose the three-dimensional force vector, muscle tension heat map and rhythmic spectrum generated by the data processing unit in an image manner in the field of view of the medical staff, the direction of force is represented by arrow, and the color of arrow (green→yellow→red) represents the safety level of current force;

[0052] Tactile feedback component 142: linear resonance motor array is arranged on the back of the hand, when it is monitored that the force of a certain area exceeds the individualized safety threshold, the motor of the area is started to vibrate, so as to prompt the medical staff to reduce the force in the form of tactile;

[0053] Voice feedback component 143: embedded intelligent broadcast system, which can issue voice prompts in Chinese or other languages to the medical staff when the system identifies abnormal force events (such as “current thenar region pressure exceeds 20%”), improve the perception efficiency and reduce the visual interference.

[0054] The system supports patient medical record import and individualized parameter modeling, and maps bone density, spasm level, tissue elastic modulus and pain threshold information into a multi-dimensional feature vector when used for the first time, combines neural network or support vector machine model to infer the individualized force safety interval of the patient. During the rehabilitation process, the system can also adaptively and dynamically adjust the threshold based on muscle tension change, electromyography data and pain feedback, forming a “medical staff-patient” closed-loop feedback system.

[0055] In summary, the multi-source sensing array constructed by the micro-array three-axis pressure sensor, the capacitive tactile sensor and the piezoresistive tension sensor can obtain the mechanical information (such as the force, the direction, the acting position and the duration) of multiple key areas of the hand in real time and accurately. The fusion posture solving technology of the flexible bending sensor and the IMU can construct a dynamic force field model in the three-dimensional space, and the force field model can be visualized through the AR interface, which effectively makes up for the technical short board of "can't see the force" and "inaccurate feeling" in the traditional rehabilitation process. The data processing unit can intelligently generate a personalized safe force application threshold interval according to the individual pathological data (such as bone density, muscle tension level, tissue elastic modulus, etc.) imported by the patient, and dynamically adjust the safe force application threshold interval during the treatment process. The adaptive ability significantly enhances the individual matching ability of the system, which is suitable for patients with various neuromuscular diseases and at different rehabilitation stages. The tactile feedback component and the voice feedback component can immediately issue an alarm when a dangerous force application behavior is monitored, helping medical staff to avoid secondary damage caused by excessive force application, especially for patients who are sensitive to force application or in the early postoperative stage. The multi-channel feedback mechanism significantly improves the response speed and risk identification ability of the rehabilitation intervention. Through data recording and AR guidance, the system can be used for medical staff training and clinical teaching, combined with the expert demonstration database and operation standards to realize the standardization of rehabilitation operation, provide effective assistance for inexperienced medical staff, and improve the overall rehabilitation quality.

[0056] In a feasible implementation, the glove further comprises:

[0057] The personalized assessment module is configured to determine the individualized safe force application threshold based on bone density, spasm level, tissue elastic modulus and pain threshold information in the patient's electronic medical record.

[0058] In a feasible implementation, the personalized assessment module comprises a feature vector encoding unit, a threshold inference unit and a dynamic adjustment module.

[0059] The feature vector encoding unit is used to extract the bone density, the spasm level, the tissue elastic modulus and the pain threshold information from the patient's electronic medical record, and map them into a multi-dimensional feature vector.

[0060] The threshold inference unit maps the multi-dimensional feature vector into an individualized safe force application threshold through a neural network model.

[0061] The dynamic adjustment module calculates a threshold adjustment factor in real time based on the changes in muscle tension, pain feedback and electromyographic signal of the patient during the rehabilitation process, and updates the safe force application threshold.

[0062] In a feasible implementation, the neural network model in the threshold inference unit comprises:

[0063] an input layer for receiving a multi-dimensional normalized feature vector containing the bone density, the spasticity level, the tissue elasticity modulus, and the pain threshold information;

[0064] at least two fully connected hidden layers, the first hidden layer comprising 16 neurons and the second hidden layer comprising 8 neurons, and a ReLU activation function, for non-linear mapping and feature combination learning of the input features;

[0065] an output layer, which is a continuous regression output layer comprising two neurons, for predicting the upper and lower limits of the individualized force application safety threshold interval of the patient, and adopting a Softplus activation function;

[0066] The neural network model is trained using a composite loss function, and the loss function comprises a mean square error term and an interval stability regularization term.

[0067] In an exemplary embodiment, the visual rehabilitation evaluation and treatment glove further comprises a personalized evaluation module for automatically determining the safe force application range of different patients during rehabilitation training according to their physiological characteristics, so as to realize individualized and precise rehabilitation strategy generation. The module provides dynamic adaptive force application safety threshold through multi-dimensional physiological index modeling and neural network intelligent reasoning, and provides accurate judgment basis for the subsequent feedback system.

[0068] The personalized evaluation module mainly comprises three sub-units: a feature vector encoding unit, a threshold reasoning unit, and a dynamic adjustment module.

[0069] Firstly, the feature vector encoding unit is used to extract core physiological parameters from the electronic medical record of the patient, including bone density (such as T value or Z value in DEXA report), spasticity level (for example, Modified Ashworth Scale score), tissue elasticity modulus (calculated by ultrasonic elastography or muscle tension sensor), and subjective pain threshold (for example, NRS score). After normalization, these indicators are mapped into a 4-dimensional standardized feature vector:

[0070]

[0071] The vector is used as the input of the neural network model.

[0072] The threshold reasoning unit is embedded with a set of multi-layer feedforward neural network model with clear structure and strong adaptability, which is used to infer the force application safety threshold interval of the patient individual according to the feature vector. The neural network model specifically comprises:

[0073] an input layer for receiving the normalized 4-dimensional physiological feature vector;

[0074] The first hidden layer includes 16 neurons and uses a ReLU activation function to preliminarily extract nonlinear features.

[0075] The second hidden layer includes 8 neurons and also uses a ReLU activation function to realize high-order feature combination.

[0076] The output layer includes 2 neurons, which respectively output the upper and lower limit values F min , F max of the predicted safe force interval, and uses a Softplus activation function to ensure that the output is positive and has numerical stability.

[0077] During the training process of the model, a composite loss function is used:

[0078]

[0079] The first term is the mean square error between the predicted value and the true labeled value, and the second term is a regularization term used to constrain the logical order of the output interval to avoid abnormal cases where the upper limit is less than the lower limit, and λ is a regularization weight hyperparameter.

[0080] During the real-time process of rehabilitation training, the dynamic adjustment module continuously receives physiological feedback information of the patient, including real-time muscle tension changes from the tension sensor in the glove, subjective pain feedback received by the voice recognition module, and optional surface electromyography (sEMG) signal amplitude changes. The system calculates a comprehensive adjustment factor Δ adj by setting feedback sensitivity factor weights (such as β1, β2, β3), and dynamically corrects the current safe force threshold interval:

[0081]

[0082] For example, when the system detects that the patient has a sudden rise in muscle tension or excessive muscle activation during force application, the system will lower the current recommended safe force upper limit by a certain percentage to prevent further pressure from causing tissue damage.

[0083] In summary, the personalized evaluation module of the embodiment is based on the pathological characteristics of the patient, combines the nonlinear expression ability of the neural network model and the feedback perception mechanism, realizes the prediction, adjustment and continuous optimization of the safe force interval, and greatly improves the individual adaptability, safety and scientificity of rehabilitation training. The design not only has good clinical practicability, but also provides strong data support and decision basis for subsequent force control and interactive feedback mechanisms.

[0084] In a feasible implementation, the above-mentioned three-axis pressure sensor is a microarray sensor composed of MWCNTs / PDMS composite material, which is packaged in a medical silica gel micro-dome structure.

[0085] In one possible implementation, the capacitive tactile sensor adopts an interdigitated electrode design and is printed on the palm elastic substrate.

[0086] In one possible implementation, the inertial measurement unit includes an accelerometer, a gyroscope and a magnetometer and is configured to calculate the hand posture and spatial trajectory in real time.

[0087] To achieve high sensitivity and distributed sensing of force exertion on the hand, the visual rehabilitation assessment and treatment glove described in the present application uses a variety of sensor materials and structural designs that are flexible, highly responsive and medically compatible, including a triaxial pressure sensor, a capacitive tactile sensor and an inertial measurement unit, which are composed of the following materials and designed as follows:

[0088] First, the triaxial pressure sensor preferably adopts a multi-walled carbon nanotube (MWCNTs) / polydimethylsiloxane (PDMS) composite material to form a microarray type piezoresistive sensor. This type of sensor has good flexibility, stretchability and biocompatibility, and is suitable for attachment to the joint movement area of the hand. In order to improve the local pressure sensing accuracy and anti-interference ability, the sensor unit is packaged in a "micro-dome structure", i.e., a medical-grade silicone is used to form an arched protrusion, so that a pre-press contact is formed between the force point and the sensing layer, thereby enhancing the response sensitivity to stress in different directions (x / y / z axes). Each dome structure can accommodate an independent triaxial piezoresistive unit, and the array is arranged in the metacarpophalangeal joint area to effectively capture the subtle pressure distribution differences generated during finger flexion or gripping.

[0089] Second, at the fingertip and fingerpad areas of the fingers, a capacitive tactile sensor is used to recognize slight touch, sliding or shearing actions. The sensor adopts an interdigitated electrode design and is printed on an elastic substrate (such as Ecoflex or TPU film) with high conductivity silver paste. The interdigitated structure changes its capacitance value when subjected to pressure or contact due to changes in dielectric thickness or coupling area between electrodes, with high sensitivity and a wide dynamic range. The sensor is printed on the elastic substrate on the palm and fingerpad areas, realizing flexible electronic integration without affecting the softness of the glove, and is suitable for detecting local subtle contact force, surface friction and action characteristics of thumb-palm operation.

[0090] Finally, to achieve high-precision tracking of the overall motion trend and spatial posture of the hand, an inertial measurement unit (IMU) is embedded in the back region of the glove. The IMU module integrates three types of sensor elements, namely accelerometers, gyroscopes, and magnetometers, for acquiring linear acceleration, angular velocity, and absolute direction information, respectively. Through a pose solving algorithm (such as the Madgwick or Mahony filter), the system can output real-time hand spatial posture information in the form of Euler angles or quaternions. At the same time, the trajectory data obtained by the IMU can be spatiotemporally fused with data from the piezoresistive and capacitive sensors, providing a global motion reference framework for subsequent three-dimensional force field modeling.

[0091] In summary, the structural design and configuration of the three types of core sensors are integrated, and the embodiment realizes high-density, multi-modal information acquisition in the key functional regions of the hand. The sensors work cooperatively to form a high-resolution force-displacement-trajectory fusion dataset, providing a solid sensing foundation for force accuracy judgment, abnormal motion recognition, and multi-channel feedback control during rehabilitation training. The implementation scheme has good practicality, flexible wearability, and clinical promotion adaptability.

[0092] Specifically, the sensors can be arranged in the manner of Table 1:

[0093]

[0094]

[0095] Table 1

[0096] To ensure the biocompatibility, flexible fit, and mechanical response performance of the glove, the sensors used in the present application adopt a dielectric layer capacitive sensor with a bionic micro-cilium array structure, which has high sensitivity (about 0.28 kPa -1 ) and a wide detection range (0-200 kPa), suitable for monitoring micro-force and medium-force level contact operations; at the same time, a flexible piezoresistive sensor composed of MWCNTs (multi-walled carbon nanotubes) / PDMS (polydimethylsiloxane) composite material is used, which has a short response time (about 80 ms) and an accuracy of ±0.1 N, suitable for high dynamic pressure change scenarios.

[0097] The overall sensor package uses Silbione TM Biomedical series medical grade silicone with a material thickness of about 0.8 mm, a tensile rate ≥ 900%, and meets the ISO 10993 biocompatibility certification standard, ensuring that the glove can be used on the skin surface for a long time, while having good cleanability and repeatable wearing performance.

[0098] In a feasible implementation manner, the specific processing steps of the above data processing unit include:

[0099] The spatial position embedding and time synchronization processing are introduced to the multi-source sensing data, and signal alignment and interference suppression are realized based on Kalman filtering;

[0100] A four-dimensional force tensor flow field is constructed, and a long short-term memory network is used to model and trend predict the continuous force vector, thereby generating a three-dimensional dynamic force field model containing force size, direction, action speed and path change information.

[0101] To achieve efficient fusion of multi-source sensing data and high-dimensional spatiotemporal force field modeling, the data processing unit in the present application adopts a multi-step data processing procedure, covering spatial position embedding, time synchronization, filtering and noise reduction, tensor modeling and time series prediction, etc., to construct a high-precision, predictable three-dimensional dynamic force field model to assist in rehabilitation evaluation and intelligent feedback regulation.

[0102] Firstly, the system collects raw data from various sensors distributed in the glove, including three-axis pressure sensors, capacitive tactile sensors, tension sensors, flexible bending sensors and inertial measurement units. These data are distributed in space on the palm, fingertips, joints and other positions, and may have problems such as sampling delay, communication jitter or inconsistent rate in time. Therefore, the data processing unit first performs spatial position embedding and time synchronization processing. Spatial position embedding refers to binding the data of each sensor with its geometric coordinates in the hand reference model to form a sensor index and spatial position mapping vector, ensuring the traceability of subsequent data in physical position. Time synchronization uses a sliding window resampling and timestamp-based interpolation method to align multi-channel signals, ensuring that all sensor data maintain a consistent time reference within each processing period.

[0103] Next, to improve the stability and modeling accuracy of the data, the system introduces a Kalman filter algorithm to perform multi-channel fusion and noise suppression processing on the synchronized raw signals. The Kalman filter uses the covariance matrix of the current observation value and the state estimate at the last time to perform weighted fusion, which can effectively filter high-frequency interference, measurement noise and transient distortion, and obtain continuous, smooth and reliable force and attitude data.

[0104] After data cleaning, the system constructs a four-dimensional force tensor flow field model based on the above processing results. The three spatial dimensions of the tensor correspond to the spatial displacements in X, Y and Z directions, and the fourth dimension is the time dimension, i.e., the tensor F(x, y, z, t) contains a complete three-dimensional force distribution image at each time frame. The value in each voxel unit represents the magnitude of the resultant force at the corresponding spatial position at the current time, and can also record information such as the force direction vector and the duration of action of the position.

[0105] To further realize the intelligent prediction of the force field evolution trend, the system inputs the time series tensor sequence into a long short-term memory network (LSTM) for modeling. The LSTM model can process long-term dependent time series data and effectively retain the key dynamic characteristics of the historical force vector by introducing a memory gating mechanism. The system predicts the development trend of the current force vector sequence at future time points by training the LSTM, and outputs the following core indicators: the change trend of the force size (ΔF); the change rate of the direction angle (dθ / dt); the moving path of the center of pressure (COP) trajectory; and the risk level prediction of the high-frequency force application area (such as the thenar high-frequency repeated force point).

[0106] Finally, the three-dimensional dynamic force field model not only has high-precision force application information at the current time, but also has a certain degree of prediction ability, can actively perceive the risk force trend, and realizes the feedforward feedback regulation combined with the personalized safety threshold. For example, before detecting that the force application trend in a specific area is about to enter the over-limit interval, the system can activate the tactile feedback or AR visual warning in advance to assist medical personnel to adjust the hand movement in time, and improve the safety and accuracy of the rehabilitation training.

[0107] In summary, the data processing flow of the embodiment improves the force field perception accuracy and spatial resolution of the system by constructing a tensor expression structure from multi-source sensor data and combining a time series deep learning method, and introduces intelligent prediction ability, realizes the rehabilitation intelligent interaction closed loop from perception to understanding, from response to pre-control.

[0108] In a feasible implementation, the AR visualization component includes an augmented reality glasses, in which the force vector is displayed in the form of an arrow, and the length of the arrow represents the force size and the color represents the safety level.

[0109] For example, to improve the intuitive perception and decision-making efficiency of medical personnel on the force application behavior of the patient's hand during rehabilitation treatment, the AR visualization component of the present application uses augmented reality glasses as the main interactive carrier to realize real-time visualization of three-dimensional force vector information, and provides intuitive force application safety prompts in combination with color coding and arrow graphics.

[0110] Specifically, the three-dimensional dynamic force field model constructed by the system through the data processing unit outputs the force size, direction and action path information of each key position (such as fingertips, heel, joint area, etc.) in real time. The force vector information is then encoded into a standard visualization element and projected into the user's field of view of the augmented reality glasses (such as HoloLens, Magic Leap, etc.).

[0111] In the display mechanism, each force vector is displayed as a spatial arrow on the virtual model of the corresponding hand part. The specific form is as follows:

[0112] The length of the arrow is proportional to the size of the applied force. For example, the greater the force, the longer the arrow. In actual calibration, the system sets the maximum arrow length to correspond to a force of 10N, and automatically limits the display beyond the range.

[0113] The direction of the arrow corresponds to the real direction of the current force (based on the joint angle and the calculation result of the inertial unit), and the correct direction is ensured through spatial coordinate mapping, and the direction changes synchronously with the hand posture.

[0114] The color of the arrow represents whether the current force is within the individualized safety threshold range. The color coding rules are as follows: green indicates that the force is within the safety range, and the system determines that no intervention is needed; yellow indicates that the force is close to the upper threshold, which prompts attention but does not trigger feedback; red indicates that the force exceeds the safety range, which will trigger the tactile or voice feedback mechanism simultaneously.

[0115] In addition, to enhance the awareness of medical personnel on the overall force distribution, the system also displays the historical force heat map of each key joint area in the AR field as a transparent overlay layer, so that they can "see" the force hot spot area and high-frequency pressure path in vision, which is convenient for judging whether the movement is deviated or fatigued misuse.

[0116] In the AR interaction detail design, the system supports the following functions: real-time frame rate rendering ≥30fps, ensuring no visual delay; supporting view locking and instruction control, medical personnel can switch different observation angles, turn on / off the heat map layer, and view historical force trends through sight or gestures; when detecting abnormal force trends (such as direction mutation, short-time high pressure), the system will dynamically flash or enlarge the arrow, and at the same time display warning prompt text on the corner of the glasses.

[0117] Through the above mechanism, the AR visualization component of the embodiment not only realizes efficient and intuitive three-dimensional force feedback information presentation, but also significantly improves the real-time judgment ability of medical personnel on patient operation safety through multi-modal fusion, providing a powerful visual decision support means for precise rehabilitation intervention.

[0118] In a feasible implementation, the above-mentioned tactile feedback component includes at least 4 linear resonance motors arranged on the back of the hand, which outputs vibration according to the force abnormality level.

[0119] For example, to enhance the real-time safety perception ability of medical staff during rehabilitation operation, the above-mentioned tactile feedback component is designed to include at least four linear resonant actuators (LRAs) arranged in the back of the hand area. Through a hierarchical vibration output mechanism, the abnormal degree of force application state is fed back to the medical staff in different intensity and frequency vibrations, forming a non-visual real-time warning channel.

[0120] Specifically, the four LRA motors are arranged in a matrix symmetric manner in the back of the hand, i.e. the corresponding parts of the back of the thumb, index finger, middle finger and little finger, covering the main feedback nerve area, so that the vibration prompt can be clearly perceived by the medical staff and will not interfere with the hand operation. The data processing unit compares the current applied force with the individualized force safety threshold while monitoring the force application behavior of each key area in real time, and triggers vibration feedback when the following conditions exist:

[0121] If the force application direction mutation causes the angle drift to exceed the set angle (such as 15°), the local pressure is concentrated in a single point area for a long time, triggering the time-intensity combined abnormality recognition mechanism.

[0122] The vibration level classification and feedback strategy can adopt the following methods:

[0123] Primary feedback (mild abnormality): such as when the force value reaches 90% of the upper limit of the threshold, a single vibration of weak shock frequency 100Hz, lasting 200ms, is triggered and executed by the corresponding back of the hand motor;

[0124] Secondary feedback (moderate abnormality): such as when the force value has exceeded the threshold but has not reached a serious over-standard, intermittent vibration (2 times) of medium shock frequency 150Hz, cycle 500ms is triggered;

[0125] Tertiary feedback (serious abnormality): such as when the force value exceeds the threshold by more than 20%, or the continuous time is over-limit, a high-intensity continuous vibration frequency of 200Hz, lasting 1.5 seconds, is triggered, and is synchronized with the voice feedback.

[0126] If abnormal force application occurs in a specific finger area (such as the tip of the index finger), the motor unit closest to the nerve mapping of that position is preferentially activated to achieve directional feedback and enhance the spatial sense of the feedback. If multiple areas simultaneously exhibit abnormalities, multiple motor units are triggered to vibrate in coordination, and a progressive rhythm is used to increase the alertness of the medical staff.

[0127] Medical staff can adjust the feedback intensity (such as amplitude increase or decrease) through the matching mobile terminal App; the system supports dynamically switching the feedback sensitivity in specific rehabilitation stages (such as the initial stage, the intensive stage) to adapt to different rehabilitation intervention strategies.

[0128] Through the above-mentioned embodiments, the present haptic feedback assembly conveys the intelligent detection result to the operator in a manner that is naturally perceptible to the body in real time, realizes a precise, fast and continuous risk prompting mechanism without relying on the visual channel, and significantly enhances the safety control ability and response efficiency in the treatment process.

[0129] In a feasible embodiment, a flexible lithium polymer battery (200 mAh, endurance 8 hours) can be built in the glove, supporting wireless charging (Qi standard).

[0130] The above examples are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal intelligent interactive visual rehabilitation assessment and treatment glove, characterized in that, include: Flexible glove body; A multi-source sensor array, comprising a micro-array triaxial pressure sensor deployed in the metacarpophalangeal joint region, a capacitive tactile sensor deployed in the fingertips and finger pads, a piezoresistive tension sensor deployed in the thenar / hypothenar eminence region, a flexible bending sensor deployed at the joints, and an inertial measurement unit deployed in the back of the hand region. The data processing unit is configured to fuse the signals collected by the multi-source sensor array and establish a three-dimensional dynamic force field model that includes force magnitude, direction, force application speed and path. The interactive feedback module includes an AR visualization component, a tactile feedback component, and a voice feedback component. The AR visualization component projects three-dimensional force field information, patient muscle tension thermograms, and rhythmic spectrum information into the field of vision of medical personnel. The tactile feedback component provides vibration prompts to medical personnel when the applied force exceeds an individualized force safety threshold. The voice feedback component broadcasts abnormal force prompts.

2. The multimodal intelligent interactive visual rehabilitation assessment and treatment glove according to claim 1, characterized in that, The gloves also include: The personalized assessment module is configured to determine the individualized force application safety threshold based on bone density, spasticity level, tissue elastic modulus, and pain threshold information in the patient's electronic medical record.

3. The multimodal intelligent interactive visual rehabilitation assessment and treatment glove according to claim 2, characterized in that, The personalized evaluation module includes a feature vector encoding unit, a threshold inference unit, and a dynamic adjustment module; The feature vector encoding unit is used to extract the bone density, spasticity level, tissue elastic modulus and pain threshold information from the patient's electronic medical record and map them into a multidimensional feature vector; The threshold reasoning unit maps the multidimensional feature vector into an individualized force application safety threshold through a neural network model. The dynamic adjustment module calculates the threshold adjustment factor in real time based on the patient's changes in muscle tone, pain feedback, and electromyographic signals during the rehabilitation process, and updates the threshold of the safe force application range.

4. The multimodal intelligent interactive visual rehabilitation assessment and treatment glove according to claim 2, characterized in that, The neural network model in the threshold inference unit includes: An input layer is used to receive a multidimensional normalized feature vector containing information on the bone density, the spasticity level, the tissue elastic modulus, and the pain threshold. At least two fully connected hidden layers, the first hidden layer contains 16 neurons and the second hidden layer contains 8 neurons, the activation function is ReLU, used to perform non-linear mapping and feature combination learning on the input features; The output layer is a continuous regression output layer containing two neurons, used to predict the upper and lower limits of the individualized force application safety threshold range for patients, using the Softplus activation function; The neural network model is trained using a composite loss function, which includes a mean squared error term and an interval stability regularization term.

5. The multimodal intelligent interactive visual rehabilitation assessment and treatment glove according to claim 1, characterized in that, The triaxial pressure sensor is a microarray sensor made of MWCNTs / PDMS composite material, encapsulated in a medical silicone microdome structure.

6. The multimodal intelligent interactive visual rehabilitation assessment and treatment glove according to claim 1, characterized in that, The capacitive tactile sensor uses an interdigital electrode design and is printed on an elastic substrate in the palm.

7. The multimodal intelligent interactive visual rehabilitation assessment and treatment glove according to claim 1, characterized in that, The inertial measurement unit includes an accelerometer, a gyroscope, and a magnetometer, and is configured to calculate hand posture and spatial trajectory in real time.

8. The multimodal intelligent interactive visual rehabilitation assessment and treatment glove according to claim 1, characterized in that, The specific processing steps of the data processing unit include: The multi-source sensor data is subjected to spatial location embedding and time synchronization processing, and signal alignment and interference suppression are achieved based on Kalman filtering. A four-dimensional force tensor flow field is constructed, and a long short-term memory network is used to model and predict the trend of continuous force vectors, generating a three-dimensional dynamic force field model containing information on force magnitude, direction, velocity and path change.

9. The multimodal intelligent interactive visual rehabilitation assessment and treatment glove according to claim 1, characterized in that, The AR visualization component includes augmented reality glasses, in which force vectors are visualized as arrows, with the arrow length representing the force magnitude and the color representing the safety level.

10. The multimodal intelligent interactive visual rehabilitation assessment and treatment glove according to claim 1, characterized in that, The tactile feedback component includes at least four linear resonant motors arranged on the back of the hand, which provide graded vibration output according to the level of abnormal force applied.

Images