Rehabilitation evaluation system and method based on multi-modal flexible sensing

Abstract

The invention belongs to the technical field of intelligent evaluation, and provides a rehabilitation evaluation system and method based on multi-modal flexible sensing, and the system comprises a multi-modal flexible sensing glove and a data processing device. The multi-mode flexible sensing glove comprises a contact layer, a sensing layer, a protective layer and a control assembly, the contact layer, the sensing layer and the protective layer are sequentially arranged from inside to outside, the control assembly is arranged on the protective layer and located on the back of the glove, and a sensing assembly is arranged in the sensing layer; the sensing assembly is used for collecting multi-modal data around a target part, and the control assembly is used for primarily processing the multi-modal data; and the data processing equipment is used for performing rehabilitation evaluation on the target part through wavelet decomposition by using a rehabilitation evaluation model established based on a convolutional neural network and a long-short-term memory network according to the primarily processed multi-modal data to obtain a rehabilitation evaluation result. According to the scheme provided by the invention, the accuracy, reliability and feasibility of a rehabilitation evaluation link are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent assessment technology, and in particular to a rehabilitation assessment system and method based on multimodal flexible sensing. Background Technology

[0002] In the field of rehabilitation medicine, therapists' assessments of patients' target areas (such as joints and muscle groups) (e.g., muscle strength testing, joint range of motion determination) are a core component of monitoring the rehabilitation process. Currently, rehabilitation assessments mainly rely on therapists' manual manipulation and subjective judgment, lacking quantitative data support. This makes it difficult to standardize assessment results. Furthermore, there are issues with the adaptability of sensors to clinical scenarios. Some existing force feedback devices use rigid sensors, which are impractical due to their large size, high rigidity, and interference with tactile feedback. Additionally, the data acquisition process is disconnected from the rehabilitation assessment, making it difficult to generate effective assessment results.

[0003] Therefore, traditional rehabilitation assessment programs suffer from technical problems such as low accuracy, poor reliability, and insufficient feasibility. Summary of the Invention

[0004] This invention provides a rehabilitation assessment system and method based on multimodal flexible sensing, which addresses the shortcomings of traditional rehabilitation assessment schemes, such as low accuracy, poor reliability, and insufficient feasibility.

[0005] On the one hand, the present invention provides a rehabilitation assessment system based on multimodal flexible sensing, including: multimodal flexible sensing gloves and data processing equipment; The multimodal flexible sensing glove includes: a contact layer, a sensing layer, a protective layer, and a control component; The contact layer, sensing layer, and protective layer are arranged sequentially from the inside to the outside. The control component is located on the protective layer and on the back of the glove. The sensing layer contains a sensing component, which is connected to the control component. The control component is connected to the data processing device. The sensing component is used to collect multimodal data around the target area, and the control component is used to perform preliminary processing on the multimodal data; The data processing device is used to perform rehabilitation assessment on the target area based on the pre-processed multimodal data, through wavelet decomposition and using a rehabilitation assessment model built on convolutional neural networks and long short-term memory networks, to obtain rehabilitation assessment results.

[0006] According to the rehabilitation assessment system based on multimodal flexible sensing provided by the present invention, the sensing components include a pressure sensing array, a bending sensor, and an inertial measurement unit arranged in different regions. The pressure sensor array is used to collect key pressure data in different areas, the bending sensor is used to collect finger joint angle data, and the inertial measurement unit is used to collect hand movement posture data. The arrangement density of the pressure sensor array in the fingertip area is 120 points / cm², and the arrangement density of the pressure sensor array in the palm area is 30 points / cm².

[0007] According to the rehabilitation assessment system based on multimodal flexible sensing provided by the present invention, the control components include: a core processor, a dual-mode wireless transmission module, and a magnetically removable battery; The core processor is connected to the sensing component, the dual-mode wireless transmission module, and the magnetically removable battery, respectively. The dual-mode wireless transmission module is connected to the data processing device.

[0008] According to the rehabilitation assessment system based on multimodal flexible sensing provided by the present invention, the contact layer is made of medical silicone, and the surface of the contact layer is provided with biomimetic micro-textures. The biomimetic micro-texture has a texture height of 50 μm and a spacing of 200 μm between adjacent textures.

[0009] According to the rehabilitation assessment system based on multimodal flexible sensing provided by the present invention, the control component performs preliminary processing on the multimodal data, including: Extract outlier data points from the multimodal data and mark the outlier data points.

[0010] According to the rehabilitation assessment system based on multimodal flexible sensing provided by the present invention, the data processing device performs rehabilitation assessment on the target area based on the pre-processed multimodal data to obtain rehabilitation assessment results, including: The pre-processed multimodal data is then subjected to noise reduction and standard format conversion to obtain valid data. The valid data is input into a pre-established rehabilitation assessment model to obtain the rehabilitation assessment results output by the rehabilitation assessment model.

[0011] The rehabilitation assessment system based on multimodal flexible sensing provided by this invention performs noise reduction and standard format conversion on the pre-processed multimodal data to obtain effective data, including: Extract and remove outliers from the pre-processed multimodal data, and convert the outlier-removed multimodal data into a one-dimensional array; Perform multi-level wavelet decomposition on the one-dimensional array to obtain multi-level key coefficients; For the high-frequency detail coefficients in the multi-layer key coefficients, the optimal noise reduction threshold for each layer is determined. Based on the optimal noise reduction threshold, the high-frequency detail coefficients of each layer are thresholded to obtain the noise-reduced high-frequency detail coefficients. Wavelet reconstruction is performed on the multi-layer key coefficients, which include the noise-reduced high-frequency detail coefficients, and then converted to a set medical standard format to obtain effective data.

[0012] According to the rehabilitation assessment system based on multimodal flexible sensing provided by the present invention, the rehabilitation assessment model includes: A convolutional neural network layer is used to extract spatial features from the effective data; A long short-term memory network layer is used to extract temporal features from the effective data; A fully connected layer is used to concatenate the spatial features and the temporal features to output a fused feature vector; An evaluation output layer is used to output rehabilitation assessment results containing muscle strength level and multiple index parameters based on the fused feature vector.

[0013] The rehabilitation assessment system based on multimodal flexible sensing provided by the present invention further includes: an audible and visual alarm; The audible and visual alarm is connected to the data processing device; The data processing device is also used to determine whether the current applied force exceeds the limit based on the pre-processed multimodal data. If the current applied force exceeds the limit, an over-limit warning signal is generated and sent to the audible and visual alarm to trigger the audible and visual alarm to perform an audible and visual warning.

[0014] On the other hand, the present invention also provides a rehabilitation assessment method based on multimodal flexible sensing, based on any of the above-described rehabilitation assessment systems based on multimodal flexible sensing, the method comprising: Multimodal data around the target area is collected by sensing components within the sensing layer; The multimodal data is initially processed by a control component located on the protective layer and on the back of the glove. Based on the pre-processed multimodal data, the data processing equipment performs rehabilitation assessment on the target area through wavelet decomposition and utilizes a rehabilitation assessment model built on convolutional neural networks and long short-term memory networks to obtain rehabilitation assessment results.

[0015] The rehabilitation assessment system and method based on multimodal flexible sensing provided by this invention, through the collaborative design of multimodal flexible sensing gloves and data processing equipment, utilizes sensing components to achieve accurate acquisition of multimodal data around the target area, avoiding information loss problems caused by discrete sensing. The control component performs preliminary processing on the acquired multimodal data, providing high-quality data input for subsequent assessment. Based on the pre-processed data, the data processing equipment directly outputs rehabilitation assessment results through wavelet decomposition and a rehabilitation assessment model built based on convolutional neural networks and long short-term memory networks, realizing a closed loop of the entire process of data acquisition, processing, and assessment. This significantly improves the objectivity and standardization of assessment results, promotes the transformation of rehabilitation assessment from qualitative to quantitative, and enhances the accuracy, reliability, and feasibility of the rehabilitation assessment process. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of the rehabilitation assessment system based on multimodal flexible sensing provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the control component in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the rehabilitation assessment method based on multimodal flexible sensing provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] The following is combined Figures 1 to 3 This invention describes the detailed scheme of a rehabilitation assessment system and method based on multimodal flexible sensing provided in embodiments of the present invention.

[0020] like Figure 1 As shown, the rehabilitation assessment system based on multimodal flexible sensing provided in this embodiment of the invention mainly includes: a multimodal flexible sensing glove 110 and a data processing device 120.

[0021] The multimodal flexible sensing glove 110 includes a contact layer, a sensing layer, a protective layer, and control components.

[0022] The contact layer, sensing layer, and protective layer are arranged sequentially from the inside to the outside. The control component is located on the protective layer and on the back of the glove. The sensing layer contains a sensing component, which is connected to the control component. The control component is connected to the data processing device 120.

[0023] The sensing component is used to collect multimodal data around the target area, and the control component is used to perform preliminary processing on the multimodal data. It can be understood that preliminary processing refers to basic data processing operations performed at the sensing device before wireless data transmission, with the aim of reducing data volume and transmission load, including but not limited to data tagging, basic filtering, and format encapsulation.

[0024] The data processing device 120 is used to perform rehabilitation assessment on the target area based on the pre-processed multimodal data, through wavelet decomposition and using a rehabilitation assessment model built on convolutional neural network and long short-term memory network, to obtain rehabilitation assessment results.

[0025] In one embodiment, the sensing component specifically includes a pressure sensing array, a bending sensor, and an inertial measurement unit arranged in different regions.

[0026] The pressure sensor array is used to collect key pressure data in different areas, the bending sensor is used to collect finger joint angle data, and the inertial measurement unit is used to collect hand movement posture data.

[0027] Key pressure data is a set of pressure data with spatiotemporal characteristics that reflects the interaction force between the target area and the contacting object during rehabilitation movements. Finger joint angle data is quantitative data describing the relative rotation angles of the bones at both ends of the joints during flexion, extension, adduction, and abduction movements. Hand movement posture data is comprehensive data describing the position, orientation, and movement state of the entire hand in three-dimensional space.

[0028] The density of the pressure sensor array in the fingertip area is 120 points / cm². 2 The density of the pressure sensor array in the palm area is 30 points / cm². 2 .

[0029] In this embodiment, the pressure sensing array can be implemented using an MXene / graphene composite film. Each detection unit of the composite film has a size of 5mm × 5mm and a sensitivity of 8.4 kPa. -1It features a high-voltage resistivity coefficient. When subjected to pressure, the density of conductive pathways within the composite film changes, leading to a change in resistance. This resistance signal is converted into a voltage signal via a serpentine copper conductor, enabling quantitative acquisition of the magnitude and distribution of applied force. The high-density arrangement in the fingertip area allows for precise capture of subtle differences in force applied to the fingertip. In practical applications, the serpentine copper conductor has a linewidth of 100μm and an elongation greater than 150%.

[0030] In some embodiments, hand movement posture data can be acquired by a 9-axis motion sensor. The 9-axis motion sensor can acquire the linear acceleration of the hand through an accelerometer, acquire the angular velocity through a gyroscope, and correct the directional deviation through a magnetometer. After fusing the above data, the range of motion and movement speed of the hand can be calculated.

[0031] In one embodiment, the contact layer may be made of medical-grade silicone, and the surface of the contact layer is provided with biomimetic micro-textures.

[0032] The biomimetic micro-texture has a texture height of 50 μm and a spacing of 200 μm between adjacent textures.

[0033] In this embodiment, the contact layer can be made of medical-grade silicone with a thickness of 0.5±0.1mm and has biomimetic micro-textures on the surface to ensure a tactile sensitivity of 0.1N.

[0034] Understandably, biomimetic micro-textures can simulate the texture of human skin, creating a tiny pressure gradient upon contact. This preserves the therapist's tactile perception of the hardness and elasticity of the patient's target area while using the flexible properties of silicone to conform to the hand's movement trajectory, thus avoiding movement interference.

[0035] In practical applications, the pressure sensor array can be bonded to the silicone substrate of the contact layer through vacuum hot pressing, and the conductive lines can be connected by curing conductive silver paste.

[0036] In this embodiment, the protective layer is made of antibacterial nylon fabric. The diameter of the pores in the protective layer is within 50μm, and it has been verified by ethylene oxide sterilization, which can effectively inhibit bacterial growth. The pore design can balance comfort and protection.

[0037] In practical applications, before rehabilitation assessment, the multimodal flexible sensing gloves can be mechanically calibrated. Specifically, standard weights, such as 0.1N, 1N, 5N, 20N, and 50N, can be applied to the multimodal flexible sensing gloves, and the voltage-force response curves can be recorded and fitted to obtain a linear equation to obtain reference information. This allows us to check whether the multimodal flexible sensing gloves are working properly.

[0038] In one embodiment, the control component performs preliminary processing on the multimodal data, specifically including: Extract outlier data points from multimodal data and mark them.

[0039] To prevent abnormal data points caused by sensor malfunctions and extreme operations (such as excessive pressure leading to sensor unit damage) from entering subsequent processes, the control component can execute a simplified anomaly detection process. Specifically, it can perform anomaly detection on multimodal data based on the normal operating threshold range of the sensor, thereby extracting abnormal data points. For detected abnormal data points, a replacement strategy using the previous valid data point can be adopted to avoid data chain breakage. At the same time, it can also drive the LED indicator on the back of the glove through the GPIO pin to indicate sensor signal abnormalities to the therapist. Specifically, the LED indicator can flash red to indicate the abnormality, and an anomaly marker bit can be added to the data frame to mark the abnormal data points.

[0040] In one embodiment, such as Figure 2 As shown, the control components specifically include: a core processor 210, a dual-mode wireless transmission module 220, and a magnetically removable battery 230.

[0041] The core processor 210 is connected to the sensing components, the dual-mode wireless transmission module 220 and the magnetically removable battery 230, respectively. The dual-mode wireless transmission module 220 is connected to the data processing device.

[0042] In this embodiment, the core processor 210 can be a RISC-V processor, which refers to a processor designed based on the RISC-V instruction set architecture. RISC-V is an open-source, free instruction set architecture with features such as simplicity and customizability.

[0043] The dual-mode wireless transmission module 220 can achieve multi-device networking through ZigBee 3.0, specifically supporting simultaneous connection of 5 gloves. BLE 5.2 enables short-range high-speed transmission with a transmission rate of over 2Mbps and a latency of less than 100ms. The magnetically removable battery 230 supports hot-swapping to avoid interruptions in the rehabilitation assessment process.

[0044] In one embodiment, the data processing device performs a rehabilitation assessment on the target area based on the pre-processed multimodal data to obtain the rehabilitation assessment results, specifically including: First, the pre-processed multimodal data is subjected to noise reduction and standard format conversion to obtain valid data.

[0045] In one specific implementation, the pre-processed multimodal data undergoes noise reduction and standard format conversion to obtain valid data, specifically including: The first step is to extract and remove outliers from the pre-processed multimodal data, and then convert the multimodal data after removing outliers into a one-dimensional array.

[0046] In this step, the multimodal data after preliminary processing needs to be preprocessed to remove abnormal isolated points caused by sensor initialization delay and transient interference from wire contact during signal acquisition. Since the multimodal data is essentially a pressure signal, the multimodal data after removing abnormal isolated points needs to be uniformly converted into a one-dimensional array of length N. N can be determined according to the evaluation duration. For example, a 5-minute evaluation corresponds to N = 5 × 60 × 100 = 30,000 data points, thus providing standardized input for subsequent wavelet decomposition.

[0047] The second step is to perform multi-level wavelet decomposition on the one-dimensional array to obtain multi-level key coefficients.

[0048] In this step, the Daubechies 5 (db5) wavelet basis function can be selected for wavelet decomposition. This basis function has the characteristics of good tight support and high smoothness, which can effectively match the gradual characteristics of the force signal in rehabilitation assessment and avoid decomposition distortion caused by mismatch between the basis function and the signal characteristics.

[0049] This embodiment specifically employs a 3-layer wavelet decomposition. The 3-layer wavelet decomposition can accurately decompose the pressure signal into 3 layers of high-frequency detail coefficients d1, d2, and d3 and 1 layer of low-frequency approximation coefficient a3. Among them, d1 corresponds to the high-frequency noise of the conductor in the frequency range of 10-50Hz, d2 corresponds to the slight operation vibration in the frequency range of 5-10Hz, d3 corresponds to the shaking of the treatment bed in the frequency range of 1-5Hz, and a3 corresponds to the core force signal in the frequency range of 0-1Hz, thereby achieving layered isolation between noise and effective signal.

[0050] Furthermore, the specific process of performing multi-level wavelet decomposition on a one-dimensional array is as follows: In the first-level decomposition stage, the first wavelet decomposition is performed on the one-dimensional array using db5 as the basis function to obtain the first-level high-frequency detail coefficients d1 (mainly including high-frequency interference such as wire noise and slight hand tremors) and the first-level low-frequency approximation coefficients a1 (including a mixed signal of force application signal and some low-frequency noise).

[0051] In the second-level decomposition stage, using db5 as the basis function, the low-frequency approximation coefficient a1 obtained in the first level is decomposed for the second time to obtain the high-frequency detail coefficient d2 of the second level (mainly containing mid-frequency interference such as slight vibration of the treatment bed) and the low-frequency approximation coefficient a2 of the second level (the force signal outline is clearer and the interference is further reduced).

[0052] In the third decomposition stage, using db5 as the basis function, the low-frequency approximation coefficient a2 obtained in the second stage is decomposed for the third time to obtain the high-frequency detail coefficient d3 of the third stage (mainly containing low-frequency interference such as the shaking of the treatment bed) and the low-frequency approximation coefficient a3 of the third stage (core effective force signal with very few interference components).

[0053] In the coefficient storage stage, the multi-layer key coefficients obtained from the decomposition, which include three layers of high-frequency detail coefficients d1, d2, and d3 and one layer of low-frequency approximation coefficient a3, are stored in matrix form. The length of each coefficient array corresponds to the length of the original signal array. Specifically, it can be automatically matched by the downsampling characteristics of wavelet decomposition. For example, if the original signal length N = 30000, the length of each coefficient array after decomposition is approximately N / 2³ = 3750.

[0054] The third step is to determine the optimal noise reduction threshold for each layer of high-frequency detail coefficients in the multi-layer key coefficients, and then perform threshold processing on the high-frequency detail coefficients of each layer based on the optimal noise reduction threshold to obtain the noise-reduced high-frequency detail coefficients.

[0055] In this step, for the three high-frequency detail coefficients d1, d2, and d3, the optimal noise reduction threshold for each layer is calculated using the SURE (Stein's Unbiased Risk Estimate) thresholding rule. The specific calculation process is as follows: For the high-frequency detail coefficient d1, due to the high noise intensity and concentrated distribution, a higher threshold λ1 is calculated using the SURE formula to ensure the complete elimination of high-frequency interference from the conductor.

[0056] For the high-frequency detail coefficient d2, with moderate noise intensity, a moderate threshold λ2 is calculated, which removes jitter while preserving small changes in the applied force signal.

[0057] For the high-frequency detail coefficient d3, which has low noise intensity and is close to the frequency of the applied force signal, a lower threshold λ3 is calculated to avoid erroneously removing the low-frequency effective components in the applied force signal.

[0058] The SURE formula is:

[0059] Where, x i Let be the i-th element in the high-frequency detail coefficient array, M be the length of the coefficient array, λ be the threshold to be optimized, and I() be an indicator function that takes the value 1 if the condition is met, and 0 otherwise. This represents the standard deviation of noise.

[0060] In practical applications, the calculated higher threshold λ1, medium threshold λ2, and lower threshold λ3 can be substituted into the SURE formula to verify whether the corresponding risk value SURE(λ) is the minimum value. If there are multiple local minima, the threshold that makes the noise removal rate greater than 90% and the effective signal retention rate greater than 95% is selected as the optimal noise reduction threshold.

[0061] Furthermore, a soft thresholding method is used to apply the obtained optimal noise reduction threshold to the three layers of high-frequency detail coefficients to remove noise components. The specific operation is as follows: For each element x in the high-frequency detail coefficient array i Adjustments will be made according to the following rules: If element x i If the value is greater than the optimal noise reduction threshold for the corresponding layer, then the coefficient value after thresholding is element x. i The difference between x and the optimal noise reduction threshold; if element x i If the coefficient is less than the negative of the optimal noise reduction threshold for the corresponding layer, then the coefficient value after thresholding is element x. i The sum of the sum and the optimal noise reduction threshold, if element x i If the absolute value of the threshold is less than the optimal noise reduction threshold, then the coefficient value after thresholding is 0.

[0062] In this embodiment, the low-frequency approximation coefficient a3 is not thresholded and its original value is directly retained, which can ensure the integrity of the force application signal.

[0063] The fourth step involves wavelet reconstruction of the multi-layer key coefficients, including the noise-reduced high-frequency detail coefficients, and conversion to a set medical standard format to obtain valid data.

[0064] Wavelet reconstruction is performed on the multi-layer key coefficients, including the denoised high-frequency detail coefficients, as follows: In the first reconstruction stage, using db5 as the basis function, the high-frequency detail coefficients d3' and a3 after noise reduction are inversely transformed to obtain the reconstructed low-frequency approximation coefficients a2'.

[0065] In the second reconstruction stage, the high-frequency detail coefficients d2' and a2' after noise reduction are inversely transformed to obtain the reconstructed low-frequency approximation coefficients a1'.

[0066] In the third reconstruction stage, the high-frequency detail coefficients d1' and a1' after noise reduction are inversely transformed to obtain the final wavelet reconstruction result.

[0067] In practical applications, the effectiveness of the wavelet reconstruction results can be verified by calculating the signal-to-noise ratio (SNR) and root mean square error (RMSE). If the SNR and RMSE meet the set threshold conditions, the system is considered to have met the criteria. For example, if the SNR is above 25dB and the RMSE is below 0.05V, the system meets the criteria. If the system does not meet the criteria, the system returns to the threshold determination stage to re-determine the optimal denoising threshold until the criteria are met.

[0068] In the format conversion stage, the pressure signal reconstructed by wavelet can be converted into a time series of applied force values ​​and output in a format that conforms to the HL7 FHIR medical standard.

[0069] Then, the valid data is input into the pre-established rehabilitation assessment model to obtain the rehabilitation assessment results output by the rehabilitation assessment model.

[0070] In this embodiment, the rehabilitation assessment model specifically includes: Convolutional neural network layers are used to extract spatial features from valid data.

[0071] Understandably, convolutional neural network layers can perform convolution operations on effective data using 3×3 convolutional kernels. The first convolutional layer extracts local force hotspots, such as pixel-level features of areas where force is concentrated. The second convolutional layer fuses the force difference features of adjacent sensing units. The third convolutional layer outputs global pressure distribution features, such as the position of the force center of gravity and pressure uniformity. Furthermore, the data dimensionality is reduced through max pooling operations, while retaining key spatial feature information.

[0072] Long Short-Term Memory (LSTM) network layers are used to extract temporal features from valid data.

[0073] It is understandable that long short-term memory network layers can extract temporal features through a bidirectional structure, namely, the forward structure captures the trend of temporal data and the backward structure captures the inverse dependencies.

[0074] The fully connected layer is used to concatenate spatial features and temporal features to output a fused feature vector.

[0075] Understandably, a fully connected layer can concatenate the spatial features extracted from the two branches with the temporal features to form a fused feature vector with unified dimensions.

[0076] The evaluation output layer is used to output rehabilitation assessment results containing muscle strength level and multiple index parameters based on the fused feature vector.

[0077] In this embodiment, the evaluation output layer can be implemented using the XGBoost model. Through training with samples from different scenarios, the evaluation output layer can output different types of rehabilitation evaluation results for different application scenarios. Taking muscle strength testing as an example, referring to the 0-5 level of the manual muscle strength testing standard, the XGBoost model learns the mapping relationship between force application characteristics (such as peak force when resisting resistance), motion characteristics (such as joint range of motion during muscle resistance), and the patient's basic information to output the muscle strength level. The specific muscle strength level settings are as follows: Grade 0: No muscle contraction, corresponding force value less than 0.1N, joint range of motion equal to 0°.

[0078] Level 3: Can complete a full range of motion against moderate resistance, with a corresponding force value in the range of 5-10N, and the joint range of motion reaches more than 90% of the normal range.

[0079] For other levels of settings, the precise division of the grade boundaries can be achieved through feature weight allocation, based on the accuracy requirement of ±2.5% of the applied force error.

[0080] In practical applications, rehabilitation assessment results also include a variety of indicators and parameters, such as range of motion, Brunnstrom stage, Fugl-Meyer assessment score, degree of spasticity, coordination score, and other key information.

[0081] Understandably, by extracting core features such as angle, motion, and force from multimodal data, a precise mapping can be established with various rehabilitation assessment indicators, realizing the transformation from raw data to rehabilitation conclusions.

[0082] Specifically, the range of motion parameter can be directly calculated based on multimodal data. The flexion sensor records the range of angle changes during finger joint flexion and extension. For example, the difference between the PIP joint of the index finger from 0 degrees (extended) to 85 degrees (flexed) can be compared with the set normal range to determine whether the joint is restricted and to what extent. The inertial measurement unit collects changes in hand posture angles, such as the difference in pitch angle from horizontal to vertical, which can assess the overall range of motion of the hand and provide a quantitative basis for wrist joint mobility.

[0083] The Brunnstrom stage indicator can be determined by combining multiple data features. When the inertial measurement unit detects a low rate of change in the finger joint angle and the key pressure data shows that the force is concentrated in the palm, it meets the characteristics of stage III, which is the peak of associated movement. If the inertial measurement unit captures the separation movement, the finger joint can flex and extend independently and the angular velocity is stable, and the key pressure data shows that the fingertip can apply force independently, it corresponds to stage V, which is the obvious separation movement. Thus, the movement characteristics of different recovery stages can be accurately matched.

[0084] The Fugl-Meyer Assessment Scale score is obtained by mapping data to each item. For the wrist dorsiflexion item, if the flexion sensor detects active dorsiflexion exceeding 30 degrees and the inertial measurement unit shows a posture angle fluctuation of less than 2 degrees within 5 seconds, 2 points are awarded. For the finger grasping item, scores are based on whether the finger joint flexion angle matches the grasping action and whether the pressure distribution is uniform. The system accumulates the scores of each item to generate the final scale score, while simultaneously identifying areas where points are lost to clarify training priorities.

[0085] The degree of spasticity can be determined through angle-force curve analysis. When the therapist passively flexes and extends the patient's fingers, the pressure sensor array records the magnitude of the force, and the bending sensor simultaneously records the angle data of the finger joint. If the resulting curve is smooth without abrupt changes, it corresponds to grade 0 without spasticity. If the curve shows a sudden increase in force at a specific angle, and the inertial measurement unit detects a sudden decrease in angular velocity, it indicates muscle stiffness. Combined with the magnitude of the abrupt change, it can be further determined to be grade I to IV spasticity, thereby quantifying the resistance during passive muscle stretching.

[0086] The coordination score is an indicator that relies on the motion data characteristics of the inertial measurement unit. It is evaluated by recording the time it takes for the hand to complete the action of touching the target part and the smoothness of the angular velocity curve: if the action takes a short time and there is no obvious angular velocity pause, it indicates good coordination; if the action takes too long, the standard deviation of angular velocity fluctuation is large, or even the action is interrupted, it reflects poor coordination, thus providing direction for targeted training.

[0087] In practical applications, the training process of the rehabilitation assessment model is as follows: First, a sample dataset containing multimodal data samples such as pressure, angle, and motion posture, along with corresponding result annotation information, is constructed. After preprocessing operations such as cleaning and standardization, it is divided into a training set, a validation set, and a test set.

[0088] Next, the parameters of each layer of the model are initialized, and a layered training strategy is adopted. First, the convolutional neural network layer and the long short-term memory network layer are trained with spatial and temporal feature loss, and then the fully connected layer is trained with cross-entropy loss.

[0089] Subsequently, the output layer is evaluated through end-to-end training using a hybrid loss function. During training, strategies such as Dropout and pruning are employed to suppress overfitting.

[0090] Finally, the model is optimized based on actual accuracy requirements, and boundary feature deviations are corrected through manual review. Ultimately, the model is validated through a test set and tested in real-world scenarios to ensure that the model's evaluation accuracy, consistency, and other indicators meet the standards. After the parameters are solidified, a rehabilitation assessment model that meets the conditions for deployment and application can be obtained.

[0091] In one embodiment, the rehabilitation assessment system based on multimodal flexible sensing may further include: an audible and visual alarm.

[0092] The audible and visual alarm is connected to the data processing equipment.

[0093] The data processing equipment is also used to determine whether the current applied force exceeds the limit based on the pre-processed multimodal data. If the current applied force exceeds the limit, an over-limit warning signal is generated and sent to the audible and visual alarm to trigger the audible and visual alarm to perform the audible and visual warning.

[0094] Understandably, during the therapist's gloved procedure, the system can monitor the applied force in real time. Specifically, it compares each pressure value in the initially processed multimodal data with a pre-set pressure threshold. This threshold can be set reasonably according to actual needs. If a set number of consecutive pressure values ​​exceed the threshold, it can be determined that the applied force is excessive. At this point, an over-force warning can be generated and issued, triggering an audible and visual alarm. In practical applications, the audible and visual alarm can be installed on the back of the glove for easy reception of the warning.

[0095] In some embodiments, the system also supports the automatic generation of quantitative reports containing peak force, average speed, and range of motion, and supports comparison with historical data to meet multi-dimensional data analysis and viewing needs.

[0096] It should be noted that the entire system incorporates a privacy protection system throughout the entire process of data transmission, storage, and use. During transmission, the dual-mode wireless transmission unit utilizes the AES-128 symmetric encryption algorithm to encrypt data collected by the sensors, such as pressure, angle, and posture, as well as basic patient information, in real time, preventing data theft or tampering during transmission. When storing data, patient privacy information is encrypted using the RSA-2048 asymmetric encryption algorithm to generate a unique encryption key, which is only authorized to rehabilitation therapists and system administrators. Furthermore, the original data undergoes anonymization to remove identifying information directly associated with the patient. In addition, the system employs a tiered access control mechanism; ordinary medical staff can only view the assessment data of their assigned patients and cannot access complete privacy information. All data operations (such as viewing and exporting) generate operation logs for auditing, ensuring strict protection of patient privacy throughout the entire data lifecycle and complying with relevant medical data security regulations.

[0097] In summary, the rehabilitation assessment system based on multimodal flexible sensing provided by the embodiments of the present invention has at least the following beneficial effects compared with existing solutions: On the one hand, fundamental improvements are achieved through material innovation and algorithm optimization. Specifically, the pressure sensing accuracy is effectively improved through the rational setting of the sensor array, and the sterilization compatibility and tactile fidelity are improved through the rational setting of layer materials and structures. The spatial resolution of pressure sensing is improved through the rational setting of the monitoring point density in the sensor array.

[0098] On the other hand, through system-level optimization, in terms of operational standardization and equipment durability, real-time automatic recording has been achieved to replace manual measurement, improving the efficiency of rehabilitation assessment; and data-driven operational specifications have improved operational consistency, significantly reducing the risk of human error; and the layered structure design of the magnetically detachable battery and protective layer has enhanced the durability and stability of the equipment.

[0099] Based on the same general inventive concept, this invention also protects a rehabilitation assessment method based on multimodal flexible sensing. The rehabilitation assessment method based on multimodal flexible sensing provided by this invention will be described below. The rehabilitation assessment method based on multimodal flexible sensing described below and the rehabilitation assessment system based on multimodal flexible sensing described above can be referred to and correspond to each other.

[0100] like Figure 3As shown, the rehabilitation assessment method based on multimodal flexible sensing provided in this embodiment of the invention is implemented based on the rehabilitation assessment system based on multimodal flexible sensing provided in the above embodiments. The method mainly includes the following steps: Step 310: Collect multimodal data around the target area using the sensing components within the sensing layer.

[0101] Step 320: Perform preliminary processing of the multimodal data using a control component located on the protective layer and on the back of the glove.

[0102] Step 330: Based on the pre-processed multimodal data, the target area is rehabilitated using wavelet decomposition and a rehabilitation assessment model built on convolutional neural networks and long short-term memory networks, and the rehabilitation assessment results are obtained.

[0103] The specific implementation of each step in the methods described in the above embodiments has been described in detail in the embodiments of the relevant systems, and will not be elaborated further here.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A rehabilitation assessment system based on multimodal flexible sensing, characterized in that, include: Multimodal flexible sensing gloves and data processing equipment; The multimodal flexible sensing glove includes: a contact layer, a sensing layer, a protective layer, and a control component; The contact layer, sensing layer, and protective layer are arranged sequentially from the inside to the outside. The control component is located on the protective layer and on the back of the glove. The sensing layer contains a sensing component, which is connected to the control component. The control component is connected to the data processing device. The sensing component is used to collect multimodal data around the target area, and the control component is used to perform preliminary processing on the multimodal data; The data processing device is used to perform rehabilitation assessment on the target area based on the pre-processed multimodal data, through wavelet decomposition and using a rehabilitation assessment model built on convolutional neural networks and long short-term memory networks, to obtain rehabilitation assessment results.

2. The rehabilitation assessment system based on multimodal flexible sensing according to claim 1, characterized in that, The sensing components include a pressure sensing array, a bending sensor, and an inertial measurement unit arranged in different regions. The pressure sensor array is used to collect key pressure data in different areas, the bending sensor is used to collect finger joint angle data, and the inertial measurement unit is used to collect hand movement posture data. The arrangement density of the pressure sensor array in the fingertip area is 120 points / cm², and the arrangement density of the pressure sensor array in the palm area is 30 points / cm².

3. The rehabilitation assessment system based on multimodal flexible sensing according to claim 1, characterized in that, The control components include: a core processor, a dual-mode wireless transmission module, and a magnetically removable battery; The core processor is connected to the sensing component, the dual-mode wireless transmission module, and the magnetically removable battery, respectively. The dual-mode wireless transmission module is connected to the data processing device.

4. The rehabilitation assessment system based on multimodal flexible sensing according to claim 1, characterized in that, The contact layer is made of medical-grade silicone, and the surface of the contact layer is provided with biomimetic micro-textures; The biomimetic micro-texture has a texture height of 50 μm and a spacing of 200 μm between adjacent textures.

5. The rehabilitation assessment system based on multimodal flexible sensing according to claim 1, characterized in that, The control component performs preliminary processing on the multimodal data, including: Extract outlier data points from the multimodal data and mark the outlier data points.

6. The rehabilitation assessment system based on multimodal flexible sensing according to claim 1, characterized in that, The data processing device performs a rehabilitation assessment on the target area based on the pre-processed multimodal data, and obtains the rehabilitation assessment results, including: The pre-processed multimodal data is then subjected to noise reduction and standard format conversion to obtain valid data. The valid data is input into a pre-established rehabilitation assessment model to obtain the rehabilitation assessment results output by the rehabilitation assessment model.

7. The rehabilitation assessment system based on multimodal flexible sensing according to claim 6, characterized in that, The pre-processed multimodal data undergoes noise reduction and standard format conversion to obtain valid data, including: Extract and remove outliers from the pre-processed multimodal data, and convert the outlier-removed multimodal data into a one-dimensional array; Perform multi-level wavelet decomposition on the one-dimensional array to obtain multi-level key coefficients; For the high-frequency detail coefficients in the multi-layer key coefficients, the optimal noise reduction threshold for each layer is determined. Based on the optimal noise reduction threshold, the high-frequency detail coefficients of each layer are thresholded to obtain the noise-reduced high-frequency detail coefficients. Wavelet reconstruction is performed on the multi-layer key coefficients, which include the noise-reduced high-frequency detail coefficients, and then converted to a set medical standard format to obtain effective data.

8. The rehabilitation assessment system based on multimodal flexible sensing according to claim 6, characterized in that, The rehabilitation assessment model includes: A convolutional neural network layer is used to extract spatial features from the effective data; A long short-term memory network layer is used to extract temporal features from the effective data; A fully connected layer is used to concatenate the spatial features and the temporal features to output a fused feature vector; An evaluation output layer is used to output rehabilitation assessment results containing muscle strength level and multiple index parameters based on the fused feature vector.

9. The rehabilitation assessment system based on multimodal flexible sensing according to claim 1, characterized in that, The system also includes: an audible and visual alarm; The audible and visual alarm is connected to the data processing device; The data processing device is also used to determine whether the current applied force exceeds the limit based on the pre-processed multimodal data. If the current applied force exceeds the limit, an over-limit warning signal is generated and sent to the audible and visual alarm to trigger the audible and visual alarm to perform an audible and visual warning.

10. A rehabilitation assessment method based on multimodal flexible sensing, characterized in that, Based on the rehabilitation assessment system based on multimodal flexible sensing as described in any one of claims 1 to 9, the method includes: Multimodal data around the target area is collected by sensing components within the sensing layer; The multimodal data is initially processed by a control component located on the protective layer and on the back of the glove. Based on the pre-processed multimodal data, the data processing equipment performs rehabilitation assessment on the target area through wavelet decomposition and utilizes a rehabilitation assessment model built on convolutional neural networks and long short-term memory networks to obtain rehabilitation assessment results.

Images