Hand injury monitoring system, method, equipment and medium
By employing multimodal acquisition and cross-modal feature fusion techniques, the problems of insufficient portability and accuracy in existing hand injury monitoring devices have been solved, enabling accurate identification and stable assessment of hand motor function.
Patent Information
- Application Number
- CN202511636116.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-10
- Publication Date
- 2025-12-12
AI Technical Summary
Existing hand injury monitoring devices suffer from problems such as poor portability, high cost, lack of integration of data acquisition and processing, susceptibility to signal interference, and instability of single-modal assessment, making it difficult to meet the accuracy and stability requirements of hand injury rehabilitation monitoring.
A multimodal acquisition device is used to simultaneously acquire finger flexion signals and electromyography signals. Cross-modal feature fusion is performed through a data processing device, including STFT time-frequency transformation, normalization, time alignment, cross-modal stitching, and model optimization, to improve recognition accuracy and robustness.
It achieves accurate identification of hand movement function, reduces the impact of noise and individual differences, provides comprehensive rehabilitation data support, and improves the portability and accuracy of hand injury monitoring.
Smart Images
Figure CN121101545A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of biomedical monitoring technology, specifically to a hand injury monitoring system, method, device, and medium. Background Technology
[0002] In the field of biomedical monitoring and rehabilitation, electromyography (EMG) signals are frequently used to analyze muscle status and overall health. As a crucial organ for movement, monitoring motor dysfunction in the hand is vital for disease rehabilitation. Currently, commercially available surface EMG acquisition devices and hand rehabilitation measurement equipment have significant shortcomings. Structurally, existing devices are large, mostly desktop-style and fixed, lacking portable wearable designs, failing to meet patients' needs for home-based monitoring. Cost-wise, the devices are expensive, dependent on professional hospital environments and operation, making daily use costs unaffordable for ordinary patients. Regarding data acquisition and processing, existing devices often employ a single data acquisition dimension, with signal acquisition and processing modules frequently being separate designs requiring complex wiring connections, making them susceptible to external interference. Furthermore, the lack of integrated signal filtering and amplification hardware results in poor data synchronization and low accuracy. In addition, while there is some research on the combined monitoring of finger kinematics data and EMG signals, existing technologies still have deficiencies in hardware design. For example, some solutions lack sensor fixation structures adapted to hand movements, making sensors prone to displacement during monitoring and affecting data acquisition accuracy. Some solutions do not integrate a visual signal feedback structure, making it impossible to intuitively present the muscle activation state.
[0003] At the algorithmic level, current rehabilitation assessments commonly employ single-modal biosignals, such as using only sEMG signals (electromyography) or inertial sensor data to evaluate a patient's hand motor abilities. However, single signals suffer from incomplete information. sEMG reflects muscle electrical activity but cannot directly reflect the spatial angle of motor execution, while joint angle sensors, although capable of capturing the amplitude of movement, struggle to reveal muscle control characteristics. Traditional algorithms often employ convolutional or LSTM networks, modeling only single-modal data and failing to simultaneously understand the correlation between the time and frequency domains. Furthermore, in clinical rehabilitation scenarios, signals are significantly affected by noise, individual differences, and muscle fatigue, making single-modal methods prone to recognition fluctuations and resulting in unstable motor function scores. Current technology lacks an assessment algorithm that can simultaneously integrate time-domain, frequency-domain, and cross-modal features while maintaining robustness across small samples and different patients.
[0004] In summary, existing technologies have shortcomings in both hardware acquisition and algorithm analysis, making it difficult to meet the portability, accuracy, and stability requirements of hand injury rehabilitation monitoring. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a hand injury monitoring system, method, and medium. By collecting hand data from multiple dimensions and integrating electromyographic and kinematic information through a data processing device, cross-modal feature complementarity enhancement is achieved, thereby improving the accuracy and robustness of hand motor function recognition.
[0006] In a first aspect, the first embodiment of the present invention provides a hand injury monitoring system, comprising: a multimodal acquisition device and a data processing device, wherein the multimodal acquisition device is used to acquire multimodal data, perform preliminary processing to obtain processed multimodal data, and transmit the processed multimodal data to the data processing device for further processing; The multimodal acquisition device includes a finger data acquisition device and an electromyography (EMG) acquisition device, and the EMG acquisition device includes a signal processing unit; The finger data acquisition device is used to collect finger bending signals generated by various hand movements of the user; The electromyography (EMG) acquisition device uses EMG electrodes to collect EMG signals generated by the muscles of the hand and forearm when the user makes hand movements. The signal processing unit is only used to amplify, filter, and perform analog-to-digital conversion on the acquired electromyographic signals to obtain the processed electromyographic signals; the finger flexion signal is acquired by a resistor voltage divider and input to the data processing device for normalization and time alignment preprocessing to obtain the processed finger flexion signal. The data processing device is used to receive the processed data, preprocess the processed electromyographic signal to obtain spectral features, preprocess the processed finger flexion signal to obtain temporal features, perform cross-modal fusion of spectral features and temporal features, and output the hand movement function recognition result.
[0007] Furthermore, the data processing device includes a first preprocessing module, a second preprocessing module, a spectral feature path module, and a timing feature path module; The first preprocessing module is used to perform STFT time-frequency transformation on the processed electromyographic signal to obtain a spectrum, and input the spectrum into the spectrum feature path module; The second preprocessing module is used to normalize and time-align the processed curvature signal to obtain curvature time-series data, and input the time-series data into the time-series feature path module; The spectral feature path module is used to extract spectral features from the spectrogram. The temporal feature path module is used to extract temporal features from curvature temporal data.
[0008] Furthermore, the data processing device also includes a feature fusion and layering module, which is used to concatenate spectral features and temporal features to obtain a unified feature matrix, perform standardization processing on the concatenated feature matrix, perform dimensionality reduction and anti-overfit processing on the standardized data to obtain optimized features, input the optimized features into the classification layer to calculate the probability distribution of different hand movement function categories, and take the category with the highest probability as the hand movement function recognition result.
[0009] Furthermore, the data processing device also includes a model optimization and evaluation module, which uses K-fold cross-validation and an optimizer to optimize model parameters and ensure model performance.
[0010] Furthermore, the temporal feature path module is used to process the curvature temporal data sequentially through a fully connected layer, a dilated convolution stack, and a Transformer encoder, and output 128-dimensional temporal features. The spectral feature path module is used to process the input spectrogram sequentially through a convolutional layer and an axial attention module to extract 256-dimensional spectral features.
[0011] Secondly, a second embodiment of the present invention provides a method for monitoring hand injuries, applicable to the data processing device in the hand injury monitoring system described in the first embodiment above, the method comprising: The data processing device acquires the processed multimodal data transmitted by the multimodal acquisition device; The data processing device is used to receive the processed multimodal data, preprocess the processed electromyographic signal to obtain spectral features, preprocess the processed finger flexion signal to obtain temporal features, fuse the spectral features and temporal features across modalities, and output the hand movement function recognition result.
[0012] Furthermore, the preprocessing of the processed electromyographic signal to obtain spectral features, and the preprocessing of the processed finger flexion signal to obtain temporal features, include the following specific methods: The processed electromyographic signal is subjected to STFT time-frequency transformation to obtain a spectrogram, which is then input into the spectral feature path. The spectral feature extraction pathway extracts spectral features from the spectrogram. The processed curvature signal is normalized and time-aligned to obtain curvature time-series data, which is then input into the time-series feature path. The temporal feature pathway extracts temporal features from curvature temporal data; The specific method for cross-modal fusion of spectral features and temporal features includes: The spectral features and temporal features are concatenated to obtain a unified feature matrix. The concatenated feature matrix is then standardized. The standardized data is then subjected to dimensionality reduction and anti-overfitting processing to obtain optimized features. The optimized features are then input into the classification layer to calculate the probability distribution of different hand movement function categories. The category with the highest probability is taken as the hand movement function recognition result.
[0013] Furthermore, after the step of cross-modal fusion of spectral features and temporal features, the method further includes: using K-fold cross-validation and optimizer to optimize model parameters and ensure model performance.
[0014] Thirdly, another embodiment of the present invention provides an electronic device comprising: a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are interconnected, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to invoke the program instructions to execute the method described in the second embodiment of the present invention.
[0015] Fourthly, an embodiment of the present invention provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the method described in the second embodiment of the present invention.
[0016] The beneficial effects of this invention are: This invention provides a hand injury monitoring system, method, device, and medium. Through a multimodal acquisition device, it simultaneously acquires finger flexion signals and electromyographic (EMG) signals, reflecting both hand movement morphology and muscle activation status. Compared to single-dimensional monitoring devices, this provides a more accurate assessment of the hand injury rehabilitation stage, offering more comprehensive data support for adjusting rehabilitation plans. The data processing device integrates EMG and kinematic information, achieving cross-modal feature complementarity enhancement, improving the model's robustness in recognizing hand movement intentions, and reducing the impact of noise, individual differences, and muscle fatigue on the evaluation results. Attached Figure Description
[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.
[0018] Figure 1 A structural block diagram of a hand injury monitoring system provided in the first embodiment of the present invention is shown; Figure 2This diagram illustrates the structure of a hand injury monitoring system according to a first embodiment of the present invention. Figure 3 The diagram shows the overall architecture of a hand injury monitoring system according to the first embodiment of the present invention. Figure 4 A schematic diagram of the processing flow of a data processing device in a hand injury monitoring system provided in the first embodiment of the present invention is shown. Figure 5 A flowchart of a hand injury monitoring method provided by a second embodiment of the present invention is shown. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0022] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.
[0023] like Figure 1-4 As shown, a hand injury monitoring system provided by the first embodiment of the present invention is illustrated, including: a multimodal acquisition device and a data processing device. The multimodal acquisition device is used to acquire multimodal data, perform preliminary processing to obtain processed multimodal data, and transmit the processed multimodal data to the data processing device for further processing. The multimodal acquisition device includes a finger data acquisition device and an electromyography (EMG) acquisition device, and the EMG acquisition device includes a signal processing unit; The finger data acquisition device is used to collect finger bending signals generated by various hand movements of the user; The electromyography (EMG) acquisition device uses EMG electrodes to collect EMG signals generated by the muscles of the hand and forearm when the user makes hand movements. The signal processing unit is only used to amplify, filter, and perform analog-to-digital conversion on the acquired electromyographic signals to enhance weak signals and suppress noise, thereby obtaining processed electromyographic signals. The finger flexion signal is acquired by a resistor voltage divider method, without the need for computational amplification, and is directly input to the data processing device for normalization and time alignment to obtain the processed finger flexion signal.
[0024] The data processing device is used to receive the processed data, preprocess the processed electromyographic signal to obtain spectral features, preprocess the processed finger flexion signal to obtain temporal features, perform cross-modal fusion of spectral features and temporal features, and output the hand movement function recognition result.
[0025] In this embodiment, the finger data acquisition device is a wearable flexible glove. Made of flexible, breathable medical-grade material, the glove ensures comfort and long-term durability. Five fingers of the glove are embedded with flexure sensors, evenly distributed on the back of each finger, ensuring accurate detection of the flexion angle of each finger when the user bends it. The glove features an elastic adjustment band to accommodate different hand shapes while ensuring the sensors remain in close contact with the skin, preventing sensor displacement due to glove loosening. Each sensor converts the finger flexion angle into an electrical signal using a resistive voltage divider principle, which is then transmitted to the electromyography (EMG) acquisition device via a cable. The resistive voltage divider and operational amplifier structure of the finger data acquisition device stably converts changes in sensor resistance into voltage signals, avoiding errors caused by signal fluctuations and improving data accuracy. The flexure sensors continuously monitor the finger flexion angle. When the user performs various hand movements (such as bending and extending fingers), the flexure sensors capture the changes in finger flexion and convert them into electrical signals, transmitting them to the EMG acquisition device. The data processing device uses a computer terminal.
[0026] The electromyography (EMG) acquisition device is the core component of the multimodal acquisition system, integrating EMG acquisition, signal processing and display, power supply, and data transmission functions. The EMG acquisition device includes an EMG acquisition unit, a signal processing unit, a display unit, a power supply unit, and a data transmission unit. EMG signal acquisition is achieved by connecting 12 EMG electrodes via wires. These electrode patches are evenly distributed on the muscles of the hand and forearm. The electrodes transmit the weak signals they acquire to the EMG signal acquisition unit for recording, amplification, and processing. The signal processing unit includes an STM32 microcontroller, a signal gain unit, a signal filter, and an analog-to-digital converter (ADC). The signal processing unit amplifies, filters, and converts the acquired finger movement signals and EMG signals to digital. The EMG signals are displayed to the user in real time on an OLED screen, showing the intensity changes of the EMG signals. The use of a low-power STM32 microcontroller reduces the device's power consumption, allowing patients to monitor anytime, anywhere, solving the problems of traditional hospital equipment being "inconvenient and costly." The signal gain module and SPDT switch structure of the electromyography (EMG) data acquisition unit can adjust the signal output mode and channel gain according to monitoring needs. Combined with the built-in signal filter of the STM32 microcontroller, it effectively filters external noise and deep muscle signal interference. The EMG acquisition device is connected to the glove via wires to ensure synchronous transmission of signals from the curvature sensor and EMG electrodes. The entire device provides stable power support via a USB interface and transmits processed data to a computer terminal via serial communication.
[0027] The working principle of the electromyography (EMG) acquisition device: The device acquires EMG signals from the hand and forearm in real time via EMG electrode patches. These electrode signals are very weak and need to be amplified by the EMG acquisition unit. The signals acquired by the electrode patches are transmitted to the EMG acquisition unit via cable, where they are amplified to increase signal strength and ensure clear recording. After amplification, the signal enters the signal processing unit, where a signal gain unit further enhances the signal strength, a signal filter removes noise, and finally, an analog-to-digital converter (ADC) converts the analog signal into a digital signal. The processed data is displayed to the user in real time on an OLED screen, showing changes in muscle activation intensity. The processed data is also transmitted via USB to a data processing device for further analysis, storage, or display.
[0028] The data processing device includes a first preprocessing module, a second preprocessing module, a spectral feature path module, and a timing feature path module. The first preprocessing module performs STFT (Short Time Fourier Transform) time-frequency transformation on the processed electromyographic signal to obtain a spectrum, and inputs the spectrum into the spectral feature path module. The second preprocessing module normalizes and aligns the processed curvature signal in time to obtain curvature timing data, and inputs the timing data into the timing feature path module. The spectral feature path module extracts spectral features from the spectrum. The timing feature path module extracts timing features from the curvature timing data.
[0029] The temporal feature path module processes the curvature temporal data sequentially through a fully connected layer, a dilated convolutional stack, and a Transformer encoder to output 128-dimensional temporal features; the spectral feature path module processes the input spectrogram sequentially through a convolutional layer and an axial attention module to extract 256-dimensional spectral features.
[0030] The data processing device also includes a feature fusion and layering module. This module concatenates spectral features and temporal features to obtain a unified feature matrix. The concatenated feature matrix is then standardized, and the standardized data undergoes dimensionality reduction and overfitting prevention to obtain optimized features. These optimized features are input into a classification layer to calculate the probability distribution of different hand movement function categories. The category with the highest probability is taken as the hand movement function recognition result. The data processing device also includes a model optimization and evaluation module. This module uses K-fold cross-validation and an optimizer to optimize model parameters and ensure model performance.
[0031] The data processing unit receives the processed data, preprocesses the processed electromyography (EMG) signals to obtain spectral features, preprocesses the processed finger flexion signals to obtain temporal features, and fuses the spectral and temporal features across modalities to output the hand movement function recognition result. The data processing unit adopts a dual-tower time-frequency fusion recognition algorithm architecture, consisting of two parallel information processing paths: a time-series feature path and a spectral feature path. Feature fusion and classification are achieved through a fusion layer. The time-series feature path inputs the finger flexion time series, passes through a Dense (fully connected) layer, a dilated convolution stack, and a Transformer encoder, and outputs 128-dimensional features. The spectral feature path inputs a time-frequency map calculated from the sEMG signal, passes through convolution and an axial attention module, and outputs 256-dimensional features. The fusion layer concatenates the temporal and spectral features, performs layer normalization, Dense (fully connected), and Dropout fusion dimensionality reduction, and outputs to the Softmax classification layer. The algorithm consists of four stages. In the signal preprocessing stage, sEMG is processed by STFT to generate a spectrogram, and the finger bending signal is normalized and aligned, with the two paired to form a bimodal sample. In the dual-tower feature encoding stage, the temporal tower extracts multi-scale temporal features, and the spectrogram tower extracts spatial frequency features. In the feature fusion stage, cross-modal fusion is achieved through feature concatenation, the GELU activation function in a fully connected layer, and a dropout layer. In the evaluation and optimization stage, K-fold cross-validation and the AdamW optimizer are used, with accuracy, F1 score, MCC (Matthews correlation coefficient), and Kappa (Cohen coefficient) as indicators to ensure model performance. Accuracy is the most basic classification performance indicator, referring to the proportion of correctly identified samples out of the total number of samples. The F1 score is the harmonic mean of precision and recall, used to balance the false positive rate and false negative rate of the algorithm, and to address the bias that may be caused by relying solely on precision or recall. MCC is a comprehensive metric based on a confusion matrix (TP, TN, FP, FN), where TP represents true positives, TN represents true negatives, FP represents false positives, and FN represents false negatives. It evaluates algorithm performance by measuring the correlation between the actual and predicted classes. Kappa measures the consistency between the algorithm's predictions and the actual results, with a particular focus on eliminating the interference of random guessing on consistency.
[0032] Temporal features reflect the dynamic changes of hand movements over time, such as the fluctuations in the speed and amplitude of finger flexion and extension. Spectral features reflect the frequency-dimensional activation characteristics of hand muscles, such as the intensity and frequency distribution of electrical activity in different muscle groups during movement. By concatenating these features, two types of core information belonging to different dimensions can be integrated into a unified feature matrix, avoiding the problem of incomplete information from a single modality (only time or only frequency). For example, relying solely on temporal features cannot determine the muscle activation state behind the movement, and relying solely on spectral features cannot determine the specific movement amplitude corresponding to muscle activation. However, the concatenated features can simultaneously cover two key types of information: movement morphology and muscle control, providing a comprehensive basis for subsequent recognition.
[0033] Temporal features (128 dimensions) and spectral features (256 dimensions) are extracted from different pathways, and their data distributions (such as mean and variance) naturally differ (for example, the numerical range of temporal features may be concentrated between 0 and 1, while the spectral features may be concentrated between 10 and 100 due to sEMG signal amplification). LayerNorm (layer normalization) standardizes the concatenated feature matrix sample by sample and feature by feature, unifying the distribution of different modal features to a similar range, typically normalizing to a mean of 0 and a variance of 0. This avoids excessive bias in subsequent Dense layers towards one type of feature due to differences in feature numerical magnitudes, such as ignoring temporal features because of their large spectral feature values. This ensures fair weighting of the two types of features in subsequent processing and improves the effectiveness of fusion.
[0034] Fully connected layer processing: The concatenated feature dimension is 416, which can easily lead to increased computational load and redundant information interference. The Dense layer uses linear transformation to map the 416-dimensional high-dimensional features to low-dimensional features (such as mapping to 64 or 128 dimensions), achieving dimensionality reduction while preserving core information, reducing the computational burden of subsequent Softmax classification layers, and improving model running efficiency.
[0035] Dropout processing: During model training and inference, Dropout randomly shuts down some feature channels to prevent the model from over-relying on a few specific features, such as the spectral features of a certain muscle group or the temporal features of a certain movement phase. This prevents the model from overfitting due to noise in the training data or individual differences, such as differences in the intensity of electromyographic signals among different patients. In other words, the accuracy is high during training but drops sharply during actual monitoring, thus improving the robustness of the model to different patients and different monitoring scenarios.
[0036] After the above optimization, the features possess the characteristics of comprehensive information, uniform distribution, dimensional adaptability, and strong robustness. When input into the Softmax classification layer, the Softmax classification layer can calculate the probability distribution of different hand motor function categories based on these features, such as finger flexion and extension, grasping, fist clenching, or mild or moderate functional impairment. Finally, the category with the highest probability is output as the hand motor function recognition result. This result directly provides a quantitative basis for home rehabilitation monitoring and remote medical assessment for stroke and arthritis patients, such as judging the quality of a patient's completion of a certain rehabilitation movement and the current level of functional impairment. This is a key link in the system to achieve the core goal of accurate monitoring and rehabilitation assessment.
[0037] The data processing device integrates electromyography and kinematic information to achieve cross-modal feature complementarity enhancement, improving the model's robustness in recognizing hand movement intentions and reducing the impact of noise, individual differences, and muscle fatigue on assessment results. Employing a low-computational-complexity attention mechanism, axial attention, and stochastic depth techniques reduces computational burden, enabling the model to be deployed on edge devices for real-time rehabilitation assessment. It possesses signal length compatibility, with automatic pruning and padding mechanisms adapting to signals at different sampling rates, enhancing the model's applicability. Feature channels correspond to specific muscle groups and movement patterns, offering strong clinical interpretability and facilitating medical personnel's understanding of rehabilitation progress and the development of personalized rehabilitation plans.
[0038] The data processing device preprocesses the processed electromyography (EMG) signal by performing a Short-Time Fourier Transform (STFT) to obtain spectral features. It also preprocesses the processed finger flexion signal by performing normalization and time alignment to obtain temporal features. Subsequently, the extracted spectral and temporal features are concatenated along the feature dimension using a concatenation operation to achieve cross-modal feature fusion. The concatenated fused features are input into a fully connected (Dense) layer, where the linear combination and interaction relationships between different modal features are learned through a weight matrix. A non-linear mapping is introduced via the GELU activation function to enhance the fusion representation capability. Finally, after passing through a Dropout layer to prevent overfitting, the features are fed into a Softmax classification layer to output the hand movement function recognition result. The data processing device integrates EMG and kinematic information to achieve cross-modal feature complementarity enhancement, improving the robustness of the model in hand movement function recognition.
[0039] The above embodiments are described in detail using home-based hand rehabilitation monitoring for stroke patients as a specific application scenario. The implementation process is as follows: First, preparations were made, including a multimodal acquisition device comprising a wearable flexible glove, an electromyography (EMG) acquisition device, and a USB data cable. A dual-tower time-frequency fusion algorithm was deployed on the computer terminal. Three-channel EMG and five-channel flexion acquisition were selected, based on the highest correlation with the fingers. The elastic band of the flexible glove was adjusted according to the patient's hand shape, and 12 EMG electrode patches were fixed to specific locations on the patient's forearm flexor muscles, extensor muscles, and interosseous muscles of the hand, ensuring that the electrode patches adhered to the skin and preventing displacement that could affect signal acquisition.
[0040] The monitoring process then begins. The patient wears a wearable flexible glove, and the electromyography (EMG) acquisition device is connected to a computer terminal via USB. The acquisition software and algorithm program are then started. Following instructions, the patient performs rehabilitation movements such as finger flexion and extension, grasping, and fist clenching, each lasting 5 seconds and repeated 10 times. During this process, the flexion sensor embedded in the glove captures changes in finger flexion angle in real time, converting them into electrical signals that are transmitted to the EMG acquisition device via cable. EMG electrode patches acquire sEMG signals from the muscles of the hand and forearm, which are also transmitted to the EMG acquisition device. The EMG acquisition unit of the EMG acquisition device initially amplifies the sEMG signals, and the signal processing unit further amplifies and filters the signals to remove noise interference. An analog-to-digital converter converts the analog signals into digital signals, while the flexion electrical signals undergo resistive voltage division and operational amplification. The processed signals are displayed in real time on an OLED screen, allowing the patient to visually understand the muscle activation state, and are also transmitted to the computer terminal via a serial port.
[0041] The computer terminal runs the algorithm preprocessing module, performing STFT processing on the sEMG digital signal. The preprocessed signal is then used as input to the spectral feature path. The curvature digital signal is Z-score normalized to eliminate individual differences, and its length is aligned to 2048 points using an automatic cropping and padding mechanism, serving as input to the temporal feature path. The dual-tower algorithm is then activated. The temporal feature path extracts 128-dimensional temporal features from the curvature time series using a Dense layer, dilated convolutional stack, and Transformer encoder. The spectral feature path extracts 256-dimensional spectral features from the sEMG spectrogram using a convolutional layer and axial attention module. The fusion layer concatenates the temporal and spectral features, processes them using LayerNorm, Dense, and Dropout, and outputs the result to the Softmax classification layer, yielding hand movement function recognition results, such as "finger flexion and extension movement recognition accuracy 85%, corresponding to mild functional impairment."
[0042] In this embodiment, through five-fold cross-validation, the average accuracy of hand movement recognition is over 85%, and the F1 score, MCC, and Kappa coefficient are higher than 0.83, 0.80, and 0.82, respectively, indicating stable evaluation results. The device is easy to wear and operate, effectively reducing the cost of patients traveling to and from the hospital and improving the convenience and efficiency of rehabilitation training.
[0043] When the system works in collaboration, the multimodal acquisition device collects the patient's sEMG and finger flexion data in real time. After processing, the data is transmitted to the terminal device. The terminal device runs the algorithm preprocessing module to generate paired bimodal samples. The dual-tower algorithm extracts and fuses the features of the samples and finally outputs the hand motor function recognition results, providing a basis for rehabilitation assessment.
[0044] The first embodiment of this invention provides a hand injury monitoring system that simultaneously acquires finger flexion signals and electromyographic (EMG) signals through a multimodal acquisition device. This reflects both hand movement morphology and muscle activation status. Compared to single-dimensional monitoring devices, it can more accurately determine the rehabilitation stage of hand injuries, providing more comprehensive data support for adjusting rehabilitation plans. The data processing device integrates EMG and kinematic information to achieve cross-modal feature complementarity enhancement, improving the model's robustness in recognizing hand movement intentions and reducing the impact of noise, individual differences, and muscle fatigue on the evaluation results.
[0045] like Figure 5 As shown, the second embodiment of the present invention provides a hand injury monitoring method applicable to the data processing device in the hand injury monitoring system described in the first embodiment above. The method includes: The data processing device acquires the processed multimodal data transmitted by the multimodal acquisition device; The data processing device is used to receive the processed multimodal data, preprocess the processed electromyographic signal to obtain spectral features, preprocess the processed finger flexion signal to obtain temporal features, fuse the spectral features and temporal features across modalities, and output the hand movement function recognition result.
[0046] Specific methods for preprocessing the processed electromyography signal to obtain spectral features and preprocessing the processed finger flexion signal to obtain temporal features include: The processed electromyographic signal is subjected to STFT time-frequency transformation to obtain a spectrogram, which is then input into the spectral feature path. The spectral feature extraction pathway extracts spectral features from the spectrogram. The processed curvature signal is normalized and time-aligned to obtain curvature time-series data, which is then input into the time-series feature path. The temporal feature pathway extracts temporal features from curvature temporal data; The specific method for cross-modal fusion of spectral features and temporal features includes: The spectral features and temporal features are concatenated to obtain a unified feature matrix. The concatenated feature matrix is then standardized. The standardized data is then subjected to dimensionality reduction and anti-overfitting processing to obtain optimized features. The optimized features are then input into the classification layer to calculate the probability distribution of different hand movement function categories. The category with the highest probability is taken as the hand movement function recognition result.
[0047] Following the step of cross-modal fusion of spectral and temporal features, the method further includes optimizing model parameters using a K-fold cross-validation and optimizer to ensure model performance.
[0048] The hand injury monitoring method provided in this embodiment of the invention is based on the same inventive concept and has the same beneficial effects as the hand injury monitoring system provided in the first embodiment above, and will not be described again here.
[0049] Another embodiment of the present invention provides an electronic device, which includes a processor, an input device, an output device, and a memory. The processor, the input device, the output device, and the memory are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to call the program instructions to execute the method described in the second embodiment above.
[0050] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0051] Input devices may include touchpads, microphones, etc., and output devices may include displays (LCDs, etc.), speakers, etc.
[0052] The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information about the device type.
[0053] In specific implementations, the processor, input device, and output device described in the embodiments of the present invention can execute the implementation methods described in the method embodiments of the present invention, or they can execute the implementation methods described in the system embodiments of the present invention, which will not be repeated here.
[0054] The present invention also provides an embodiment of a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the methods described in the above embodiments.
[0055] The computer-readable storage medium can be an internal storage unit of the terminal described in the foregoing embodiments, such as the terminal's hard drive or memory. The computer-readable storage medium can also be an external storage device of the terminal, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the terminal. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.
[0056] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0057] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the terminals and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0058] In the several embodiments provided in this application, it should be understood that the disclosed terminals and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.
[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.
Claims
1. A hand injury monitoring system, characterized in that, include: A multimodal acquisition device and a data processing device are provided. The multimodal acquisition device is used to acquire multimodal data, perform preliminary processing to obtain processed multimodal data, and transmit the processed multimodal data to the data processing device for further processing. The multimodal acquisition device includes a finger data acquisition device and an electromyography (EMG) acquisition device, and the EMG acquisition device includes a signal processing unit; The finger data acquisition device is used to collect finger bending signals generated by various hand movements of the user; The electromyography (EMG) acquisition device uses EMG electrodes to collect EMG signals generated by the muscles of the hand and forearm when the user makes hand movements. The signal processing unit amplifies, filters, and performs analog-to-digital conversion on the acquired electromyographic signals to obtain processed electromyographic signals; the finger flexion signal is acquired by a resistor voltage divider and input to the data processing device for normalization and time alignment preprocessing to obtain processed finger flexion signal. The data processing device is used to receive the processed data, preprocess the processed electromyographic signal to obtain spectral features, preprocess the processed finger flexion signal to obtain temporal features, perform cross-modal fusion of spectral features and temporal features, and output the hand movement function recognition result.
2. The hand injury monitoring system as described in claim 1, characterized in that, The data processing device includes a first preprocessing module, a second preprocessing module, a spectral feature path module, and a time-series feature path module; The first preprocessing module is used to perform STFT time-frequency transformation on the processed electromyographic signal to obtain a spectrum, and input the spectrum into the spectrum feature path module; The second preprocessing module is used to normalize and time-align the processed curvature signal to obtain curvature time-series data, and input the time-series data into the time-series feature path module; The spectral feature path module is used to extract spectral features from the spectrogram. The temporal feature path module is used to extract temporal features from curvature temporal data.
3. The hand injury monitoring system as described in claim 2, characterized in that, The data processing device further includes a feature fusion and layering module, which is used to concatenate spectral features and temporal features to obtain a unified feature matrix, perform standardization processing on the concatenated feature matrix, perform dimensionality reduction and anti-overfit processing on the standardized data to obtain optimized features, input the optimized features into the classification layer to calculate the probability distribution of different hand movement function categories, and take the category with the highest probability as the hand movement function recognition result.
4. The hand injury monitoring system as described in claim 3, characterized in that, The data processing device also includes a model optimization and evaluation module, which uses K-fold cross-validation and optimizer to optimize model parameters and ensure model performance.
5. The hand injury monitoring system as described in claim 2, characterized in that, The temporal feature path module is used to process the curvature temporal data sequentially through a fully connected layer, a dilated convolution stack, and a Transformer encoder, and output 128-dimensional temporal features. The spectral feature path module is used to process the input spectrogram sequentially through a convolutional layer and an axial attention module to extract 256-dimensional spectral features.
6. A method for monitoring hand injuries, characterized in that, The data processing apparatus applicable to the hand injury monitoring system as described in any one of claims 1-5, the method comprising: The data processing device acquires the processed multimodal data transmitted by the multimodal acquisition device; The data processing device is used to receive the processed multimodal data, preprocess the processed electromyographic signal to obtain spectral features, preprocess the processed finger flexion signal to obtain temporal features, fuse the spectral features and temporal features across modalities, and output the hand movement function recognition result.
7. The hand injury monitoring method as described in claim 6, characterized in that, The specific methods for preprocessing the processed electromyography signal to obtain spectral features and preprocessing the processed finger flexion signal to obtain temporal features include: The processed electromyographic signal is subjected to STFT time-frequency transformation to obtain a spectrogram, which is then input into the spectral feature path. The spectral feature extraction pathway extracts spectral features from the spectrogram. The processed curvature signal is normalized and time-aligned to obtain curvature time-series data, which is then input into the time-series feature path. The temporal feature pathway extracts temporal features from curvature temporal data; The specific method for cross-modal fusion of spectral features and temporal features includes: The spectral features and temporal features are concatenated to obtain a unified feature matrix. The concatenated feature matrix is then standardized. The standardized data is then subjected to dimensionality reduction and anti-overfitting processing to obtain optimized features. The optimized features are then input into the classification layer to calculate the probability distribution of different hand movement function categories. The category with the highest probability is taken as the hand movement function recognition result.
8. The hand injury monitoring method as described in claim 7, characterized in that, Following the step of cross-modal fusion of spectral and temporal features, the method further includes optimizing model parameters using a K-fold cross-validation and optimizer to ensure model performance.
9. An electronic device, comprising: The system includes a processor, an input device, an output device, and a memory, which are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to invoke the program instructions to execute the method as described in any one of claims 6-8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 6-8.