A gait rehabilitation evaluation method and system based on multi-modal data space-time features, a terminal and a storage medium

Abstract

The application relates to the technical field of data processing, and discloses a gait rehabilitation evaluation method and system based on multi-modal data space-time characteristics, a terminal and a storage medium.The method comprises the following steps: in a certain period interval, the electromyography, electroencephalogram and exoskeleton robot IMU signals of a motor disorder population are synchronously collected under a certain movement paradigm, and the movement function score results of the patients are synchronously recorded by doctors according to a scale; data preprocessing, channel selection and feature extraction are performed on the electromyography, electroencephalogram and exoskeleton robot IMU signals; a model is established based on a neural network, movement evaluation is performed according to a fused feature array, movement function quantitative scoring is obtained, and the movement function state of the evaluated person is classified according to the score.The multi-modal synchronous data rehabilitation evaluation model of the movement function of the motor disorder population is constructed according to multiple information mining, and the movement function state of the evaluated person is quickly evaluated, predicted and classified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a gait rehabilitation evaluation method and system based on multi-modal data spatiotemporal features, a terminal and a computer readable storage medium. BACKGROUND

[0002] Human movement function evaluation is an important topic in the field of rehabilitation medicine and sports science. Currently, movement function evaluation mainly has the following methods:

[0003] (1) Clinical evaluation: Clinical evaluation is a commonly used method of movement function evaluation, which includes observation of the patient's movement function, muscle strength, coordination, balance ability, etc. However, clinical evaluation mainly depends on the experience and observation of doctors, and may be affected by subjective factors, and it is difficult to quantitatively evaluate the movement function of patients.

[0004] (2) Neurophysiological evaluation: Neurophysiological evaluation includes electromyography and nerve conduction testing, which can evaluate the function of muscles and nerves. This evaluation method can provide objective data, but may have certain requirements for patient comfort and safety.

[0005] (3) Movement function evaluation scale: Movement function evaluation scale is a tool for quantitatively evaluating the movement function of patients with movement disorders, including functional independence measurement, Fugl-Meyer movement evaluation (Fugl-Meyer movement evaluation is a standardized and quantitative evaluation scale specifically for evaluating the recovery of motor function of patients with hemiplegia after stroke). These scales can provide detailed movement function scores, but may have difficulty in comprehensively evaluating all aspects of the patient's movement function.

[0006] (4) Robot-assisted evaluation: Robot-assisted evaluation can provide precise measurement of movement trajectory and muscle activity, but may be costly and require high skill from the patient and operator.

[0007] (5) Self-evaluation: Self-evaluation is a patient self-reporting evaluation method that can understand the patient's perception and feelings of their own movement function. However, self-evaluation may be affected by the patient's level of consciousness, language ability, and emotional state.

[0008] Among the above existing evaluation methods, clinical evaluation, scale evaluation and self-evaluation all belong to qualitative evaluation, which are easily affected by the subjective state of the doctor or patient, resulting in inaccurate evaluation. Although neurophysiological evaluation and robot-assisted evaluation are quantitative evaluations, they have the defects of single modality, single dimension and lack of neural pathway information.

[0009] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0010] The main purpose of the present application is to provide a gait rehabilitation evaluation method and system based on multi-modal data spatiotemporal features, a terminal and a computer readable storage medium, aiming to solve the problem of single evaluation dimension, incomplete information coverage, lack of deep mining and effective utilization of signal information in the prior art, resulting in low evaluation accuracy.

[0011] To achieve the above-mentioned purpose, the present application provides a gait rehabilitation evaluation method based on multi-modal data spatiotemporal features, which comprises the following steps:

[0012] Synchronously collecting electromyographic signals, electroencephalographic signals and exoskeleton robot IMU signals of the movement disorder population under a preset movement paradigm, and synchronously recording the movement function score results of the movement disorder population;

[0013] Data preprocessing and feature extraction are performed on the electromyographic signals and the electroencephalographic signals to obtain first features, and data preprocessing and feature extraction are performed on the exoskeleton robot IMU signals to obtain second features;

[0014] The first features and the second features are subjected to feature fusion processing to obtain fusion features, and a time series evaluation network model is trained according to the fusion features and the movement function score results to obtain a target time series evaluation network model;

[0015] The processed brain-muscle coupling signal features and IMU signal gait features of the evaluated person are input into the target time series evaluation network model, the movement function quantitative score of the evaluated person is output, and the movement function state of the evaluated person is graded according to the movement function quantitative score to obtain a grading result.

[0016] In addition, to achieve the above-mentioned purpose, the present application also provides a gait rehabilitation evaluation system based on multi-modal data spatiotemporal features, wherein the gait rehabilitation evaluation system based on multi-modal data spatiotemporal features comprises:

[0017] A data acquisition and recording module is used for synchronously collecting electromyographic signals, electroencephalographic signals and exoskeleton robot IMU signals of the movement disorder population under a preset movement paradigm, and synchronously recording the movement function score results of the movement disorder population;

[0018] A data preprocessing and feature extraction module is used for data preprocessing and feature extraction of the electromyographic signals and the electroencephalographic signals to obtain first features, and data preprocessing and feature extraction of the exoskeleton robot IMU signals to obtain second features;

[0019] The data fusion and model training module is configured to perform feature fusion processing on the first feature and the second feature to obtain a fusion feature, train a time sequence evaluation network model according to the fusion feature and the motor function score result, and obtain a target time sequence evaluation network model.

[0020] The evaluation prediction and grading module is configured to input the processed brain-muscle coupling signal feature and the IMU signal gait feature of the evaluated person into the target time sequence evaluation network model, output a motor function quantitative score of the evaluated person, grade the motor function state of the evaluated person according to the motor function quantitative score, and obtain a grading result.

[0021] In addition, to achieve the above object, the present application also provides a terminal, wherein the terminal comprises a memory, a processor, and a gait rehabilitation evaluation program based on multi-modal data spatiotemporal features stored on the memory and executable on the processor, and the gait rehabilitation evaluation program based on multi-modal data spatiotemporal features implements the steps of the gait rehabilitation evaluation method based on multi-modal data spatiotemporal features when executed by the processor.

[0022] In addition, to achieve the above object, the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a gait rehabilitation evaluation program based on multi-modal data spatiotemporal features, and the gait rehabilitation evaluation program based on multi-modal data spatiotemporal features implements the steps of the gait rehabilitation evaluation method based on multi-modal data spatiotemporal features when executed by a processor.

[0023] In the present application, the electromyographic signals, the electroencephalographic signals and the IMU signals of the exoskeleton robot of the motor disorder population are synchronously collected under a preset motor paradigm, and the motor function score results of the motor disorder population are synchronously recorded; the electromyographic signals and the electroencephalographic signals are preprocessed and feature-extracted to obtain first features, and the IMU signals of the exoskeleton robot are preprocessed and feature-extracted to obtain second features; the first features and the second features are subjected to feature fusion processing to obtain fusion features, a time sequence evaluation network model is trained according to the fusion features and the motor function score results, and a target time sequence evaluation network model is obtained; the processed brain-muscle coupling signal features and the IMU signal gait features of the evaluated person are input into the target time sequence evaluation network model, a motor function quantitative score of the evaluated person is output, the motor function state of the evaluated person is graded according to the motor function quantitative score, and a grading result is obtained. The present application establishes the correlation between the scale score and the brain-muscle coupling result and the IMU signal of the exoskeleton robot, constructs a multi-modal synchronous data rehabilitation evaluation model of the motor function of the motor disorder population according to multi-information mining, realizes the rapid evaluation prediction and grading of the motor function state of the evaluated person, and provides a new method for clinical rehabilitation evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a flow chart of a preferred embodiment of the gait rehabilitation evaluation method based on spatiotemporal features of multi-modal data of the present application;

[0025] Figure 2 is a schematic diagram of the motor function rehabilitation evaluation process in a preferred embodiment of the gait rehabilitation evaluation method based on spatiotemporal features of multi-modal data of the present application;

[0026] Figure 3 is a structure diagram of a preferred embodiment of the gait rehabilitation evaluation system based on spatiotemporal features of multi-modal data of the present application;

[0027] Figure 4 is a structure diagram of a preferred embodiment of the terminal of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and advantages of the present application more clear and explicit, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0029] In the present application, the electromyography, electroencephalogram and exoskeleton robot IMU signals of the movement disorder population are synchronously collected in a certain movement paradigm at a certain cycle interval (such as every week), and the movement function score results (movement function score) of the patients are synchronously recorded by the doctors according to the scale. The electromyography, electroencephalogram and exoskeleton robot IMU signals are subjected to data preprocessing, channel selection, feature extraction, and then a model is established based on a neural network to evaluate the movement according to the fused feature array.

[0030] The gait rehabilitation evaluation method based on spatiotemporal features of multi-modal data of the preferred embodiment of the present application, as shown in Figure 1 and Figure 2 The gait rehabilitation evaluation method based on spatiotemporal features of multi-modal data includes the following steps:

[0031] Step S10, synchronously collect electromyography signals, electroencephalogram signals and exoskeleton robot IMU signals of the movement disorder population under a preset movement paradigm, and synchronously record the movement function score results of the movement disorder population.

[0032] Specifically, the experimental paradigm is designed: mainly the lower limb movement paradigm, the electromyography, electroencephalogram and exoskeleton robot IMU (Inertial Measurement Unit, inertial measurement unit) signal collection site and time length, clinical evaluation cycle and experimental collection cycle. After the patient wears the exoskeleton walking aid robot and the brain electromyography synchronous collection equipment, certain adaptive walking training is carried out, and then ten-meter walking test is carried out.

[0033] The experiment paradigm is performed in cycles, the scores of the patients by the physician according to the FMA (Fugl-Meyer, motor function assessment) scale are recorded, and the electromyographic signals, electroencephalographic signals and exoskeleton robot IMU signals (i.e. IMU signals in Figure 2 The electroencephalographic signals are commonly sampled at a frequency of 250 Hz, 500 Hz or 1000 Hz, the electromyographic signals are commonly sampled at a frequency of 500 Hz or 1000 Hz, and the exoskeleton robot IMU is commonly sampled at a frequency of about 100 Hz.

[0034] Step S20, data preprocessing and feature extraction are performed on the electromyographic signals and the electroencephalographic signals to obtain first features, and data preprocessing and feature extraction are performed on the exoskeleton robot IMU signals to obtain second features.

[0035] Specifically, the first features include root mean square, mean absolute value, median frequency and average power frequency; the second features include spatio-temporal features and gait symmetry features; the spatio-temporal features include step frequency and step length; and the gait symmetry features include left-right gait cycle symmetry index and gait similarity of left and right legs. The electromyographic signals and the electroencephalographic signals are filtered, artifact removed and data segmented, and the root mean square (RMS), mean absolute value (MAV), median frequency (MF) and mean power frequency (MPF) of the electromyographic signals and the electroencephalographic signals are extracted; the exoskeleton robot IMU signals are filtered to eliminate noise, and the original nine-axis data (including three-axis angular velocity, three-axis acceleration and three-axis magnetic field) are converted into joint angles using an extended Kalman filter algorithm; and the spatio-temporal features and gait symmetry features of the exoskeleton robot IMU signals after data preprocessing are extracted.

[0036] The original nine-axis data (including three-axis angular velocity, three-axis acceleration and three-axis magnetic field) are converted into joint angles using an extended Kalman filter algorithm:

[0037] Prediction process:

[0038] ;

[0039] Correction process:

[0040] ;

[0041] wherein, is a state transition function f The state xthe Jacobian matrix of the state k is the joint angle prior estimate state at time k and control input f is obtained by the state transition function k is the covariance matrix of the state prior estimate at time k is the covariance matrix of the state posterior estimate at time T denotes the transpose; Q is the process noise covariance matrix; is the Jacobian matrix of the observation function h ( x ) with respect to the state x K is the Kalman gain; R is the observation noise covariance matrix; k is the joint angle posterior estimate of the state at time k is the observation value at time is the predicted observation value obtained by the observation function h ( x ) with respect to the state k is the covariance matrix of the state posterior estimate at time I is the identity matrix.

[0042] Gait features: A series of limb movement features can be obtained by preprocessing the IMU signals of the exoskeleton robot. Gait is a typical manifestation of the periodic movement of the lower limbs of the human body. Based on the three-axis acceleration and three-axis angular velocity of the lower limbs collected by the IMU, core features reflecting the integrity, stability and symmetry of the gait can be extracted.

[0043] Gait cycle and spatiotemporal features: By cutting the hip joint angle, individual gait cycles can be extracted. Specifically, the trough of the hip joint is taken as the starting point of the gait cycle, and the next trough is taken as the end point of a complete gait cycle. The gait cycle includes the support phase (i.e. from trough to peak) and the swing phase (from peak to trough), and the spatiotemporal features include step frequency and step length.

[0044] The calculation of the step frequency (i.e. the number of steps per unit time) is as follows:

[0045] ;

[0046] wherein, C is the step frequency, is the total number of steps,​​​​​​​​​​ is the total measurement time.

[0047] The calculation of the step length (the horizontal distance from the "heel strike" of one lower limb to the "heel strike" of the opposite lower limb) is:

[0048] ;

[0049] wherein, SL is the step length, is the walking speed, measured by the sensor at the pelvis, is the gait cycle length.

[0050] Gait symmetry: refers to the consistency of the left and right lower limbs in terms of gait cycle, exercise intensity, etc., which is a core indicator for evaluating unilateral motor dysfunction (such as hemiplegia after stroke, lower limb fracture after surgery), and poor symmetry usually indicates insufficient muscle strength or abnormal neural control of the affected side.

[0051] Left-right gait cycle symmetry index: reflects the difference in gait cycle length between the left and right lower limbs, and the symmetry index of the normal population is close to 0. The proportion of the support phase on the affected side (e.g. the hemiplegic side of stroke patients) is usually longer than that on the healthy side, and the absolute value of the index increases. The calculation of the left-right gait cycle symmetry index is:

[0052] ;

[0053] wherein, SI is the left-right gait cycle symmetry index, is the left leg swing period time, is the left leg gait cycle time, is the right leg swing period time, is the right leg gait cycle time.

[0054] Dynamic Time Warping (DTW): is an algorithm for measuring the similarity of time series, which constructs a distance matrix between two curves and finds the optimal path with the smallest cumulative distance. By performing nonlinear time alignment on the joint angle time series of the left and right legs, the gait similarity of the left and right legs can be analyzed. The calculation of the gait similarity of the left and right legs is:

[0055] ;

[0056] wherein, is the Euclidean distance between two time series X i and Y j to be compared, X iThe first time series of the left leg joint angle i One sampling point, Y j The first time series of the right leg joint angle j One sampling point, The smaller the value, the more it indicates and The higher the similarity, n Time series X Length, m Time series Y Length, Dist nm For the left leg n The sampling point and the right leg m Euclidean distance between sampling points D The final distance matrix is ​​obtained. DTW(X,Y) Representing time series X and Y Dynamic time-planning distance, This is the set of minimum cumulative distances for each cell calculated during the dynamic programming process. For the best alignment path p Upper The cumulative distance corresponding to each grid For the best alignment path p The cumulative distance (i.e., the total number of grids contained in the path).

[0057] In the calculated DTW distance matrix, the more light-colored bands there are, the more abrupt the optimal path changes, indicating that the patient's gait symmetry is lower.

[0058] Step S30: Perform feature fusion processing on the first feature and the second feature to obtain fused features, and train the temporal evaluation network model based on the fused features and the motion function scoring results to obtain the target temporal evaluation network model.

[0059] Specifically, brain-muscle coupling analysis based on transfer spectrum entropy was performed on preprocessed synchronized EEG and EMG signals to explore the synergistic relationship between cerebral cortex activity and peripheral skeletal muscle activity in the time or frequency domain.

[0060] First, based on the movement, the EEG and EMG signals are divided into lengths... L Non-overlapping time windows and The method involves converting each time window into a symbol sequence using a quantile-based binning approach. Specifically, each segment... or The amplitude is mapped to a finite symbolic alphabet. The mapping rule is based on the quantile division of the electroencephalogram (EEG) amplitude, where, express EEG amplitude at any given time express Electromyographic amplitude at time t, Indicates the first The and the first The dividing points of each amplitude interval, X and Y These represent discrete time series of electroencephalogram (EEG) and electromyogram (EMG), respectively. This indicates the size of the symbol set (i.e., the number of intervals divided into).

[0061] calculate X Symbolization results , , It is a symbolic alphabet:

[0062] ;

[0063] calculate Y Symbolization results :

[0064] ;

[0065] in, This represents the symbolic result of each EEG amplitude point. This represents the symbolic result for each electromyographic amplitude point.

[0066] This transformation preserves the signal’s local nonlinear dynamic structure while reducing its sensitivity to amplitude variations and noise, enabling robust symbol modeling in subsequent frequency domain analysis.

[0067] After symbolization, a Fast Fourier Transform is performed on each symbol sequence to extract frequency components. and They represent X and Y The Fourier spectrum calculation result for the time-varying symbol sequence is as follows: and Subsequently, the amplitude spectrum is extracted, and a second sign transformation is performed in the frequency domain. Specifically, each frequency point... Amplitude data for all time periods are categorized by Each level is categorized, ultimately forming a frequency domain symbol sequence. and .

[0068] Transfer entropy is used to quantify the directional flow of information between two time series. In general, given...Y The transfer entropy measures X the predictive gain of Y future states.

[0069] The transfer entropy is defined as

[0070] .

[0071] where is the predictive gain of X future states, Y and and denote the past time period vectors of X and Y , respectively, denotes the EMG amplitude at time t .

[0072] For each , the probability of Y occurring depends on its own preceding symbol as well as the preceding symbol in X , therefore, the transfer spectral entropy from to X at Y is defined as

[0073] .

[0074] where is the empirical probability estimated from all epoch symbol sequences; this formulation reflects the directional spectral coupling from X to Y , and by computing over all frequencies, a frequency-resolved profile of the directional connections is obtained.

[0075] Similarly, the transfer spectral entropy from Y to X at is defined as

[0076] .

[0077] After obtaining the results of brain-muscle coupling analysis, the mean coupling values of the three frequency bands, alpha (8-13 Hz), beta (13-30 Hz), and gamma (30-45 Hz), are extracted as evaluation features. For the mean coupling value of the frequency band , it is defined as

[0078] ​​ ;

[0079] wherein, represents the minimum frequency, represents the maximum frequency, N represents the number of frequencies covered by the frequency band interval.

[0080] Based on the brain-muscle coupling, gait features of the exoskeleton robot IMU signal, and the motor function score results, a time sequence evaluation network model is trained to have the ability to capture spatio-temporal features to evaluate motor function, and a target time sequence evaluation network model (i.e., a trained time sequence evaluation network model) is obtained. The time sequence evaluation network model is a hybrid neural network model combining a double-channel input convolutional neural network (CNN) and a long short-term memory network (LSTM). The training process is as follows:

[0081] The training data set is input into the fusion neural network model, the mean squared error (MSE) is used as the loss function, the Adam optimizer is used for model parameter optimization, the learning rate is set to 0.001, the iteration number is 100, the batch size is 32, and the model parameters are continuously adjusted through back propagation to minimize the loss function value of the model. Every 10 iterations, the model is tested using the test data set, the mean absolute error (MAE) and the determination coefficient (R²) between the predicted score of the model and the sample labeled score are calculated, when the MAE is less than 5 and the R² is greater than 0.9, the model training is stopped, and the trained fusion neural network model is obtained.

[0082] Network architecture: one channel is used to input the brain-muscle coupling signal features, the CNN layer of this channel is used to extract the local spatial features of the brain-muscle coupling signal features, and the LSTM layer is used to extract the time sequence features of the brain-muscle coupling signal features; another channel is used to input the IMU signal features, the CNN layer of this channel is used to extract the local spatial features of the IMU signal features, and the LSTM layer is used to extract the time sequence features of the IMU signal features; then the features extracted by the two channels are fused through the fully connected layer, and finally the motor function evaluation results are output through the output layer, the evaluation results are quantified by scores of 0-100, and the higher the score, the better the motor function.

[0083] In step S40, the processed brain-muscle coupling signal features and IMU signal gait features of the evaluated person are input into the target timing evaluation network model, and a motor function quantitative score of the evaluated person is output, and the motor function state of the evaluated person is graded according to the motor function quantitative score to obtain a grading result.

[0084] Specifically, the grading result includes normal motor function, mild motor dysfunction, moderate motor dysfunction, and severe motor dysfunction. The processed brain-muscle coupling signal features and IMU signal gait features of the evaluated person are input into the target timing evaluation network model (i.e., the trained motor evaluation model, i.e., the motor evaluation model in Figure 2

[0085] The present application uses the simultaneously collected EEG, EMG, and exoskeleton robot IMU three-modal signals for comprehensive evaluation, performs brain-muscle coupling analysis and motion feature extraction on the signals selected through key channels, realizes the collaborative extraction and fusion of information from the whole chain of “neural control-muscle activity-limb movement”, and breaks through the limitation of incomplete coverage of single modal information. The present application synchronously collects the EEG, EMG, and exoskeleton robot IMU three-modal signals of the movement disorder population under a certain periodic interval and a certain motion paradigm, and synchronously records the results of the scoring by the doctors according to the scale. The brain-muscle coupling analysis based on the transfer spectrum entropy is performed on the preprocessed EMG signals and EEG signals, and then the timing features are extracted. The joint angle calculation is performed on the exoskeleton robot IMU signals to obtain the spatial and temporal features of limb movement (such as joint position and motion speed change). The present application proposes to establish the correlation between the traditional internationally recognized scale score and the brain-muscle coupling result and the exoskeleton robot IMU signal, and to construct a multi-modal synchronous data rehabilitation evaluation model of the motor function of the movement disorder population according to multiple information mining, thereby providing a new method for clinical rehabilitation evaluation.

[0086] ​The application is directed to three modal signal characteristics, and a customized preprocessing procedure is designed to remove artifacts by band-pass filtering and fuse data. The application uses the main algorithm of brain-muscle coupling and gait analysis to model the changes in the motor function of the movement disorder population, establish a rehabilitation evaluation model, correlate the multi-modal signal characteristics with the rehabilitation process, and provide accurate basis for clinical intervention. The application uses the collaborative extraction of deep learning models, uses CNN and LSTM networks to capture the time series dynamic correlation of the input signal sequence (such as the time sequence synchronization of brain neural activity and muscle contraction, the delay relationship between muscle activity and limb movement), and obtains a time series rehabilitation evaluation model through parallel structure fast training, realizing the mapping from the input signal sequence to the output score sequence. The application proposes a method for modeling the relationship between the EEG, EMG and exoskeleton robot IMU signals of the movement disorder population and their human motor function in time sequence, which is helpful for quickly evaluating the long-term evolution of the human motor function in time sequence.

[0087] Further, as shown in Figure 3 based on the above-mentioned gait rehabilitation evaluation method based on multi-modal data spatiotemporal characteristics, the application also correspondingly provides a gait rehabilitation evaluation system based on multi-modal data spatiotemporal characteristics, wherein the gait rehabilitation evaluation system based on multi-modal data spatiotemporal characteristics comprises:

[0088] A data acquisition and recording module 51 is configured to synchronously acquire the EMG signal, EEG signal and exoskeleton robot IMU signal of the movement disorder population under a preset movement paradigm, and synchronously record the motor function score result of the movement disorder population;

[0089] A data preprocessing and feature extraction module 52 is configured to perform data preprocessing and feature extraction on the EMG signal and the EEG signal to obtain first features, and perform data preprocessing and feature extraction on the exoskeleton robot IMU signal to obtain second features;

[0090] A data fusion and model training module 53 is configured to perform feature fusion processing on the first features and the second features to obtain fusion features, train a time series evaluation network model according to the fusion features and the motor function score result, and obtain a target time series evaluation network model;

[0091] An evaluation prediction and grading module 54 is configured to input the processed brain-muscle coupling signal features and IMU signal gait features of the evaluated person into the target time series evaluation network model, output the motor function quantitative score of the evaluated person, grade the motor function state of the evaluated person according to the motor function quantitative score, and obtain a grading result.

[0092] Further, as shown in Figure 4As shown, based on the above gait rehabilitation evaluation method and system based on spatiotemporal features of multi-modal data, the application also correspondingly provides a terminal, which comprises a processor 10, a memory 20 and a display 30. Figure 4 Only part of the components of the terminal are shown, but it should be understood that all the shown components are not required to be implemented, and more or less components can be alternatively implemented.

[0093] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as a hard disk or a memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Further, the memory 20 can include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software and various data installed on the terminal, such as program codes of the terminal, etc. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores a gait rehabilitation evaluation program 40 based on spatiotemporal features of multi-modal data, which can be executed by the processor 10, so as to implement the gait rehabilitation evaluation method based on spatiotemporal features of multi-modal data in the application.

[0094] The processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, which is used to run program codes or process data stored in the memory 20, such as to execute the gait rehabilitation evaluation method based on spatiotemporal features of multi-modal data, etc.

[0095] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 30 is used to display information of the terminal and to display a visualized user interface. The processor 10, the memory 20 and the display 30 of the terminal communicate with each other through a system bus.

[0096] In an embodiment, when the processor 10 executes the gait rehabilitation evaluation program 40 based on spatiotemporal features of multi-modal data in the memory 20, the steps of the gait rehabilitation evaluation method based on spatiotemporal features of multi-modal data as described above are implemented.

[0097] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a gait rehabilitation evaluation program based on multi-modal data spatiotemporal features, and the gait rehabilitation evaluation program based on multi-modal data spatiotemporal features, when executed by a processor, implements the steps of the gait rehabilitation evaluation method based on multi-modal data spatiotemporal features.

[0098] In summary, the application provides a gait rehabilitation evaluation method, system, terminal and computer readable storage medium based on multi-modal data spatiotemporal features. The method comprises: synchronously collecting electromyography signals, electroencephalography signals and exoskeleton robot IMU signals of a movement disorder population under a preset movement paradigm, and synchronously recording movement function score results of the movement disorder population; performing data preprocessing and feature extraction on the electromyography signals and the electroencephalography signals to obtain first features, and performing data preprocessing and feature extraction on the exoskeleton robot IMU signals to obtain second features; performing feature fusion processing on the first features and the second features to obtain fused features, training a time sequence evaluation network model according to the fused features and the movement function score results to obtain a target time sequence evaluation network model; inputting processed brain-muscle coupling signal features and IMU signal gait features of an evaluated person into the target time sequence evaluation network model, outputting a movement function quantitative score of the evaluated person, and grading a movement function state of the evaluated person according to the movement function quantitative score to obtain a grading result. The application establishes a correlation between scale scores and brain-muscle coupling results and exoskeleton robot IMU signals, constructs a multi-modal synchronous data rehabilitation evaluation model of movement function of a movement disorder population according to multi-information mining, realizes rapid evaluation and grading of a movement function state of an evaluated person, and provides a new method for clinical rehabilitation evaluation.

[0099] It should be noted that, in this document, the terms "comprising" and "including" or any other variant thereof are intended to cover a non-exclusive inclusion, such that processes, methods, articles, or terminals including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include inherent elements of such processes, methods, articles, or terminals. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or terminal including the element.

[0100] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware (such as a processor, a controller, etc.) to complete, and the program can be stored in a computer readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The computer readable storage medium can be a memory, a disk, an optical disk, etc.

[0101] It is to be understood that the application is not limited to the examples described above, which can be modified or adapted in several ways by those skilled in the art without departing from the scope of the present application, as defined by the appended claims.

Claims

1. A gait rehabilitation evaluation method based on multi-modal data spatio-temporal features, characterized in that, The gait rehabilitation evaluation method based on multi-modal data space-time features comprises: Synchronously collecting electromyography signals, electroencephalography signals and exoskeleton robot IMU signals of a movement disorder group under a preset movement paradigm, and synchronously recording movement function score results of the movement disorder group; Performing data preprocessing and feature extraction on the electromyography signals and the electroencephalography signals to obtain first features, and performing data preprocessing and feature extraction on the exoskeleton robot IMU signals to obtain second features; Performing feature fusion processing on the first features and the second features to obtain fused features, training a time sequence evaluation network model according to the fused features and the movement function score results to obtain a target time sequence evaluation network model; Inputting processed brain-muscle coupling signal features and IMU signal gait features of an evaluated person into the target time sequence evaluation network model, outputting a movement function quantitative score of the evaluated person, and grading a movement function state of the evaluated person according to the movement function quantitative score to obtain a grading result; The first features comprise root mean square, average absolute value, median frequency and average power frequency; The second features comprise space-time features and gait symmetry features; The space-time features comprise step frequency and step length; the gait symmetry features comprise left-right gait cycle symmetry indexes and gait similarity of left and right legs; The data preprocessing and feature extraction on the electromyography signals and the electroencephalography signals to obtain the first features, and the data preprocessing and feature extraction on the exoskeleton robot IMU signals to obtain the second features specifically comprise: Filtering, artifact removal and data segmentation are performed on the electromyography signals and the electroencephalography signals to extract root mean square, average absolute value, median frequency and average power frequency of the electromyography signals and the electroencephalography signals respectively; Filtering is performed on the exoskeleton robot IMU signals to eliminate noise, and an extended Kalman filtering algorithm is used to convert original nine-axis data into joint angles, wherein the original nine-axis data comprise three-axis angular velocity, three-axis acceleration and three-axis magnetic field; Space-time features and gait symmetry features of the exoskeleton robot IMU signals after data preprocessing are extracted; The conversion of the original nine-axis data into joint angles by using the extended Kalman filtering algorithm specifically comprises: a prediction process and a correction process; ； The calculation of the step frequency is: ； in, It is a state transition function f State x The Jacobian matrix; yes k The state of the joint angle prior estimate at time t, using k The state at time -1 and control input Through state transition function f get; yes k The covariance matrix of the prior estimate of the state at each time step; yes k The covariance matrix of the posterior state estimate at time -1; T Indicates transpose; Q The process noise covariance matrix; It is the observation function h ( x ) for state x The Jacobian matrix; K It is the Kalman gain; R It is the observation noise covariance matrix; yes k The posterior estimate of the joint angle at time point; yes k The observed value at time; It is to utilize Through observation function h ( x The predicted observations obtained; yes k The covariance matrix of the posterior estimate of the state at time step; I It is an identity matrix.

2. The gait rehabilitation assessment method based on multi-modal data spatio-temporal features according to claim 1, characterized in that, The calculation of the step length is: ； wherein, C is the step frequency, is the total number of steps, is the total measurement time; SL ； wherein, The calculation of the left-right gait cycle symmetry indexes is: is the step length, is the walking speed, measured by sensors at the pelvis, is the gait cycle length; SI ； wherein, The calculation of the gait similarity of left and right legs is: is the left-right gait cycle symmetry index, is the left leg swing phase time, is the left leg gait cycle time, is the right leg swing phase time, is the right leg gait cycle time; Dist DTW(X,Y) ； wherein, is the Euclidean distance between two time series X i and Y j , X i is the i th sample point of the left leg joint angle time series, Y j is the j th sample point of the right leg joint angle time series, n is the length of the time series X , m is the length of the time series Y , The feature fusion processing on the first features and the second features to obtain the fused features specifically comprises: nm is the Euclidean distance between the n th sample point of the left leg and the m th sample point of the right leg, D is the final distance matrix, calculating on all frequencies to obtain a frequency resolution profile of directional connection; denotes the dynamic time warping distance between the time series X and Y , is the set of minimum accumulated distances of each cell in the dynamic programming process, is the accumulated distance corresponding to the th cell on the optimal alignment path p , is the number of accumulated distances on the optimal alignment path p .

3. The gait rehabilitation assessment method based on multi-modal data spatio-temporal features according to claim 2, characterized in that, The time sequence evaluation network model is a hybrid neural network model combining a convolutional neural network with a long short-term memory network with dual-channel input; According to the action, the electroencephalogram signal and the electromyogram signal are divided into non-overlapping time windows with a length of L and Each time window is converted into a symbol sequence using a quantile-based binning method.​ Each segment or The amplitude is mapped to a finite symbolic alphabet. ,in, express EEG amplitude at any given time express Electromyographic amplitude at time t, Indicates the first The and the first The dividing points of each amplitude interval, X and Y These represent discrete time series of electroencephalogram (EEG) and electromyogram (EMG), respectively. Indicates the size of the symbol set; Computing X the result of the encoding of the symbolization : ； Computing Y the result of the encoding of the symbolization : ； wherein, represents a symbolization result of each electroencephalogram amplitude point, represents a symbolization result of each electromyogram amplitude point; a fast Fourier transform is performed on each symbol sequence to extract the frequency components, and denote X and Y the symbol sequences varying in time, the corresponding Fourier spectrum calculation results are and ; The amplitude spectrum is extracted and a second symbol transformation is performed in the frequency domain to convert each frequency point The amplitude data of all time periods are binned into grades to form a sequence of frequency domain symbols and ; Defining transfer entropy : ； wherein, for measuring X the prediction gain of the future state, Y the prediction gain of the future state, and respectively represent X and Y past period vectors of represent t the electromyographic amplitude at the instant For each , the probability of the occurrence of a symbol Y based on the previous symbol depends on the previous symbol and X the previous symbol in the sequence , the transfer entropy from X to Y at is defined as: ； wherein is the empirical probability estimated from all the epoch symbol sequences; ​ From Y to X At the transfer spectrum entropy is defined as: ； After obtaining the results of the brain-muscle coupling analysis, the mean values of the three frequency bands are extracted as evaluation features. For the mean values of the frequency band coupling of the frequency bands are defined as ​ ； wherein, represents the minimum frequency, represents the maximum frequency, N represents the number of frequencies covered by the frequency band interval.

4. The gait rehabilitation assessment method based on multi-modal data spatio-temporal features according to claim 1, characterized in that, ​ The grading result includes normal motor function, mild motor dysfunction, moderate motor dysfunction, and severe motor dysfunction. 5.A gait rehabilitation assessment system based on spatiotemporal features of multi-modal data, characterized in that, The gait rehabilitation evaluation system based on multi-modal data spatiotemporal features is used to implement the gait rehabilitation evaluation method based on multi-modal data spatiotemporal features according to any one of claims 1-4, and the gait rehabilitation evaluation system based on multi-modal data spatiotemporal features comprises: A data acquisition and recording module is configured to synchronously acquire electromyographic signals, electroencephalographic signals, and exoskeleton robot IMU signals of a motor disorder population under a preset motor paradigm, and synchronously record motor function score results of the motor disorder population; A data preprocessing and feature extraction module is configured to perform data preprocessing and feature extraction on the electromyographic signals and the electroencephalographic signals to obtain first features, and perform data preprocessing and feature extraction on the exoskeleton robot IMU signals to obtain second features; A data fusion and model training module is configured to perform feature fusion processing on the first features and the second features to obtain fusion features, train a time series evaluation network model according to the fusion features and the motor function score results, and obtain a target time series evaluation network model; An evaluation prediction and grading module is configured to input processed brain-muscle coupling signal features and IMU signal gait features of an evaluated person into the target time series evaluation network model, output a motor function quantitative score of the evaluated person, grade a motor function state of the evaluated person according to the motor function quantitative score, and obtain a grading result.

6. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a gait rehabilitation evaluation program based on multi-modal data spatiotemporal features stored on the memory and executable on the processor, and the gait rehabilitation evaluation program based on multi-modal data spatiotemporal features implements steps of the gait rehabilitation evaluation method based on multi-modal data spatiotemporal features according to any one of claims 1-4 when executed by the processor.

7. A computer readable storage medium characterized in that, The computer-readable storage medium stores a gait rehabilitation evaluation program based on multi-modal data spatiotemporal features, and the gait rehabilitation evaluation program based on multi-modal data spatiotemporal features implements steps of the gait rehabilitation evaluation method based on multi-modal data spatiotemporal features according to any one of claims 1-4 when executed by the processor.

Images