Gait detection and anomaly recognition system based on multi-sensor fusion

Abstract

The invention provides a gait detection and anomaly recognition system based on multi-sensor fusion, and the system comprises a sensor integration module which is used for deploying a thin film pressure sensor and a lightweight surface myoelectricity sensor and carrying out the data collection; the data preprocessing module is used for preprocessing the collected data; the feature extraction and fusion module is used for extracting and fusing features of the collected data to obtain a multi-modal feature vector; the gait monitoring module is used for constructing a recognition model; the abnormal recognition module is used for carrying out recognition and real-time feedback on abnormal gaits by utilizing a recognition model; and the doctor end interaction module is used for designing a mobile end APP and a doctor end management platform. According to the invention, through the multi-modal data and the portable wearable device, the diversity of detection modes is increased, the detection precision is improved, the method can be applied to multiple scenes, and the environmental limitation is broken; and meanwhile, by deploying a deep learning model, the intelligent degree of gait anomaly recognition is improved, and comprehensive technical support is provided for rehabilitation of the patient.

Description

Technical Field

[0001] The present invention relates to the technical field of gait detection, and in particular to a gait detection and abnormal recognition system based on multi-sensor fusion. Background Art

[0002] Stroke, commonly known as "apoplexy", includes ischemic stroke and hemorrhagic stroke. When a stroke occurs, due to the blockage or rupture of blood vessels in the brain, local brain tissue is damaged. If the damaged brain tissue happens to be the area that controls the motor and sensory functions of one side of the body, it will cause hemiplegia on the contralateral side of the body, that is, stroke hemiplegia. Specifically, it is manifested as weakened or lost muscle strength in one limb, abnormal muscle tension, loss of motor control ability, and at the same time, it may be accompanied by symptoms such as decreased sensation and abnormal reflexes. Stroke patients often have various lower limb walking function disorders, such as movement control disorders, balance and stability disorders, etc., which bring many inconveniences to the daily activities of patients.

[0003] Most stroke hemiplegia patients are likely to recover a certain walking ability after timely and effective treatment and rehabilitation training. Precise gait detection can help doctors and rehabilitation therapists formulate personalized rehabilitation plans and improve the rehabilitation effect. However, the existing detection methods currently have the following problems: 1) Detection singularity: Existing technologies such as wearable devices and pressure-sensitive pads have relatively single data sources. Although gait data can be obtained, there is still a gap between the obtained data and the natural gait of patients, resulting in low gait detection accuracy. 2) Complex operation: Many devices require professional personnel to operate, such as attaching marker points and data processing. It is difficult for patients and ordinary medical staff to use them independently. 3) Many limitations: Devices such as optical systems and pressure-sensitive pads require a specific laboratory environment and cannot be used in the daily lives of patients. Summary of the Invention

[0004] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide a gait detection and abnormal recognition system based on multi-sensor fusion. Through multi-modal data and portable wearable devices, it not only increases the diversity of detection methods and improves the detection accuracy, but also can be applied in multiple scenarios and breaks the environmental limitations. At the same time, by deploying a deep learning model, the intelligence level of gait abnormal recognition is improved, providing comprehensive technical support for the rehabilitation of patients.

[0005] To achieve the above purpose, the present invention provides the following solution: A gait detection and abnormal recognition system based on multi-sensor fusion, comprising:

[0006] A sensor integration module, used to deploy thin-film pressure sensors on the insole, deploy lightweight surface electromyography sensors on the relevant muscle groups in the calf or foot area, and use a microcontroller to synchronously collect and preliminarily process sensor data to obtain original pressure data and original electromyography data;

[0007] A data preprocessing module, used for preprocessing the original pressure data and the original electromyographic data to obtain standard pressure data and standard electromyographic data;

[0008] A feature extraction and fusion module, for extracting plantar pressure features and electromyographic signal features based on the standard pressure data and the standard electromyographic data, and fusing the plantar pressure features and the electromyographic signal features to obtain a multimodal feature vector;

[0009] The gait monitoring module is used to design a deep learning model and use the gait data of healthy people and the abnormal gait data of stroke hemiplegic patients to train the model and obtain a recognition model;

[0010] An abnormality recognition module, used to use the recognition model to perform abnormal gait recognition and real-time feedback on the real-time multimodal feature vector;

[0011] The doctor-side interactive module is used to design a mobile APP that displays the gait monitoring interface, rehabilitation training suggestions and historical records, and to design a doctor-side management platform for patient file management, data statistical analysis and rehabilitation training customization;

[0012] Among them, the sensor integration module, the data preprocessing module, the feature extraction and fusion module, the gait monitoring module, the abnormality recognition module and the doctor-side interaction module are interconnected.

[0013] Optionally, the sensor integration module includes:

[0014] The pressure monitoring unit is used to calculate the density and position of the thin film pressure sensor according to the shape of the patient's sole and the pressure distribution, obtain the sensor array, and then use the SPI or I2C bus to connect the sensor array to monitor the pressure data;

[0015] The electromyographic signal monitoring unit is used to monitor the electromyographic signal using a lightweight surface electromyographic sensor, and to design an amplification circuit and a filtering circuit to amplify and filter the electromyographic signal, and then use an analog-to-digital converter to digitize the electromyographic signal to obtain electromyographic data;

[0016] A synchronous acquisition unit is used to synchronize the clock signal of the sensor, and synchronously acquire pressure data and electromyographic data using a microcontroller, perform preliminary filtering and denoising on the acquired data, obtain raw pressure data and raw electromyographic data, and then use wireless transmission technology to transmit data;

[0017] The power supply unit is used to utilize a rechargeable lithium battery for power supply and is combined with a piezoelectric energy harvester to utilize the sole pressure for auxiliary power supply.

[0018] Optionally, the data preprocessing module includes:

[0019] A pressure data preprocessing unit, for smoothing the pressure signal based on the raw pressure data using a moving average filter, removing transient spike noise, processing abnormal data values ​​using a median filter, and monitoring and correcting the zero offset of the sensor to obtain standard pressure data;

[0020] The electromyographic data preprocessing unit is used to remove low-frequency baseline drift and high-frequency noise based on the original electromyographic data using a bandpass filter, suppress power frequency interference using a notch filter, and separate low-frequency components and high-frequency components through wavelet transformation to remove low-frequency components and obtain standard electromyographic data.

[0021] Optionally, the feature extraction and fusion module includes:

[0022] A plantar pressure feature extraction unit, used to extract the spatial distribution characteristics, time series characteristics and foot symmetry index of the plantar pressure based on the standard pressure data;

[0023] An electromyographic signal feature extraction unit, used for extracting time domain features, frequency domain features and time-frequency domain features based on the standard electromyographic data;

[0024] The multimodal fusion unit is used to combine the attention mechanism to splice and fuse the plantar pressure feature and the electromyographic signal feature to obtain a high-dimensional feature vector, input the high-dimensional feature vector into a classification model or a regression model, and output a gait state or anomaly score.

[0025] Optionally, the plantar pressure feature extraction unit includes:

[0026] The spatial distribution feature subunit is used to calculate the pressure center of each frame of data according to the measurement points of the thin film pressure sensor array, obtain the pressure center trajectory of the sole, calculate the ratio of the total pressure of the toe and the heel in each frame of data, obtain the toe-heel pressure ratio, calculate the total pressure difference between the left and right feet, and obtain the pressure difference between the left and right feet;

[0027] The time series feature subunit is used to extract the gait cycle by using the periodic changes of the plantar pressure signal, calculate the step length and stride by using the maximum horizontal distance difference between the plantar pressure centers of the left and right feet in the walking direction, and calculate the time ratio of the swing period and the support period by using the contact and departure characteristics of the plantar pressure signal;

[0028] The foot symmetry index subunit is used to calculate the pressure integrals of the left and right feet and extract the maximum pressure peak in each gait cycle. The pressure integrals, maximum pressure peaks and the time ratio of the swing phase to the support phase of the left and right feet are combined to obtain a symmetry score to quantify gait symmetry.

[0029] Optionally, the plantar pressure center trajectory is used to analyze gait stability, the toe-to-heel pressure ratio is used to analyze the leg pedal force of the patient's gait and determine whether there is foot drop or dragging, and the left and right foot pressure difference is used to evaluate gait symmetry and identify insufficient pressure on the hemiplegic side; wherein, the expression of the plantar pressure center trajectory is:

[0030]

[0031] Among them, P i is the pressure value of the i-th sensor, x i ,y i are the position coordinates of the i-th sensor, COP x , COP y are the pressure centers of consecutive frames in the x and y directions respectively.

[0032] Optionally, the electromyographic signal feature extraction unit includes:

[0033] A time domain feature subunit is used to calculate the root mean square value based on the standard electromyographic data to obtain the muscle activation intensity, calculate the integrated electromyographic data to obtain the total muscle activation amount, and calculate the signal waveform length to evaluate the muscle activation complexity;

[0034] A frequency domain feature subunit, used for calculating the power spectrum distribution and the center frequency and frequency entropy of the power spectrum based on the standard electromyographic data;

[0035] The time-frequency domain feature subunit is used to extract the features of different frequency bands based on the standard electromyographic data by using discrete wavelet transform, and to extract the time-frequency features by using short-time Fourier transform to analyze the changing trend of muscle activation pattern over time.

[0036] Optionally, the expression for the mean square root is:

[0037]

[0038] Where N is the total number of sampling points, a i is the amplitude of the electromyographic signal at the i-th sampling point, The sum of the squares of the amplitudes of the sampling points;

[0039] The expression of integrated myoelectricity is:

[0040]

[0041] Among them, |a i | is the absolute value of the EMG signal amplitude at the i-th sampling point.

[0042] Optionally, the gait monitoring module includes:

[0043] An architecture design unit, used to combine a convolutional neural network, a long short-term memory network and a Transformer to obtain an initial model; wherein the convolutional neural network is used to extract the plantar pressure feature, the long short-term memory network is used to extract the electromyographic signal feature, and the Transformer is used to globally model the plantar pressure feature and the electromyographic signal feature to capture the key mode of gait;

[0044] The model training unit is used to collect gait data of healthy people and hemiplegic stroke patients to obtain a comprehensive data set, use the comprehensive data set to perform supervised learning of the initial model, and use a stratified cross-validation method to evaluate model performance, optimize hyperparameters, complete model training, and obtain a recognition model.

[0045] Optionally, the abnormality identification module includes:

[0046] The abnormal feature library unit is used to extract key abnormal features during the model training process and store them in the abnormal feature library;

[0047] An abnormality recognition unit, inputting the real-time multimodal feature vector into the recognition model, extracting the real-time gait feature, matching the real-time gait feature with the abnormality feature library to obtain an abnormal gait, and then performing abnormal classification and abnormality scoring on the abnormal gait;

[0048] The real-time feedback unit is used to preset an abnormal score threshold. When the abnormal score is greater than the abnormal score threshold, abnormal feedback is performed through the mobile APP, and the abnormal gait data and the abnormal score are uploaded to the doctor-side management platform.

[0049] The present invention discloses the following technical effects by providing a gait detection and abnormality recognition system based on multi-sensor fusion:

[0050] 1. Multimodal fusion: Different from the traditional single detection method, the present invention incorporates plantar pressure and bioelectric signals into gait detection at the same time, which can not only reflect the mechanical characteristics of the lower limbs, but also capture the neural regulation characteristics of muscle activity, with higher detection accuracy and robustness.

[0051] 2. Simple operation and portable: The present invention deploys a thin film pressure sensor (which is flexible and thin and can adapt to the curved surface of the insole) on the insole, and deploys a lightweight surface electromyography sensor (the electrode material can be made of flexible conductive polymer to ensure wearing comfort and signal stability) on the relevant muscle areas of the calf or foot. While taking into account accurate sampling, it makes wearing more comfortable, and patients can also achieve long-term and high-frequency testing in daily environments.

[0052] 3. Intelligent identification: By designing a deep learning model, it can automatically learn the spatiotemporal characteristics of plantar pressure and electromyographic signals, identify abnormal gait, improve the accuracy of gait abnormality identification, and provide real-time feedback on abnormal information to help patients adjust their exercise methods immediately and recover faster.

[0053] 4. Multi-scenario application: It can be used in rehabilitation departments, community medical institutions and even home environments, providing a more flexible solution for rehabilitation monitoring and personalized treatment of hemiplegic stroke patients.

[0054] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 A schematic diagram of a system architecture provided by an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of a flow chart of data collection and processing provided by an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of the process of gait detection and recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] like Figure 1 As shown, the present invention provides a gait detection and abnormality recognition system based on multi-sensor fusion, including an interconnected sensor integration module, a data preprocessing module, a feature extraction and fusion module, a gait monitoring module, an abnormality recognition module and a doctor-side interaction module.

[0062] 1. Sensor integration module

[0063] like Figure 2 As shown, a thin film pressure sensor is deployed on the insole, a lightweight surface electromyography sensor is deployed on the calf or foot related muscle group area, and a microcontroller is used to synchronously collect and preliminarily process the sensor data to obtain raw pressure data and raw electromyography data; the sensor integration module includes:

[0064] 1.1 Pressure monitoring unit

[0065] It is used to calculate the density and position of the thin film pressure sensor according to the shape and pressure distribution of the patient's sole to obtain a sensor array, including:

[0066] Heel: The bearing point, used to analyze gait stability;

[0067] Arch: Analyze the arch support and evaluate gait abnormalities (such as flat feet and hemiplegic side center of gravity shift);

[0068] Forefoot: Assess push-off force, reflecting the swing phase of the gait cycle.

[0069] Then use the SPI or I2C bus to connect the sensor array (for synchronous sampling) to monitor the pressure data.

[0070] 1.2 EMG signal monitoring unit

[0071] It is used to monitor the electromyographic signals of key lower limb muscle groups (such as tibialis anterior and gastrocnemius) using lightweight surface electromyographic sensors. Flexible conductive polymers are used as electrode materials to ensure wearing comfort and signal stability. Amplifier circuits and filter circuits are designed to amplify and filter electromyographic signals, and then analog-to-digital converters are used to digitize the electromyographic signals to obtain electromyographic data.

[0072] Amplification circuit: Design a high input impedance amplifier (such as INA333) to improve the detection sensitivity of weak electromyographic signals. The amplification factor is set to 1000 times to ensure that the signal amplitude is suitable for subsequent processing.

[0073] Filter circuit: Use a bandpass filter (20-500Hz) to remove low-frequency baseline drift and high-frequency noise. Add a notch filter (50Hz / 60Hz) to suppress power frequency interference.

[0074] 1.3 Synchronous acquisition unit

[0075] It is used to synchronize the clock signal of the sensor, and use a microcontroller (such as ARM Cortex-M4 or M33) to synchronously collect pressure data and electromyographic data, perform preliminary filtering and denoising on the collected data, obtain raw pressure data and raw electromyographic data, and then use wireless transmission technology to transmit data.

[0076] 1.4 Power supply unit

[0077] It is used to use a rechargeable lithium battery (such as 3.7V 500mAh, which supports long-term continuous operation) for power supply, and combined with a piezoelectric energy harvester, it uses the sole pressure for auxiliary power supply to extend the battery life.

[0078] Among them, the piezoelectric energy harvester is a device that converts mechanical energy into electrical energy using the piezoelectric effect. Its working principle is based on the piezoelectric effect. When external mechanical stress (such as vibration) acts on the piezoelectric material, the positive and negative charge centers inside the material are relatively displaced, resulting in a potential difference at both ends of the material. These charges are collected by electrodes and converted into electrical energy. This conversion process does not require an external power source or fuel, so it is environmentally friendly and self-powered.

[0079] 2. Data preprocessing module

[0080] like Figure 2 As shown, it is used to preprocess the original pressure data and the original electromyographic data to obtain standard pressure data and standard electromyographic data; the data preprocessing module includes:

[0081] 2.1 Pressure data preprocessing unit

[0082] It is used to smooth the pressure signal based on the raw pressure data using a moving average filter (window size is 5-10 data points), remove short-term spike noise, process abnormal data values ​​(such as discrete points caused by transient poor contact of the pressure sensor) using a median filter, and monitor and correct the zero point offset of the sensor to ensure the accuracy of the data and obtain standard pressure data.

[0083] 2.2 EMG data preprocessing unit

[0084] Based on the raw electromyographic data, a bandpass filter (20-500 Hz) is used to remove low-frequency baseline drift and high-frequency noise, a notch filter (50 Hz or 60 Hz) is used to suppress power frequency interference, and a wavelet transform is used to separate low-frequency components and high-frequency components to remove low-frequency components and obtain standard electromyographic data.

[0085] 3. Feature extraction and fusion module

[0086] like Figure 2 As shown, it is used to extract the plantar pressure feature and the electromyographic signal feature based on the standard pressure data and the standard electromyographic data, and fuse the plantar pressure feature and the electromyographic signal feature to obtain a multimodal feature vector; the feature extraction and fusion module includes:

[0087] 3.1 Plantar pressure feature extraction unit

[0088] Used to extract the spatial distribution characteristics, time series characteristics and foot symmetry index of plantar pressure based on the standard pressure data; the plantar pressure feature extraction unit includes:

[0089] 3.11 Spatial distribution feature subunits

[0090] 1) Center of plantar pressure (COP) trajectory

[0091] According to the measurement points of the plantar film pressure sensor array, the center of pressure (COP) of each frame of data is calculated:

[0092]

[0093] Among them, P i is the pressure value of the i-th sensor, x i ,y i are the position coordinates of the i-th sensor, COP x , COP y are the pressure centers of consecutive frames in the x and y directions respectively. x and COP y Connect to generate the plantar pressure center trajectory for analyzing gait stability.

[0094] 2) Toe / Heel Pressure Ratio

[0095] The toe-heel pressure ratio is calculated by calculating the sum of the toe-heel pressure ratio in each frame of data, which is used to analyze whether the leg pedaling force of the patient's gait is normal and to determine whether there is foot drop or dragging step.

[0096] 3) Pressure difference between left and right feet

[0097] The total pressure difference between the left and right feet was calculated to obtain the left and right foot pressure difference, which was used to evaluate gait symmetry and identify insufficient pressure on the hemiplegic side.

[0098] 3.12 Time Series Feature Subunit

[0099] It is used to utilize the periodic changes of plantar pressure signals to extract the gait cycle (such as the time interval from heel contact to the next heel contact), calculate the step length and stride width using the maximum horizontal distance difference between the plantar pressure centers of the left and right feet in the walking direction, and calculate the time ratio of the swing period (no contact) to the support period (with contact) through the contact and departure characteristics of the plantar pressure signals.

[0100] 3.13 Foot symmetry index subunit

[0101] It is used to calculate the pressure integrals of the left and right feet and extract the maximum pressure peak in each gait cycle. The pressure integrals, maximum pressure peak and the time ratio of the swing phase to the support phase of the left and right feet are combined to obtain a symmetry score to quantify gait symmetry.

[0102] 3.2 EMG signal feature extraction unit

[0103] Used to extract time domain features, frequency domain features and time-frequency domain features based on the standard electromyographic data; the electromyographic signal feature extraction unit includes:

[0104] 3.21 Time Domain Feature Subunit

[0105] It is used to calculate the root mean square value based on the standard electromyographic data to obtain the muscle activation intensity, calculate the integrated electromyographic data to obtain the total muscle activation amount, and calculate the signal waveform length to evaluate the complexity of muscle activation.

[0106] The expression for the mean square root is:

[0107]

[0108] Where N is the total number of sampling points, a i is the amplitude of the electromyographic signal at the i-th sampling point, The sum of the squared amplitudes of the sampling points.

[0109] The expression of integrated electromyography is:

[0110]

[0111] Among them, |a i | is the absolute value of the EMG signal amplitude at the i-th sampling point.

[0112] 3.22 Frequency Domain Feature Subunit

[0113] The method is used to calculate the power spectrum distribution and the center frequency and frequency entropy of the power spectrum based on the standard electromyographic data.

[0114] Power spectrum distribution: The power spectrum distribution of the electromyographic signal is calculated by fast Fourier transform (FFT) to extract the main frequency components.

[0115] Center frequency: The center frequency of the power spectrum is calculated and used to analyze the frequency characteristics of muscle activation.

[0116] Spectral entropy: Spectral entropy is calculated by normalizing the power spectrum, reflecting the randomness of muscle activation.

[0117] 3.23 Time-frequency domain feature subunit

[0118] It is used to extract the features of different frequency bands based on the standard electromyographic data using discrete wavelet transform (DWT), extract time-frequency features using short-time Fourier transform (STFT), and analyze the changing trend of muscle activation patterns over time.

[0119] 3.3 Multimodal Fusion Unit

[0120] It is used to combine the attention mechanism to splice and fuse the plantar pressure feature and the electromyographic signal feature to obtain a high-dimensional feature vector, input the high-dimensional feature vector into a classification model (such as random forest, support vector machine) or a regression model, and output the gait state or abnormality score.

[0121] Among them, the attention mechanism can dynamically adjust the weights of plantar pressure and electromyographic features according to gait events (such as support period and swing period) to highlight key features.

[0122] 4. Gait monitoring module

[0123] like Figure 3 As shown, it is used to design a deep learning model, and use the gait data of healthy people and the abnormal gait data of hemiplegic stroke patients to train the model to obtain a recognition model; the gait monitoring module includes:

[0124] 4.1 Architecture Design Unit

[0125] It is used to combine the convolutional neural network, the long short-term memory network and the transformer to obtain an initial model; wherein the convolutional neural network is used to extract the plantar pressure characteristics, the long short-term memory network is used to extract the electromyographic signal characteristics, and the transformer is used to globally model the plantar pressure characteristics and the electromyographic signal characteristics to capture the key patterns of gait. For example:

[0126] 1) CNN (Convolutional Neural Network):

[0127] Extract the spatial distribution characteristics of plantar pressure signals (such as COP trajectory, pressure distribution map);

[0128] Input: 2D data from the plantar pressure sensor array;

[0129] Output: eigenvector representing the spatial characteristics of the pressure distribution.

[0130] 2) LSTM (Long Short-Term Memory Network):

[0131] Capture the temporal characteristics of gait signals (e.g., gait cycle, myoelectric activation pattern);

[0132] Input: time series data (plantar pressure and electromyographic signals);

[0133] Output: time series features reflecting the dynamic changes of gait.

[0134] 3) Transformer:

[0135] For modeling the global spatiotemporal relationship between plantar pressure and electromyographic signals;

[0136] A multi-head attention mechanism is introduced at each time step to focus on key gait events (such as heel contact and ground lift).

[0137] 4.2 Model Training Unit

[0138] Used to collect gait data of healthy people and hemiplegic stroke patients to obtain a comprehensive data set, including:

[0139] Plantar pressure data (such as COP trajectory, left and right foot pressure difference);

[0140] EMG signal data (such as RMS, spectral entropy);

[0141] Note the type of gait abnormality (eg, shortened stride, foot drop, shuffling gait).

[0142] The comprehensive data set is used to perform supervised learning of the initial model, and a stratified cross-validation method is used to evaluate model performance, optimize hyperparameters, complete model training, and obtain a recognition model.

[0143] 5. Abnormal identification module

[0144] like Figure 3 As shown, the recognition model is used to identify abnormal gait and provide real-time feedback on the real-time multimodal feature vector; the abnormal recognition module includes:

[0145] 5.1 Abnormal Feature Library Unit

[0146] It is used to extract key abnormal features (such as abnormal COP trajectory and electromyography activation delay) during the model training process and store them in the abnormal feature library for use in the real-time recognition stage.

[0147] 5.2 Anomaly Identification Unit

[0148] The real-time multimodal feature vector is input into the recognition model to extract the real-time gait features (such as gait cycle, symmetry index, electromyographic activation pattern), and the real-time gait features are matched with the abnormal feature library to obtain abnormal gait, and then the abnormal gait is classified as abnormal (such as foot drop, dragging step) and abnormally scored (such as 0-100 points).

[0149] 5.3 Real-time feedback unit

[0150] It is used to preset an abnormal score threshold. When the abnormal score is greater than the abnormal score threshold, abnormal feedback is given through the mobile APP, and the abnormal gait data and the abnormal score are uploaded to the doctor-side management platform.

[0151] 6. Doctor-side interaction module

[0152] It is used to design mobile APP to display gait monitoring interface, rehabilitation training suggestions and historical records, and to design doctor-side management platform to manage patient files, perform data statistics analysis and customize rehabilitation training. For example:

[0153] 1) Monitoring interface

[0154] Real-time gait parameter display: Receive real-time gait characteristic data (such as pace, stride, symmetry index). Use charts (such as line charts, bar charts) to dynamically display the changing trend of gait parameters. Introduce animated gait simulation (such as dynamic plantar pressure distribution charts or virtual human walking animations) to let patients intuitively understand the gait status.

[0155] Abnormal prompt: Real-time monitoring of gait abnormalities. When certain indicators (such as gait symmetry score) are below the threshold, the patient is reminded through voice prompts or pop-up alarms.

[0156] 2) Rehabilitation training suggestions

[0157] Abnormal gait analysis: When an abnormal gait is detected (such as shortened stride, foot drop), the APP recommends corresponding rehabilitation training programs from the built-in training library based on the type of abnormality.

[0158] Training content: Balance training for unstable gait (such as single-leg standing, dynamic balance exercises). Active ankle training for foot drop (such as ankle flexion and extension exercises, resistance training).

[0159] Training guidance: Provide voice guidance and animation demonstrations to help patients complete training movements correctly. Support patients to monitor gait data in real time during training and dynamically adjust training intensity.

[0160] 3) Historical Records

[0161] Data storage and display: Store detailed data of each gait monitoring of the patient (such as gait parameters, abnormal event timestamps). Provide a visualization interface of historical data (such as trend line charts, bar charts) to help patients and their families understand the progress of rehabilitation.

[0162] Abnormal event playback: supports playback of gait data (such as plantar pressure distribution, electromyography signal activation mode) when abnormal events occur, and intuitively displays abnormal characteristics.

[0163] 4) Patient file management

[0164] Patient information storage: Each patient's file contains basic information (name, age, gender, medical record number), medical history information (stroke type, hemiplegic side, rehabilitation stage), and gait monitoring data (historical gait parameters, abnormal event records).

[0165] File retrieval and update: Supports doctors to quickly retrieve patient files and update patients' gait data and rehabilitation progress in real time.

[0166] 5) Data Statistical Analysis

[0167] Personal trend analysis: Provide historical trend analysis of patients' gait parameters (such as the change curve of stride length and speed). Identify the changing pattern of abnormal gait during the patient's rehabilitation process (such as whether the abnormal frequency decreases).

[0168] 6) Customized rehabilitation training

[0169] Prescription generation: Doctors prescribe personalized rehabilitation prescriptions through the platform based on the patient's gait data and abnormal classification. The prescriptions include: exercise intensity adjustment (such as increasing walking time, reducing training frequency), specific exercise recommendations (such as increasing balance training, active ankle training), etc.

[0170] Therefore, the present invention provides a gait detection and abnormality recognition system based on multi-sensor fusion, which not only increases the diversity of detection methods and improves detection accuracy through multimodal data and portable wearable devices, but also can be applied in multiple scenarios to break environmental limitations; at the same time, by deploying deep learning models, the intelligence level of gait abnormality recognition is improved, providing comprehensive technical support for patient rehabilitation.

[0171] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0172] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A gait detection and abnormality recognition system based on multi-sensor fusion, characterized in that: include: A sensor integration module is used to deploy a thin film pressure sensor on the insole, deploy a lightweight surface electromyography sensor on the relevant muscle group area of ​​the calf or foot, and use a microcontroller to synchronously collect and preliminarily process the sensor data to obtain raw pressure data and raw electromyography data; A data preprocessing module, used for preprocessing the original pressure data and the original electromyographic data to obtain standard pressure data and standard electromyographic data; A feature extraction and fusion module, for extracting plantar pressure features and electromyographic signal features based on the standard pressure data and the standard electromyographic data, and fusing the plantar pressure features and the electromyographic signal features to obtain a multimodal feature vector; The gait monitoring module is used to design a deep learning model and use the gait data of healthy people and the abnormal gait data of stroke hemiplegic patients to train the model and obtain a recognition model; An abnormality recognition module, used to use the recognition model to perform abnormal gait recognition and real-time feedback on the real-time multimodal feature vector; The doctor-side interactive module is used to design a mobile APP that displays the gait monitoring interface, rehabilitation training suggestions and historical records, and to design a doctor-side management platform for patient file management, data statistical analysis and rehabilitation training customization; Among them, the sensor integration module, the data preprocessing module, the feature extraction and fusion module, the gait monitoring module, the abnormality recognition module and the doctor-side interaction module are interconnected.

2. The gait detection and abnormality recognition system based on multi-sensor fusion according to claim 1 is characterized in that: The sensor integrated module comprises: The pressure monitoring unit is used to calculate the density and position of the thin film pressure sensor according to the shape of the patient's sole and the pressure distribution, obtain the sensor array, and then use the SPI or I2C bus to connect the sensor array to monitor the pressure data; The electromyographic signal monitoring unit is used to monitor the electromyographic signal using a lightweight surface electromyographic sensor, and to design an amplification circuit and a filtering circuit to amplify and filter the electromyographic signal, and then use an analog-to-digital converter to digitize the electromyographic signal to obtain electromyographic data; A synchronous acquisition unit is used to synchronize the clock signal of the sensor, and synchronously acquire pressure data and electromyographic data using a microcontroller, perform preliminary filtering and denoising on the acquired data, obtain raw pressure data and raw electromyographic data, and then use wireless transmission technology to transmit data; The power supply unit is used to utilize a rechargeable lithium battery for power supply and is combined with a piezoelectric energy harvester to utilize the sole pressure for auxiliary power supply.

3. The gait detection and abnormality recognition system based on multi-sensor fusion according to claim 2 is characterized in that: The data preprocessing module comprises: A pressure data preprocessing unit, for smoothing the pressure signal based on the raw pressure data using a moving average filter, removing transient spike noise, processing abnormal data values ​​using a median filter, and monitoring and correcting the zero offset of the sensor to obtain standard pressure data; The electromyographic data preprocessing unit is used to remove low-frequency baseline drift and high-frequency noise based on the original electromyographic data using a bandpass filter, suppress power frequency interference using a notch filter, and separate low-frequency components and high-frequency components through wavelet transformation to remove low-frequency components and obtain standard electromyographic data.

4. The gait detection and abnormality recognition system based on multi-sensor fusion according to claim 3 is characterized in that: The feature extraction and fusion module includes: A plantar pressure feature extraction unit, used to extract the spatial distribution characteristics, time series characteristics and foot symmetry index of the plantar pressure based on the standard pressure data; An electromyographic signal feature extraction unit, used for extracting time domain features, frequency domain features and time-frequency domain features based on the standard electromyographic data; The multimodal fusion unit is used to combine the attention mechanism to splice and fuse the plantar pressure feature and the electromyographic signal feature to obtain a high-dimensional feature vector, input the high-dimensional feature vector into a classification model or a regression model, and output a gait state or anomaly score.

5. The gait detection and abnormality recognition system based on multi-sensor fusion according to claim 4 is characterized in that: The plantar pressure feature extraction unit comprises: The spatial distribution feature subunit is used to calculate the pressure center of each frame of data according to the measurement points of the thin film pressure sensor array, obtain the pressure center trajectory of the sole, calculate the ratio of the total pressure of the toe and the heel in each frame of data, obtain the toe-heel pressure ratio, calculate the total pressure difference between the left and right feet, and obtain the pressure difference between the left and right feet; The time series feature subunit is used to extract the gait cycle by using the periodic changes of the plantar pressure signal, calculate the step length and stride by using the maximum horizontal distance difference between the plantar pressure centers of the left and right feet in the walking direction, and calculate the time ratio of the swing period and the support period by using the contact and departure characteristics of the plantar pressure signal; The foot symmetry index subunit is used to calculate the pressure integrals of the left and right feet and extract the maximum pressure peak in each gait cycle. The pressure integrals, maximum pressure peaks and the time ratio of the swing phase to the support phase of the left and right feet are combined to obtain a symmetry score to quantify gait symmetry.

6. The gait detection and abnormality recognition system based on multi-sensor fusion according to claim 5, characterized in that: The plantar pressure center trajectory is used to analyze gait stability, the toe-to-heel pressure ratio is used to analyze the leg pedal force of the patient's gait and determine whether there is foot drop or dragging, and the left and right foot pressure difference is used to evaluate gait symmetry and identify insufficient pressure on the hemiplegic side; wherein, the expression of the plantar pressure center trajectory is: Among them, P i is the pressure value of the i-th sensor, x i ,y i are the position coordinates of the i-th sensor, COP x , COP y are the pressure centers of consecutive frames in the x and y directions respectively.

7. The gait detection and abnormality recognition system based on multi-sensor fusion according to claim 6 is characterized in that: The electromyographic signal feature extraction unit comprises: A time domain feature subunit is used to calculate the root mean square value based on the standard electromyographic data to obtain the muscle activation intensity, calculate the integrated electromyographic data to obtain the total muscle activation amount, and calculate the signal waveform length to evaluate the muscle activation complexity; A frequency domain feature subunit, used for calculating the power spectrum distribution and the center frequency and frequency entropy of the power spectrum based on the standard electromyographic data; The time-frequency domain feature subunit is used to extract the features of different frequency bands based on the standard electromyographic data by using discrete wavelet transform, and to extract the time-frequency features by using short-time Fourier transform to analyze the changing trend of muscle activation pattern over time.

8. The gait detection and abnormality recognition system based on multi-sensor fusion according to claim 7 is characterized in that: The expression for the mean square root is: Where N is the total number of sampling points, a i is the amplitude of the electromyographic signal at the i-th sampling point, The sum of the squares of the amplitudes of the sampling points; The expression of integrated electromyography is: Among them, |a i | is the absolute value of the EMG signal amplitude at the i-th sampling point.

9. The gait detection and abnormality recognition system based on multi-sensor fusion according to claim 8, characterized in that: The gait monitoring module comprises: An architecture design unit, used to combine a convolutional neural network, a long short-term memory network and a Transformer to obtain an initial model; wherein the convolutional neural network is used to extract the plantar pressure feature, the long short-term memory network is used to extract the electromyographic signal feature, and the Transformer is used to globally model the plantar pressure feature and the electromyographic signal feature to capture the key mode of gait; The model training unit is used to collect gait data of healthy people and hemiplegic stroke patients to obtain a comprehensive data set, use the comprehensive data set to perform supervised learning of the initial model, and use a stratified cross-validation method to evaluate model performance, optimize hyperparameters, complete model training, and obtain a recognition model.

10. The gait detection and abnormality recognition system based on multi-sensor fusion according to claim 9, characterized in that: The abnormality identification module comprises: The abnormal feature library unit is used to extract key abnormal features during the model training process and store them in the abnormal feature library; An abnormality recognition unit, inputting the real-time multimodal feature vector into the recognition model, extracting the real-time gait feature, matching the real-time gait feature with the abnormality feature library to obtain an abnormal gait, and then performing abnormal classification and abnormality scoring on the abnormal gait; The real-time feedback unit is used to preset an abnormal score threshold. When the abnormal score is greater than the abnormal score threshold, abnormal feedback is performed through the mobile APP, and the abnormal gait data and the abnormal score are uploaded to the doctor-side management platform.

Images