Fatigue gait pattern recognition equipment and recognition method based on multi-modal sensor

Abstract

The invention relates to a fatigue gait mode recognition device and recognition method based on a multi-modal sensor, and the method comprises the steps: collecting multi-modal sensing data based on the multi-modal sensor, carrying out the preprocessing of the multi-modal sensing data, carrying out the gait segmentation, and obtaining a data sample after the gait segmentation, including the multi-modal sensing data and corresponding data features; constructing an improved feature fusion model, and training with a data sample until the model is stable; collecting multi-modal sensing data, preprocessing the multi-modal sensing data, inputting the preprocessed multi-modal sensing data into the trained model, and outputting a fatigue gait pattern recognition result; the equipment comprises two groups of multi-mode sensors for acquiring multi-mode sensing data, the synchronous transceiving device corresponds the two groups of matched multi-mode sensing data and then sends the data, and the controller acquires the multi-mode sensing data and recognizes a fatigue gait mode. According to the method, high-precision classification of fatigue gait modes is realized, and the accuracy is improved by 15%-20% compared with that of a traditional single-mode method; based on dynamic threshold classification and multi-dimensional feature analysis, user specific rehabilitation suggestions are generated, and the walking ability and safety are improved.

Description

Technical Field

[0001] The present invention relates to the fields of biomedical engineering, motion analysis technology and sensor technology, and in particular to a fatigue gait pattern recognition device and recognition method based on a multimodal sensor. Background Art

[0002] Gait refers to the posture and movement characteristics exhibited by the human body during walking. It is influenced by a variety of factors, including behavioral habits, occupation, education, age, and gender. It can also be altered by pathological conditions such as the nervous system, musculoskeletal system, and metabolic diseases. The control of walking is complex, involving central command, body balance, and coordinated control. Disorders in any of these components can affect gait, potentially compensating for the abnormality through other force-generating locations or masking it directly, leading to increasingly severe abnormalities and ultimately affecting people's normal lives.

[0003] Normal gait is characterized by stability, periodicity and rhythmicity, directionality, coordination, and individual differences. However, these characteristics often change significantly under fatigue or pathological conditions. Currently, gait analysis research mainly focuses on the identification and classification of normal and pathological gait patterns, but there is relatively little research on the identification of fatigue gait patterns. Existing methods mainly rely on patient self-report, manual assessment, or traditional gait analysis instruments. Such methods lack objectivity and real-time performance, and are unable to fully reflect individual characteristics and dynamic changes, limiting the accuracy of rehabilitation assessment and training.

[0004] For example, Chinese patent publication number CN118177780A discloses a method and system for identifying poor posture gait based on a convolutional neural network. The method uses plantar pressure signals collected by smart insoles and combines deep learning technology to automatically identify and determine whether the tester's gait is poor posture. However, the patent only targets poor posture gait and does not cover the dynamic characteristics of fatigue. For another example, Chinese patent publication number CN108742637A discloses a body state detection method and detection system based on a gait recognition device. The method uses an air pressure sensor to detect changes in air pressure in the insole cavity and identify gait information such as foot landing status, step frequency, and walking style. However, this method relies on fixed rules and is difficult to adapt to the physiological differences and dynamic changes in gait characteristics of different individuals. It also cannot achieve personalized and continuous monitoring and identification of fatigue gait. Summary of the Invention

[0005] In response to the problems existing in the existing technology, the present invention provides a fatigue gait pattern recognition device and recognition method based on multimodal sensors, which combines with a deep learning model to realize real-time monitoring and classification of human gait fatigue status, and is particularly suitable for health monitoring and motor dysfunction analysis in smart wearable devices.

[0006] The technical solution adopted by the present invention is a fatigue gait pattern recognition method based on a multimodal sensor. The method collects multimodal sensing data based on the multimodal sensor, performs gait segmentation after preprocessing, and obtains data samples after gait segmentation, including multimodal sensing data and corresponding data features;

[0007] Build an improved feature fusion model and train it with data samples until the model is stable;

[0008] Collect multimodal sensor data, input it into the trained model after preprocessing, and output the fatigue gait pattern recognition results.

[0009] Preferably, the multimodal sensing data includes muscle movement data, plantar mechanics data and foot movement data.

[0010] Preferably, the muscle motion data, plantar mechanics data and foot motion data are processed by a parallel filtering circuit of a low-pass filter and a high-pass filter, the plantar mechanics data and the foot motion data are filtered by a low-pass filter, and the muscle motion data is filtered by a high-pass filter; the data processed by the parallel filtering circuit are aligned to obtain pre-processed multimodal sensing data.

[0011] Preferably, based on the pre-processed multimodal sensing data, the gait phases are divided sequentially according to the gait cycle, and each dimension of the multimodal sensing data in any gait phase is normalized to obtain multidimensional features for characterizing the gait.

[0012] Preferably, the multidimensional features include time domain features, gait irregularity features, gait symmetry features, statistical features, and nonlinear features.

[0013] Preferably, the improved feature fusion model includes a deep convolutional neural network, an improved DR-SE module, a feature fusion module and a classification module connected in sequence.

[0014] In the present invention, a DCNN-DR-SE feature fusion model is constructed, and the standard SE module is passed through the DR (Dimensionality Reduction) module. The former reduces the spatial dimension of the feature through a pooling operation with a stride of 2, thereby reducing the amount of calculation.

[0015] Preferably, the deep convolutional neural network includes parallel convolutional layers corresponding to different modal data, all of which are connected to an average pooling layer and a batch normalization layer; each convolutional layer is linearly corrected with a ReLu function;

[0016] The improved DR-SE module includes a dimensionality reduction layer and a SE module connected in sequence;

[0017] The output end of the batch normalization layer is connected to the input end of the dimensionality reduction layer;

[0018] The output of the batch normalization layer is added to the output of the SE module and then input into the feature fusion module.

[0019] Preferably, the feature fusion module includes a convolution layer, an activation function layer and a maximum pooling layer connected in sequence;

[0020] The classification module includes a global average pooling layer, a fully connected layer, a Sigmoid activation function, and a dynamic threshold classification layer.

[0021] A fatigue gait pattern recognition device based on a multimodal sensor, the recognition device comprising:

[0022] 2 sets of multimodal sensors for collecting multimodal sensing data;

[0023] a synchronous transceiver, used to send two sets of matched multimodal sensor data in correspondence;

[0024] The controller is used to obtain the multimodal sensing data sent by the synchronous transceiver and perform fatigue gait pattern recognition using the fatigue gait pattern recognition method based on the multimodal sensor.

[0025] Preferably, any of the multimodal sensors includes a pressure sensor, an inertial sensor and several electromyographic sensors, and is provided with a data acquisition module and a charging module in conjunction with the pressure sensor, the inertial sensor and the several electromyographic sensors.

[0026] The present invention relates to a fatigue gait pattern recognition device and method based on a multimodal sensor. The device collects multimodal sensor data based on the multimodal sensor, performs gait segmentation after preprocessing, and obtains data samples after gait segmentation, including multimodal sensor data and corresponding data features; constructs an improved feature fusion model, and trains the model with the data samples until the model is stable; collects multimodal sensor data, inputs the trained model after preprocessing, and outputs fatigue gait pattern recognition results; the device includes two groups of multimodal sensors for collecting multimodal sensor data, uses a synchronous transceiver to match the two groups of multimodal sensor data and transmits them; a controller obtains the multimodal sensor data and performs fatigue gait pattern recognition.

[0027] The beneficial effect of the present invention lies in that, by using multimodal sensors to collect lower limb muscle fatigue, motion dynamics characteristics and mechanical parameters in real time, combined with the improved DCNN-DR-SE feature fusion model, high-precision classification of fatigue gait patterns (normal, fatigue, severe fatigue) is achieved, and the classification accuracy is improved by 15%-20% compared with the traditional single-modal method; based on dynamic threshold classification and multidimensional feature analysis, user-specific rehabilitation suggestions are generated to improve walking ability and safety. It is suitable for scenarios such as lower limb postoperative rehabilitation and fall prevention for the elderly, and supports health monitoring, such as athlete training load assessment, clinical rehabilitation, such as quantification of gait disorders in stroke patients and integration of smart wearable devices, and realizes real-time data transmission and telemedicine linkage through Bluetooth modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 It is a schematic diagram of the device structure of the present invention;

[0030] Figure 3 Schematic diagram of the division of gait phases in the present invention;

[0031] Figure 4 This is a structural block diagram of the DCNN-DR-SE feature fusion model in the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific implementations described herein are only used to explain the present invention and are not intended to limit the present invention.

[0033] The present invention relates to a fatigue gait pattern recognition method based on a multimodal sensor. The method collects multimodal sensing data based on the multimodal sensor, performs gait segmentation after preprocessing, and obtains data samples after gait segmentation, including multimodal sensing data and corresponding data features;

[0034] Build an improved feature fusion model and train it with data samples until the model is stable;

[0035] Collect multimodal sensor data, input it into the trained model after preprocessing, and output the fatigue gait pattern recognition results.

[0036] The present invention also relates to an identification device, comprising:

[0037] 2 sets of multimodal sensors for collecting multimodal sensing data;

[0038] A synchronous transceiver device, used to send signals after matching two sets of multi-modal sensors;

[0039] The controller is used to obtain the multimodal sensing data sent by the synchronous transceiver and perform fatigue gait pattern recognition using the fatigue gait pattern recognition method based on the multimodal sensor.

[0040] In practical applications, the multimodal sensing data includes muscle motion data, plantar mechanics data, and foot motion data.

[0041] The muscle motion data, plantar mechanics data and foot motion data are processed by a parallel filtering circuit composed of a low-pass filter and a high-pass filter, the plantar mechanics data and the foot motion data are filtered by a low-pass filter, and the muscle motion data is filtered by a high-pass filter; the data processed by the parallel filtering circuit are aligned to obtain pre-processed multimodal sensing data.

[0042] In the present invention, any of the multimodal sensors includes a pressure sensor, an inertial sensor and a plurality of myoelectric sensors, and is equipped with a data acquisition module and a charging module in conjunction with the pressure sensor, the inertial sensor and the plurality of myoelectric sensors;

[0043] Specifically, the multimodal sensor includes an electromyographic sensor for the calf, a sensor for embedding in the insole, and a small number of inertial sensors attached to the dorsum of the foot. The former is attached to the gastrocnemius and soleus muscles of the calf of both legs when in use, and uses four EMG electromyographic sensors. The latter includes a pressure sensor embedded in the insoles of both feet, an inertial sensor, and a small number of inertial sensors attached to the dorsum of the foot, and is equipped with a data acquisition module and a charging module to meet data collection needs, and a synchronous transceiver device to realize data transmission.

[0044] Pressure sensors are located on the toes, soles, and heels of the feet, while inertial sensors are located on the dorsum, toes, and soles of the feet. The layout of both feet is mirrored, with a total of 6 inertial sensors and 6 force sensors per foot.

[0045] The synchronous transceiver device only sets an antenna corresponding to the left foot or the right foot. Both feet send and receive signals through this antenna to ensure that the signals transmitted by both feet are synchronized. Bluetooth is used for signal transmission between the two feet.

[0046] In the present invention, EMG electromyographic sensors are used to obtain the user's muscle movement data, and force sensors and inertial sensors corresponding to the soles of the feet are used to obtain the user's mechanical data and movement data, thereby monitoring the electromyographic signals in motion and gait analysis in real time, providing key information on muscle function and movement trajectory under fatigue conditions.

[0047] After acquiring multimodal sensor signals, the following filtering strategies were used to preprocess the data to suppress noise and extract information from the valid frequency band. For the pressure sensor (mechanical data) and inertial sensor (motion data), a bandpass filter was used with a low-cutoff frequency of 15 Hz, a high-cutoff frequency of 150 Hz, and a center frequency of 47 Hz. This filter retained the signal within the 15 Hz to 150 Hz frequency band, eliminating low-frequency motion artifacts such as slow foot movement and high-frequency noise, such as sensor circuit interference. For the muscle movement signals collected by the electromyography (EMG) sensor, a high-pass filter was used with a cutoff frequency of 30 Hz to remove low-frequency interference below 30 Hz, such as skin contact noise and baseline drift, while retaining the valid frequency band from 30 Hz to 300 Hz, which reflects muscle activity. A Butterworth filter design with a fourth-order filter ensured a flat passband and steep stopband attenuation.

[0048] The transfer function of the filter is:

[0049]

[0050] in, is the cutoff frequency, and n is the filter order.

[0051] Based on the preprocessed multimodal sensor data, the gait phases are divided sequentially according to the gait cycle, and each dimension of the multimodal sensor data in any gait phase is normalized to obtain multidimensional features for characterizing gait.

[0052] The multidimensional features include time domain features, gait irregularity features, gait symmetry features, statistical features, and nonlinear features.

[0053] In the present invention, the dynamic threshold method is used to divide the collected multimodal sensor data into the following six continuous phases of the gait cycle, namely initial contact, load response, support phase, early swing phase, end swing phase and support phase. The judgment conditions and data of each phase are shown in Table 1;

[0054] Table 1 Gait cycle phase determination conditions and data

[0055]

[0056] After distinguishing the gait phases, the multimodal data within each gait phase are independently normalized as follows:

[0057] Pressure data comes from the foot sensor:

[0058] , is the phase mean, is the standard deviation within the phase;

[0059] The electromyographic signal comes from the EMG sensor:

[0060] , is the mean value of the electromyographic signal; is the standard deviation of the EMG signal; EMG is the raw EMG signal value;

[0061] Acceleration data comes from inertial sensors, .

[0062] After normalization, the multi-dimensional features corresponding to each gait phase can be extracted as follows:

[0063] Time domain feature extraction: extract the cadence feature (the number of steps per unit time) from the pressure sensor data; extract the step length feature (the displacement calculated by integrating the acceleration in the swing phase) from the inertial sensor data; and extract the peak acceleration value (the maximum acceleration value in the support phase) from the inertial sensor data.

[0064] Gait irregularity feature extraction, calculation of step interval standard deviation based on pressure signal timestamp; calculation of Lyapunov exponent through acceleration time series, ,in, is the standard deviation of the pace, is the step interval time, is the mean;

[0065] Gait symmetry feature extraction, data obtained from pressure sensors, left and right foot support phase difference, ;

[0066] Statistical feature extraction: obtaining the statistical mean / variance of the acceleration signal in each phase from the inertial sensor; obtaining the skewness / kurtosis of the electromyographic signal distribution morphology analysis from the EMG sensor;

[0067] Nonlinear feature extraction, phase space reconstruction based on acceleration signal, and calculation of maximum Lyapunov exponent to obtain chaotic characteristics quantification.

[0068] Based on these features, a representation of fatigue can be obtained. After matching with the data samples, the data samples are divided into training set and test set in a ratio of 7:3 for model training. The trained model can effectively capture the changes in gait under fatigue state and help identify fatigue state.

[0069] The improved feature fusion model includes a deep convolutional neural network, an improved DR-SE module, a feature fusion module and a classification module connected in sequence.

[0070] In the specific implementation process of the present invention, the model includes four parts.

[0071] The first part is the deep convolutional neural network DCNN, which is used to realize feature extraction;

[0072] The deep convolutional neural network includes parallel convolutional layers corresponding to different modal data, and all convolutional layers are connected to average pooling layers and batch normalization layers;

[0073] Stage 1: Data input and primary feature extraction;

[0074] Phase 1-1 Data input;

[0075] Through multimodal sensors, including pressure sensors, inertial sensors, and electromyographic sensors, the user's plantar pressure distribution, three-dimensional acceleration / angular velocity signals, and calf electromyographic signals are collected;

[0076] Sensor layout: Plantar pressure sensors are embedded in the insoles and distributed in the toes, arch, and heel areas, with a total of six pressure detection points per foot. Inertial sensors are fixed to the dorsum and sole of the foot to collect motion trajectory data. Myoelectric sensors are attached to the gastrocnemius and soleus muscles of the calf to monitor muscle fatigue signals.

[0077] Stage 1-2 data preprocessing operations;

[0078] Mechanical data (pressure and inertial sensor signals) were low-pass filtered (cutoff frequency 15 Hz) to suppress high-frequency noise, and electromyographic signals were high-pass filtered (cutoff frequency 30 Hz) to eliminate baseline drift. Normalization processing was also performed to standardize the sensor data within each gait phase (mean 0, variance 1).

[0079] Stage 1-3 Feature extraction process;

[0080] The convolution layer takes the preprocessed multimodal data as input and extracts local features using a 3×3 convolution kernel. Specifically, the data from the inertial sensor and electromyography sensor is processed in parallel with the data from the pressure sensor.

[0081] ReLU activation function layer introduces nonlinear mapping to meet ;

[0082] Average pooling layer: Through a 2×2 average pooling layer with a stride of 2, the feature space dimension is gradually reduced from H×W to H / 2×W / 2 and H / 4×W / 4 in two steps, reducing the feature space dimension to 1 / 4 of the original size;

[0083] Batch normalization layer normalizes features and accelerates model convergence;

[0084] Output multi-scale feature map .

[0085] The second part is feature enhancement (improved DR-SE module), which is used to implement lightweight attention mechanism and feature optimization;

[0086] The improved DR-SE module includes a dimensionality reduction layer and a SE module (compression excitation module) connected in sequence;

[0087] The dimension reduction layer is a maximum pooling layer with a stride of 2, which is used to reduce the spatial size of the input feature map from Reduce to , reduce computational complexity;

[0088] The channel attention mechanism based on the SE module compresses spatial information through the global average pooling layer, and finally generates the channel weight vector by the fully connected layer and the Sigmoid activation function to meet the following requirements:

[0089] ,

[0090] Among them, Z is the maximum pooling output, 、 is the weight of the fully connected layer, δ is the ReLU activation function, is the Sigmoid activation function;

[0091] Based on the standard Squeeze-and-Excitation (SE) module, this paper reduces the amount of computation by optimizing and compressing the number of neurons in the fully connected layer to 1 / 4 of the original number of channels, reducing the feature size by 50%.

[0092] Finally, the channel weights are multiplied by the original features channel by channel to enhance the key feature responses.

[0093] Part 3: Multimodal feature fusion for cross-modal feature integration and dimensionality reduction

[0094] The output end of the batch normalization layer is connected to the input end of the dimensionality reduction layer;

[0095] The output of the batch normalization layer is added to the output of the SE module and then input into the feature fusion module;

[0096] The feature fusion module includes a convolution layer, an activation function layer and a maximum pooling layer connected in sequence.

[0097] DCNN (dimension ×H×W) and DR-SE (dimensions ×H×W) modules are spliced ​​in the channel dimension to generate fusion features ( + )×H×W ;

[0098] Then, the number of channels of the 1X1 convolution layer is reduced to ×H×W;

[0099] Use the ReLU activation function to introduce nonlinearity, allowing the model to learn more complex feature representations;

[0100] The maximum pooling layer is used to further compress the feature size to .

[0101] Part 4 Dynamic Classification and Result Output for Adaptive Classification and Fatigue Assessment

[0102] The classification module includes a global average pooling layer, a fully connected layer, a Sigmoid activation function, and a dynamic threshold classification layer;

[0103] The features with spatial dimensions of H×W×C are compressed into a vector through the global average pooling layer GAP ;

[0104] The vector V is mapped to the classification space using a fully connected layer, using the Sigmoid activation function. The optimization is based on a composite loss function consisting of an improved BCELoss (binary cross entropy loss) and focal loss. The output dimension is 3, corresponding to normal, fatigue, and severe fatigue.

[0105] Since fatigue states may coexist (such as fatigue and severe fatigue), the Sigmoid activation function is used to activate each category independently, and the output of each category is independently mapped to [0,1] interval, indicating the confidence that the input sample belongs to category c, ,in, is the original output of the cth neuron in the fully connected layer;

[0106] The improved BCE + Focal Loss composite loss function is used for training to alleviate the problem of category imbalance. The loss function is:

[0107]

[0108] Among them, C is the number of categories of 3; is the true label of category c (0 or 1), is the category weight, ≥0 is the focusing coefficient, which is used to suppress the loss contribution of easy-to-classify samples;

[0109] Finally, dynamic threshold classification is performed and the dynamic classification threshold is set according to the distribution of training data:

[0110] Normal gait: P normal ≥ 0.5 and P fatigue < 0.4 and P severe fatigue < 0.3;

[0111] Fatigue gait: P fatigue ≥ 0.4 and P severe fatigue < 0.3;

[0112] Severe fatigue: P severe fatigue ≥ 0.3.

[0113] Phase 4-4 Result Output and Application

[0114] After dynamic threshold classification, the recognition results are transmitted via the Bluetooth module, allowing data representing the fatigue state to be labeled.

[0115] A specific embodiment is given below.

[0116] The inertial sensor, an MPU6050, boasts an output accuracy of 0.01° and a measurement range of ±180°. It measures 4cm × 4cm × 0.9mm (QFN-24 package). The sensor integrates a voltage regulator and temperature compensation module, and its power supply is compatible with 3.3V embedded systems. It transmits data via an SPI digital interface, acquiring acceleration and angular velocity signals in real time.

[0117] The pressure sensor, model IMS004-C20B, utilizes a six-channel thin-film piezoresistive sensor array. Each channel features an independent high-precision differential amplifier circuit (with adjustable gain from 10 to 1000), which converts resistance changes into a linear voltage signal. The output range is 0-3.3V (full scale 200N, sensitivity 15mV / N), compatible with the host controller's 12-bit ADC input (resolution 0.8mV), and is compatible with 3.3V-5V embedded systems.

[0118] Inertial sensors and force sensors collaborate to collect multimodal plantar data. The inertial sensors are directly connected to the main controller via a digital interface, collecting acceleration and angular velocity signals in real time. The pressure sensor signals are then impedance-matched and signal-conditioned by four low-noise instrumentation amplifiers (INA128) before being input into the main controller's ADC module for analog-to-digital conversion. The collected data is transmitted to the terminal device in real time via Bluetooth.

[0119] The data acquisition module uses the STM32F103ZET6 as the main controller, integrates the ARM Cortex-M3 core, supports multiple UART serial ports (baud rate 115200bps) and 12-bit ADC (sampling rate 1MHz), and builds a low-latency, high-precision data acquisition platform.

[0120] The Bluetooth module uses the HC-05 (Bluetooth 2.1 + EDR protocol) with an operating voltage of 3.3V-5V. It connects to the main controller via a UART interface, enabling wireless data transmission and remote monitoring. Its effective transmission distance is 10 meters (in an unobstructed environment), meeting the needs of real-time data transmission.

[0121] EMG electromyographic sensors are attached to the gastrocnemius and soleus muscles of the calf to monitor muscle activity signals in real time, providing key information on muscle function and movement trajectory under fatigue conditions.

[0122] Pressure sensors and inertial sensors are embedded in the soles of the feet or attached to the instep of the feet to monitor in real time the user's lower limb muscle fatigue, dynamic motion characteristics and mechanical changes during walking.

[0123] The present invention also relates to a computer-readable storage medium having stored thereon a fatigue gait pattern recognition program based on a multimodal sensor. When the program is executed by a processor, the fatigue gait pattern recognition method based on a multimodal sensor can be implemented.

[0124] The present invention also relates to a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the aforementioned fatigue gait pattern recognition method based on multimodal sensors is implemented.

[0125] Although the preferred embodiments of the present invention have been described, those skilled in the art may make appropriate changes and modifications to these embodiments after understanding the core creative concepts of the present invention. Therefore, the appended claims are intended to be interpreted as covering the preferred embodiments of the present invention and all reasonable changes and modifications thereof.

[0126] Obviously, those skilled in the art may make various changes and modifications to the present invention, as long as such changes and modifications do not depart from the spirit and scope of the present invention. If such changes and modifications fall within the scope of the claims of the present invention and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A fatigue gait pattern recognition method based on multimodal sensors, characterized by: The method collects multimodal sensing data based on a multimodal sensor, performs gait segmentation after preprocessing, and obtains data samples after gait segmentation, including multimodal sensing data and corresponding data features; Build an improved feature fusion model and train it with data samples until the model is stable; Collect multimodal sensor data, input it into the trained model after preprocessing, and output the fatigue gait pattern recognition results.

2. The fatigue gait pattern recognition method based on multimodal sensors according to claim 1, characterized in that: The multimodal sensing data includes muscle motion data, plantar mechanics data and foot motion data.

3. The fatigue gait pattern recognition method based on multimodal sensors according to claim 2, characterized in that: Processing the muscle motion data, plantar mechanics data, and foot motion data with a parallel filtering circuit of a low-pass filter and a high-pass filter, wherein the plantar mechanics data and the foot motion data are filtered by the low-pass filter, and the muscle motion data is filtered by the high-pass filter; The data processed by the parallel filter circuit are aligned to obtain pre-processed multimodal sensing data.

4. The fatigue gait pattern recognition method based on multimodal sensors according to claim 1, characterized in that: Based on the preprocessed multimodal sensor data, the gait phases are divided sequentially according to the gait cycle, and each dimension of the multimodal sensor data in any gait phase is normalized to obtain multidimensional features for characterizing gait.

5. The fatigue gait pattern recognition method based on multimodal sensors according to claim 4 is characterized in that: The multidimensional features include time domain features, gait irregularity features, gait symmetry features, statistical features, and nonlinear features.

6. The fatigue gait pattern recognition method based on multimodal sensors according to claim 1, characterized in that: The improved feature fusion model includes a deep convolutional neural network, an improved DR-SE module, a feature fusion module and a classification module connected in sequence.

7. The fatigue gait pattern recognition method based on multimodal sensors according to claim 6, characterized in that: The deep convolutional neural network includes parallel convolutional layers corresponding to different modal data, and all convolutional layers are connected to average pooling layers and batch normalization layers; The improved DR-SE module includes a dimensionality reduction layer and a SE module connected in sequence; The output end of the batch normalization layer is connected to the input end of the dimensionality reduction layer; The output of the batch normalization layer is added to the output of the SE module and then input into the feature fusion module.

8. The fatigue gait pattern recognition method based on multimodal sensors according to claim 6, characterized in that: The feature fusion module includes a convolution layer, an activation function layer and a maximum pooling layer connected in sequence; The classification module includes a global average pooling layer, a fully connected layer, a Sigmoid activation function, and a dynamic threshold classification layer.

9. A fatigue gait pattern recognition device based on a multimodal sensor, characterized by: The identification device comprises: 2 sets of multimodal sensors for collecting multimodal sensing data; a synchronous transceiver, used to send two sets of matched multimodal sensor data in correspondence; The controller is used to obtain multimodal sensing data sent by the synchronous transceiver and perform fatigue gait pattern recognition using the fatigue gait pattern recognition method based on multimodal sensors as described in any one of claims 1 to 8.

10. The fatigue gait pattern recognition device based on a multimodal sensor according to claim 9, characterized in that: Any of the multimodal sensors includes a pressure sensor, an inertial sensor and a plurality of myoelectric sensors, and is provided with a data acquisition module and a charging module in conjunction with the pressure sensor, the inertial sensor and the plurality of myoelectric sensors.

Images