Fatigue gait pattern recognition device and method based on multi-modal sensors

Abstract

The present application relates to a fatigue gait pattern recognition device and method based on a multi-modal sensor, multi-modal sensor data is collected based on a multi-modal sensor, gait segmentation is performed after preprocessing, data samples after gait segmentation are obtained, including multi-modal sensor data and corresponding data features; an improved feature fusion model is constructed, and the model is trained to be stable with data samples; multi-modal sensor data is collected, and the preprocessed data is input into the trained model, and the fatigue gait pattern recognition result is output; the device includes two groups of multi-modal sensors for collecting multi-modal sensor data, and the synchronous transceiver device sends the corresponding two groups of multi-modal sensor data after matching; the controller acquires the multi-modal sensor data and recognizes the fatigue gait pattern. The present application realizes high-precision classification of fatigue gait pattern, and the accuracy is improved by 15%-20% compared with traditional single-mode method; based on dynamic threshold classification and multi-dimensional feature analysis, user-specific rehabilitation suggestions are generated, and walking ability and safety are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of biomedical engineering, motion analysis technology and sensor technology, in particular to a fatigue gait pattern recognition device and method based on multi-modal sensors. BACKGROUND

[0002] Gait refers to the posture and movement behavior characteristics exhibited by the human body during walking, which is influenced by various factors such as behavior habits, occupation, education, age, and gender, and also changes due to pathological conditions such as nervous system, skeletal muscle system, and metabolic diseases. The control of walking is very complex, including central command, body balance and coordination control, and any imbalance in any link can affect gait and compensate for abnormalities through other force positions or directly mask, thus causing the abnormalities involved in the imbalance to become more and more serious, until affecting people's normal life.

[0003] Normal gait has stability, periodicity and rhythm, directionality, coordination, and individual differences. However, under fatigue or pathological conditions, these characteristics often change significantly. Currently, gait analysis research mainly focuses on the recognition and classification of normal and pathological gait patterns, but relatively less on fatigue gait pattern recognition. Existing methods mainly rely on patient self-reporting, manual evaluation or traditional gait analysis instruments. Such methods lack objectivity and real-time performance, and also fail to fully reflect individualized characteristics and dynamic changes, limiting the precision of rehabilitation evaluation and training.

[0004] For example, Chinese Patent Publication No. CN118177780A discloses a posture-improper gait recognition method and system based on a convolutional neural network, which automatically identifies and judges whether the gait of a tester belongs to a posture-improper gait by collecting plantar pressure signals from an intelligent insole and combining deep learning technology; however, this patent only targets posture-improper gait and does not cover dynamic characteristics under fatigue conditions. For another example, Chinese Patent Publication No. CN108742637A discloses a body state detection method and system based on a gait recognition device, which detects changes in air pressure in the insole cavity through an air pressure sensor to recognize gait information such as foot landing state, step frequency, and walking style; however, this method relies on fixed rules and is difficult to adapt to physiological differences and gait dynamic characteristic changes of different individuals, and also cannot achieve personalized and continuous monitoring and recognition of fatigue gait. SUMMARY

[0005] To address the problems of the prior art, the present application provides a fatigue gait pattern recognition device and method based on multi-modal sensors, which realizes real-time monitoring and classification of human gait fatigue state in combination with a deep learning model, and is particularly suitable for motion dysfunction analysis in health monitoring and intelligent wearable devices.

[0006] The technical scheme adopted by the present application is a fatigue gait pattern recognition method based on a multi-modal sensor, which collects multi-modal sensor data based on the multi-modal sensor, performs gait segmentation after preprocessing, and obtains data samples after gait segmentation, including multi-modal sensor data and corresponding data features.

[0007] An improved feature fusion model is constructed, and the model is trained to be stable by using data samples.

[0008] Multi-modal sensor data is collected, and the preprocessed data is input into the trained model to output the fatigue gait pattern recognition result.

[0009] Preferably, the multi-modal sensor data includes muscle movement data, plantar mechanics data and foot movement data.

[0010] Preferably, the muscle movement data, the plantar mechanics data and the foot movement data are processed by a parallel filter circuit of a low-pass filter and a high-pass filter, the plantar mechanics data and the foot movement data are filtered by the low-pass filter, and the muscle movement data is filtered by the high-pass filter; the data processed by the parallel filter circuit is aligned to obtain the preprocessed multi-modal sensor data.

[0011] Preferably, based on the preprocessed multi-modal sensor data, the gait phases are sequentially divided according to the gait cycle, and each dimension of the multi-modal sensor data in any gait phase is normalized to obtain multi-dimensional features for representing the gait.

[0012] Preferably, the multi-dimensional features include time domain features, gait irregularity features, gait symmetry features, statistical features and nonlinear features.

[0013] Preferably, the improved feature fusion model includes a deep convolutional neural network, an improved DR-SE module, a feature fusion module and a classification module connected in sequence.

[0014] In the present application, a DCNN-DR-SE feature fusion model is constructed, so that the standard SE module passes through a DR (Dimensionality Reduction) module, and the former reduces the spatial dimension of the features by a pooling operation with a stride of 2 to reduce the calculation amount.

[0015] Preferably, the deep convolutional neural network includes parallel convolutional layers corresponding to different modal data, all the convolutional layers are connected to an average pooling layer and a batch normalization layer; each convolutional layer is linearly corrected by a ReLu function here;

[0016] The improved DR-SE module includes a dimension reduction layer and an SE module connected in sequence.

[0017] The output end of the batch normalization layer is connected to the input end of the dimension reduction layer.

[0018] The output of the batch normalization layer is added to the output of the SE module and then input into a feature fusion module.

[0019] Preferably, the feature fusion module comprises a convolution layer, an activation function layer and a max-pooling layer connected in sequence.

[0020] The classification module comprises a global average pooling layer, a fully connected layer, a Sigmoid activation function and a dynamic threshold classification layer.

[0021] A fatigue gait pattern recognition device based on multi-modal sensors, the recognition device comprising:

[0022] Two groups of multi-modal sensors for collecting multi-modal sensor data;

[0023] A synchronous transceiving device for transmitting the corresponding two groups of multi-modal sensor data after matching;

[0024] A controller for obtaining the multi-modal sensor data transmitted by the synchronous transceiving device and performing fatigue gait pattern recognition by using the fatigue gait pattern recognition method based on multi-modal sensors.

[0025] Preferably, any of the multi-modal sensors comprises a pressure sensor, an inertial sensor and a plurality of electromyography sensors, and the pressure sensor, the inertial sensor and the plurality of electromyography sensors are provided with a data acquisition module and a charging module.

[0026] The present application relates to a fatigue gait pattern recognition device and method based on multi-modal sensors, which collects multi-modal sensor data based on multi-modal sensors, performs gait segmentation after preprocessing, obtains data samples after gait segmentation, including multi-modal sensor data and corresponding data features, constructs an improved feature fusion model, trains the model with data samples until the model is stable, collects multi-modal sensor data, inputs the preprocessed data into the trained model, and outputs the fatigue gait pattern recognition result; the device comprises two groups of multi-modal sensors for collecting multi-modal sensor data, a synchronous transceiving device for transmitting the corresponding two groups of multi-modal sensor data after matching, and a controller for obtaining the multi-modal sensor data and performing fatigue gait pattern recognition.

[0027] The application has the beneficial effects that through the multi-modal sensor, the lower limb muscle fatigue degree, the motion dynamics characteristics and the mechanical parameters are collected in real time, the improved DCNN-DR-SE feature fusion model is combined, the high-precision classification (normal, fatigue, severe fatigue) of the fatigue gait mode is realized, the classification accuracy is improved by 15%-20% compared with the traditional single-mode method, the user-specific rehabilitation suggestion is generated based on the dynamic threshold classification and multi-dimensional feature analysis, the walking ability and safety are improved, and the application is suitable for the lower limb postoperative rehabilitation, the fall prevention of the elderly and the like, the health monitoring such as the training load evaluation of athletes, the clinical rehabilitation such as the gait disorder quantification of stroke patients and the integration of intelligent wearable devices, the real-time data transmission and remote medical linkage are realized through the Bluetooth module. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The method flowchart of the application is shown in the figure;

[0029] Figure 2 The device structure schematic diagram of the application is shown in the figure;

[0030] Figure 3 The gait phase division schematic diagram in the application is shown in the figure;

[0031] Figure 4 The structure block diagram of the DCNN-DR-SE feature fusion model in the application is shown in the figure. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the application more clear, the application is further described in detail below in combination with the embodiments. It should be understood that the specific implementation described herein is only used to explain the application, and is not used to limit the application.

[0033] The application relates to a fatigue gait mode recognition method based on a multi-modal sensor.

[0034] An improved feature fusion model is constructed, and the data sample is used for training until the model is stable.

[0035] The multi-modal sensor data is collected, and after preprocessing, the trained model is input, and the fatigue gait mode recognition result is output.

[0036] The application also relates to a recognition device.

[0037] Two groups of multi-modal sensors are used to collect multi-modal sensor data.

[0038] A synchronous transceiver is used to transmit the corresponding two groups of multi-modal sensors.

[0039] a controller for acquiring the multi-modal sensing data transmitted by the synchronous transceiving device and performing fatigue gait pattern recognition by using the multi-modal sensor-based fatigue gait pattern recognition method.

[0040] In actual application, the multi-modal sensing data includes muscle movement data, plantar mechanics data and foot movement data.

[0041] The muscle movement data, the plantar mechanics data and the foot movement data are processed by a parallel filter circuit of a low-pass filter and a high-pass filter, the plantar mechanics data and the foot movement data are filtered by the low-pass filter, and the muscle movement data is filtered by the high-pass filter; the data processed by the parallel filter circuit is aligned to obtain the pre-processed multi-modal sensing data.

[0042] In the application, any multi-modal sensor includes a pressure sensor, an inertial sensor and a plurality of electromyography sensors, and a data acquisition module and a charging module are arranged in cooperation with the pressure sensor, the inertial sensor and the plurality of electromyography sensors.

[0043] Specifically, the multi-modal sensor includes electromyography sensors arranged on the calf part and sensors embedded in insoles, and a small number of inertial sensors attached to the instep; the former is attached to the gastrocnemius muscle and the soleus muscle of the calf part of the legs during use, and four EMG electromyography sensors are used; the latter includes pressure sensors embedded in insoles, inertial sensors and a small number of inertial sensors attached to the instep, and a data acquisition module and a charging module are arranged in cooperation to meet the data acquisition, so that the data transmission is realized by the synchronous transceiving device.

[0044] The pressure sensors are arranged at the toes, the soles and the heels of the feet, the inertial sensors are arranged at the insteps, the toes and the soles, the layout of the two feet is mirror image, there are six inertial sensors for one foot, and there are six force sensors for one foot.

[0045] The synchronous transceiving device is provided with an antenna corresponding to the left foot or the right foot, the two feet transmit and receive signals through the antenna, the synchronization of the signal transmission of the two feet is ensured, and the signal transmission is realized by using Bluetooth between the two feet.

[0046] In the application, the muscle movement data of the user is acquired by using the EMG electromyography sensor, the mechanics data and the movement data of the user are acquired by using the force sensors and the inertial sensors corresponding to the soles, and the electromyography signals in the movement and the gait analysis are monitored in real time, so that the key information of the muscle function and the movement trajectory in the fatigue state is provided.

[0047] After acquiring the multi-modal sensor signals, the following filtering strategy is adopted for data preprocessing to suppress noise and extract effective frequency band information. For the pressure sensor (mechanical data) and inertial sensor (motion data), a band-pass filter is used, with the parameters set as a low cutoff frequency of 15 Hz, a high cutoff frequency of 150 Hz, and a center frequency of 47 Hz. The signals within the frequency band of 15 Hz to 150 Hz are retained, and low-frequency motion artifacts such as slow foot movement and high-frequency noise such as sensor circuit interference are eliminated. For the muscle movement signals collected by the electromyography sensor (EMG), a high-pass filter is used, with the parameter set as a cutoff frequency of 30 Hz. Low-frequency interference such as skin contact noise and baseline drift below 30 Hz is filtered out, and the 30 Hz to 300 Hz effective frequency band reflecting muscle activity is retained. A Butterworth filter design is used, with a 4th order, to ensure a flat passband and steep attenuation in the stopband.

[0048] The transfer function of the filter is:

[0049]

[0050] where, is the cutoff angular frequency, and n is the filter order.

[0051] Based on the preprocessed multi-modal sensor data, the gait phases are sequentially divided according to the gait cycle, and each dimension of the multi-modal sensor data within each gait phase is normalized to obtain multi-dimensional features for characterizing the gait.

[0052] The multi-dimensional features include time domain features, gait irregularity features, gait symmetry features, statistical features, and nonlinear features.

[0053] In the present application, the collected multi-modal sensor data is divided into the following 6 consecutive phase gait cycles using a dynamic threshold method: initial contact, load response, support phase, pre-swing, swing termination, and support phase. The phase determination conditions and data are shown in Table 1.

[0054] Table 1: Gait cycle phase determination conditions and data

[0055]

[0056] After distinguishing the gait phases, the multi-modal data within each gait phase is independently normalized as follows:

[0057] The pressure data is derived from the plantar sensor:

[0058] , is the mean value within the phase, is the standard deviation within the phase;

[0059] The myoelectric signal is derived from the EMG sensor:

[0060] , is the mean value of the myoelectric signal; is the standard deviation of the myoelectric signal; EMG is the original myoelectric signal value;

[0061] The acceleration data is derived from the inertial sensor, .

[0062] After normalization, the multi-dimensional features corresponding to each gait phase can be extracted, as follows:

[0063] Time domain feature extraction: the step frequency feature of the number of steps per unit time is extracted from the pressure sensor data; the step length feature of the displacement calculated by integrating the acceleration in the swing phase is extracted from the inertial sensor data; the acceleration peak value of the maximum acceleration value in the support phase is extracted from the inertial sensor data;

[0064] Gait irregularity feature extraction: the standard deviation of the step interval is calculated based on the pressure signal timestamp; the Lyapunov exponent is calculated through the acceleration time sequence, wherein, is the step standard deviation, is the step interval time, is the mean value;

[0065] Gait symmetry feature extraction: the data from the pressure sensor is obtained, and the difference between the left and right foot support phases, ;

[0066] Statistical feature extraction: the statistical mean / variance of the acceleration signal in each phase is obtained from the inertial sensor; the skewness / kurtosis of the myoelectric signal distribution shape analysis is obtained from the EMG sensor;

[0067] Nonlinear feature extraction: the phase space is reconstructed based on the acceleration signal, and the maximum Lyapunov exponent is calculated to obtain the chaos characteristic quantization.

[0068] According to these features, the representation of fatigue state can be obtained, and after matching with the data sample, the data sample is divided into a training set and a test set in a ratio of 7:3, which is used for training the model. The trained model can effectively capture the changes of gait in the fatigue state, helping to identify the fatigue state.

[0069] The improved feature fusion model comprises a deep convolutional neural network, an improved DR-SE module, a feature fusion module and a classification module connected in sequence.

[0070] In the specific implementation process of the present application, the model comprises four parts.

[0071] The first part is a deep convolutional neural network (DCNN) for feature extraction;

[0072] The deep convolutional neural network comprises parallel convolutional layers corresponding to different modal data respectively, all the convolutional layers being connected to an average pooling layer and a batch normalization layer;

[0073] Stage 1: data input and primary feature extraction;

[0074] Stage 1-1: data input;

[0075] Through multi-modal sensors including pressure sensors, inertial sensors and electromyography sensors, the plantar pressure distribution, three-dimensional acceleration / angular velocity signals and calf electromyography signals of the user are collected;

[0076] Sensor layout: the plantar pressure sensors are embedded in the insole and distributed in the toe, arch and heel areas, with a total of 6 pressure detection points for one foot; the inertial sensors are fixed on the dorsum and plantar of the foot to collect motion trajectory data; the electromyography sensors are attached to the gastrocnemius and soleus muscles of the calf to monitor muscle fatigue signals;

[0077] Stage 1-2: data preprocessing operation;

[0078] Low-pass filtering (cutoff frequency 15 Hz) is adopted for the mechanical data (pressure, inertial sensor signals) to suppress high-frequency noise, and high-pass filtering (cutoff frequency 30 Hz) is adopted for the electromyography signals to eliminate baseline drift; normalization processing is performed, and the sensor data within each gait phase is standardized (mean value 0, variance 1);

[0079] Stage 1-3: feature extraction process;

[0080] Convolutional layer: the preprocessed multi-modal data is input, and local features are extracted through a 3x3 convolutional kernel; specifically, the data obtained by the inertial sensors and electromyography sensors are processed in parallel with the data obtained by the pressure sensors;

[0081] ReLU activation function layer: a nonlinear mapping is introduced, satisfying ;

[0082] Average pooling layer: through a 2x2 average pooling layer with a stride of 2, the feature space dimension is gradually reduced from HxW to H / 2xW / 2 and H / 4xW / 4 in two steps, and the feature space dimension is reduced to 1 / 4 of the original size;

[0083] Batch normalization layer: the features are normalized to accelerate model convergence;

[0084] Output multi-scale feature maps .

[0085] The second part, feature enhancement (improved DR-SE module), is used to implement a lightweight attention mechanism and feature optimization.

[0086] The improved DR-SE module includes a dimensionality reduction layer and an SE module (compressed excitation module) connected in sequence.

[0087] The dimensionality reduction layer is a max-pooling layer with a stride of 2, used to reduce the spatial size of the input feature map from... Reduced to This reduces computational complexity;

[0088] The channel attention mechanism based on the SE module compresses spatial information through a global average pooling layer, and finally generates channel weight vectors using a fully connected layer and a sigmoid activation function, satisfying the following conditions:

[0089] ,

[0090] Where Z is the max-pooling output. , Here, δ represents the weights of the fully connected layer, and δ is the ReLU activation function. Use the Sigmoid activation function;

[0091] This invention, based on the standard Squeeze-and-Excitation (SE) module, optimizes and reduces computational load, compressing the number of neurons in the fully connected layer to 1 / 4 of the original number of channels and reducing the feature size by 50%.

[0092] Finally, the channel weights are multiplied with the original features channel by channel to enhance the response of key features.

[0093] Part 3: Multimodal feature fusion, used for cross-modal feature integration and dimensionality reduction.

[0094] The output of the batch normalization layer is connected to the input of the dimensionality reduction layer.

[0095] The output of the batch normalization layer is added to the output of the SE module and then input into the feature fusion module;

[0096] The feature fusion module includes sequentially connected convolutional layers, activation function layers, and max pooling layers.

[0097] DCNN (dimension) ×H×W) and DR-SE (dimensions) The features extracted by the ×H×W module are concatenated along the channel dimension to generate fused features. + )×H×W;

[0098] Subsequently, a 1x1 convolutional layer was used to reduce the dimensionality by fusing the feature channels through the convolutional kernel. xH x W;

[0099] Using ReLU activation function introduces nonlinearity, making the model learn more complex feature representation;

[0100] Further compress the feature size to .

[0101] The fourth part is dynamic classification and result output, which is used for adaptive classification and fatigue assessment

[0102] The classification module includes global average pooling layer, fully connected layer, Sigmoid activation function, dynamic threshold classification layer;

[0103] The feature with spatial dimension H x W x C is compressed to vector by global average pooling layer GAP;

[0104] The vector V is mapped to the classification space by the fully connected layer, using the Sigmoid activation function, and optimized based on the improved BCELoss (binary cross-entropy loss) and Focal Loss (focal loss) composite loss function, with an output dimension of 3, corresponding to normal, fatigue, and severe fatigue respectively;

[0105] Since fatigue states may coexist (such as fatigue and severe fatigue), the Sigmoid activation function is used to independently activate each class, and the output of each class is independently mapped to the interval [0, 1], representing the confidence of the input sample belonging to class c, where is the original output of the cth neuron of the fully connected layer;

[0106] The improved BCE + Focal Loss composite loss function is used for training, which alleviates the class imbalance problem, and the loss function is

[0107]

[0108] where C is the number of classes, which is 3; is the true label of class c (0 or 1), is the class weight, ≥0 is the focus coefficient, which is used to suppress the loss contribution of easy-to-classify samples;

[0109] Finally, dynamic threshold classification is performed, and dynamic classification thresholds are set according to the distribution of training data:

[0110] Normal gait: Pnormal ≥ 0.5 and Pfatigue < 0.4 and Psevere fatigue < 0.3;

[0111] Fatigue gait: Pfatigue≥0.4 and Psevere fatigue<0.3;

[0112] Severe fatigue: Psevere fatigue≥0.3.

[0113] Stage 4-4 result output and application

[0114] After dynamic threshold classification, the recognition result is transmitted through the Bluetooth module, so that the data representing the fatigue state can be labeled.

[0115] A specific embodiment is given below.

[0116] The inertial sensor model is MPU6050, the output precision is 0.01°, the measurement range is ±180°, and the volume is 4cm×4cm×0.9mm (QFN-24 package). The sensor internally integrates a voltage stabilizing circuit and a temperature compensation module, and is compatible with 3.3V embedded systems for power supply. The data is transmitted through an SPI digital interface, and the acceleration and angular velocity signals are collected in real time.

[0117] The pressure sensor model is IMS004-C20B, which uses a six-channel thin-film piezoresistive sensor array. Each channel is configured with an independent high-precision differential amplifier circuit (gain adjustable range 10-1000) to convert resistance changes into linear voltage signals. The output range is 0-3.3V (full scale 200N, sensitivity 15mV / N), which is compatible with 12-bit ADC input (resolution 0.8mV) of the main controller, and is compatible with 3.3V-5V embedded systems for power supply.

[0118] The inertial sensor and the force sensor cooperate to complete the plantar multi-modal data acquisition. The inertial sensor directly connects to the main controller through a digital interface to collect acceleration and angular velocity signals in real time. The pressure sensor signal is input into the ADC module of the main controller after impedance matching and signal conditioning by a four-channel low-noise instrumentation amplifier (model INA128) for analog-to-digital conversion. The collected data is transmitted in real time to the terminal device through the Bluetooth module.

[0119] The data acquisition module uses STM32F103ZET6 as the main controller, which integrates an ARM Cortex-M3 core, supports multiple UART serial ports (baud rate 115200bps) and 12-bit ADC (sampling rate 1MHz), and builds a low-latency, high-precision data acquisition platform.

[0120] The Bluetooth module uses HC-05 (Bluetooth 2.1+EDR protocol) with a working voltage of 3.3V-5V, which is connected to the main controller through a UART interface to realize wireless data transmission and remote monitoring. Its effective transmission distance is 10 meters (in an obstacle-free environment), which meets the real-time data transmission requirements.

[0121] EMG muscle electricity sensor is pasted on calf gastrocnemius and soleus, and is used for monitoring muscle activity signal in real time, and provides key information of muscle function and motion track under fatigue state.

[0122] Pressure sensor and inertial sensor are embedded in foot pad or pasted on instep, and real-time monitoring of lower limb muscle fatigue degree, dynamic motion characteristics and mechanical change of user in walking process is realized.

[0123] The application also relates to a computer readable storage medium, which stores a fatigue gait pattern recognition program based on a multi-modal sensor.

[0124] The application also relates to a computer device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor.

[0125] Although the preferred embodiments of the present application have been described, those skilled in the art can make appropriate changes and modifications to the embodiments after understanding the core creative concept of the present application. Therefore, the appended claims are intended to cover the preferred embodiments of the present application and all reasonable changes and modifications thereof.

[0126] Obviously, those skilled in the art can make various modifications and variations to the present application, as long as these modifications and variations do not deviate from the spirit and scope of the present application. If these modifications and variations belong to the scope of the claims of the present application and equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A multi-modal sensor based fatigue gait pattern recognition method, characterized by: The method is based on multi-modal sensor to collect multi-modal sensor data, and after preprocessing, gait segmentation is performed, based on the preprocessed multi-modal sensor data, the gait phases are sequentially divided according to the gait cycle, and each dimension of the multi-modal sensor data in any gait phase is normalized to obtain multi-dimensional features for representing gait; Obtain the data sample after gait segmentation, including multi-modal sensor data and corresponding data features; An improved feature fusion model is constructed, and the data sample is trained until the model is stable; the improved feature fusion model includes a deep convolutional neural network, an improved DR-SE module, a feature fusion module, and a classification module connected in sequence; the deep convolutional neural network includes parallel convolutional layers corresponding to different modal data, all convolutional layers connected to an average pooling layer and a batch normalization layer; The improved DR-SE module includes a dimension reduction layer and an SE module connected in sequence; The output end of the batch normalization layer is connected to the input end of the dimension reduction layer; The output of the batch normalization layer and the output of the SE module are added and input into the feature fusion module; Collect multi-modal sensor data, input the preprocessed model after preprocessing, and output fatigue gait pattern recognition results.

2. The multi-modal sensor based fatigue gait pattern recognition method as claimed in claim 1, wherein: The multi-modal sensor data includes muscle movement data, plantar mechanics data, and foot movement data.

3. The multi-modal sensor based fatigue gait pattern recognition method as claimed in claim 2, wherein: The muscle movement data, plantar mechanics data, and foot movement data are processed by a parallel filter circuit of a low-pass filter and a high-pass filter, the plantar mechanics data and the foot movement data are filtered by the low-pass filter, and the muscle movement data is filtered by the high-pass filter; Align the data processed by the parallel filter circuit to obtain the preprocessed multi-modal sensor data.

4. The multi-modal sensor based fatigue gait pattern recognition method as claimed in claim 1, wherein: The multi-dimensional features include time domain features, gait irregularity features, gait symmetry features, statistical features, and nonlinear features.

5. The multi-modal sensor based fatigue gait pattern recognition method as claimed in claim 1, wherein: The feature fusion module includes a convolutional layer, an activation function layer, and a max pooling layer connected in sequence; The classification module includes a global average pooling layer, a fully connected layer, a Sigmoid activation function, and a dynamic threshold classification layer.

6. A multi-modal sensor based fatigue gait pattern recognition device, characterized by: The recognition device includes: 2 sets of multi-modal sensors for collecting multi-modal sensor data; A synchronous transceiver device for transmitting the corresponding 2 sets of multi-modal sensor data; A controller for obtaining multi-modal sensor data transmitted by the synchronous transceiver device and performing fatigue gait pattern recognition based on the multi-modal sensor-based fatigue gait pattern recognition method of any one of claims 1-5.

7. A multi-modal sensor based fatigue gait pattern recognition device as claimed in claim 6, wherein: Any of the multi-modal sensors includes a pressure sensor, an inertial sensor, and a plurality of electromyography sensors, and a data acquisition module and a charging module are provided in cooperation with the pressure sensor, the inertial sensor, and the plurality of electromyography sensors.

Images