A human fatigue state evaluation method, system, device and storage medium based on multi-source sensor fusion

Abstract

This invention discloses a method, system, device, and storage medium for assessing human fatigue state based on multi-source sensor fusion, relating to the field of intelligent sensing wearable technology. It includes acquiring plantar pressure signals and limb acceleration signals during human movement, and combining them to obtain a multi-channel time-series signal. Using the multi-channel time-series signal as input, a fatigue level is output based on a human fatigue state assessment model. This invention achieves collaborative perception and deep fusion of multi-source information, constructs a subjective-objective collaborative assessment system, and completes a refined assessment of fatigue levels.

Description

Technical Field

[0001] This invention relates to a method, system, device, and storage medium for assessing human fatigue state based on multi-source sensor fusion, belonging to the field of intelligent sensing wearable technology. Background Technology

[0002] Human fatigue assessment has significant applications in sports training, rehabilitation medicine, and other fields. Existing fatigue assessment methods based on wearable sensors have the following main shortcomings: (1) Disconnect between subjective and objective assessment: Subjective assessment methods rely on individual self-perception, are easily affected by psychological factors, and are difficult to quantify accurately; objective assessment methods often involve invasive sampling or rely on sophisticated laboratory equipment, making real-time monitoring difficult. Existing methods fail to effectively integrate subjective perception with objective physiological signals and lack a coordinated subjective and objective assessment system.

[0003] (2) Incomplete information from a single sensor: Although the inertial measurement unit can capture kinematic information, it is not sensitive to dynamic characteristics such as the force on the sole of the foot; although the plantar pressure sensor can reflect the force distribution, it lacks the perception of limb movement posture. A single sensor is difficult to fully depict the multidimensional physiological changes under fatigue.

[0004] (3) Insufficient feature fusion depth: Existing methods mostly use simple feature splicing or statistical feature extraction, which fail to make full use of the deep correlation information of multi-source sensor data in time and space dimensions.

[0005] (4) The granularity of fatigue classification is too coarse: Most methods only distinguish between the two-level states of "fatigue / non-fatigue", which cannot quantify the degree of fatigue in a refined way and make it difficult to guide users to make differentiated exercise adjustments.

[0006] Therefore, there is an urgent need for a fatigue state assessment system and method that can integrate multi-source sensor information, construct a subjective and objective collaborative assessment system, and achieve refined fatigue grading. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system, device and storage medium for assessing human fatigue state based on multi-source sensor fusion, so as to realize the collaborative perception and deep fusion of multi-source information, construct a subjective and objective collaborative assessment system, and complete the refined assessment of fatigue level.

[0008] To achieve the above objectives, the present invention is implemented using the following technical solution: In a first aspect, the present invention provides a method for assessing human fatigue state based on multi-source sensor fusion, comprising: The plantar pressure signal and limb acceleration signal during human movement are acquired and combined to obtain a multi-channel time-series signal. Using multi-channel time-series signals as input, the fatigue level is obtained by outputting a human fatigue state assessment model. The human fatigue state assessment model includes a local feature extraction unit, a feature enhancement unit, a multi-scale feature fusion unit, and a classifier connected in sequence. The local feature extraction unit is used to extract local spatiotemporal features from multi-channel time-series signals; The feature enhancement unit is used to enhance local spatiotemporal features in the channel dimension and spatial dimension respectively through a parallel dual-branch attention mechanism to obtain enhanced local spatiotemporal features; The multi-scale feature fusion unit is used to extract gait features of different scales from enhanced local spatiotemporal features and fuse them to obtain a fused feature vector; The classifier is used to output the corresponding fatigue level based on the fused feature vector.

[0009] Furthermore, the plantar pressure signal of the human foot includes plantar pressure signals of multiple anatomical regions of the human foot, including the medial side of the heel, the lateral side of the heel, the medial arch, the first metatarsal bone, the area between the second and third metatarsal bones, the area between the fourth and fifth metatarsal bones, the phalangeal region, and the lateral side of the midfoot.

[0010] Furthermore, the human foot pressure signal is acquired by a multi-channel flexible piezoresistive sensor array integrated in the insole, the multi-channel flexible piezoresistive sensor array including multiple channels corresponding to each anatomical region; the limb acceleration signal during human movement is acquired by an inertial measurement unit.

[0011] Furthermore, the method for acquiring the multi-channel timing signal includes: The preprocessed signal is obtained by filtering and denoising the human foot pressure signal and the limb acceleration signal during human movement. Gait periodicity segmentation is performed on the preprocessed signal to obtain a gait periodic sequence; A multi-channel timing signal is obtained by timing alignment of the gait periodic sequence.

[0012] Furthermore, the step of segmenting the preprocessed signal into gait cycles to obtain a gait cycle sequence includes: Multiple gait cycles are obtained by dividing the interval between the pressure peaks generated at adjacent heel strike moments into a complete gait cycle; The gait cycle sequence is obtained by unifying gait cycles of different lengths to the same length using cubic spline interpolation.

[0013] Furthermore, the feature enhancement unit includes a channel attention branch and a spatial attention branch; In the channel attention branch, the local spatiotemporal features are generated by one-dimensional convolution to form a first query feature, a first key feature, and a first value feature, respectively. The channel attention weights are determined based on the first query feature and the first key feature, and the first value feature is weighted based on the channel attention weights to obtain the channel attention features. In the spatial attention branch, the local spatiotemporal features are generated by one-dimensional convolution to generate a second query feature, a second key feature, and a second value feature, respectively. The spatial attention weight is determined based on the second query feature and the second key feature, and the first value feature is weighted based on the spatial attention weight to obtain the spatial attention feature. The channel attention features and spatial attention features are sequentially processed by global average pooling, convolution, and normalization to obtain the channel fusion weights and spatial fusion weights, respectively. Channel attention features are weighted based on channel fusion weights, spatial attention features are weighted based on spatial fusion weights, and the weighted channel attention features and the weighted spatial attention features are fused to obtain enhanced local spatiotemporal features.

[0014] Furthermore, the multi-scale feature fusion unit includes four parallel branches, namely a global average pooling branch and three branches composed of dilated convolutions, and the dilation rate of the dilated convolutions in each branch is set to 1, 3 and 6 respectively. The branch corresponding to the dilation rate of the dilated convolution is 1, which is used to extract local detail features within a single gait cycle; the branch corresponding to the dilation rate of the dilated convolution is 3, which is used to extract regional context features between the metatarsal region and the arch region; the branch corresponding to the dilation rate of the dilated convolution is 6, which is used to extract global fatigue progressive features across multiple consecutive gait cycles; and the global average pooling branch is used to extract the overall pressure distribution features across all gait cycles. The local detail features, regional context features, global fatigue progressive features, and overall pressure distribution features are fused to obtain a fused feature vector.

[0015] In a second aspect, the present invention also provides a human fatigue state assessment system based on multi-source sensor fusion, used to implement the human fatigue state assessment method based on multi-source sensor fusion as described in any one of the first aspects, comprising: The multi-channel timing signal acquisition module is configured to acquire human plantar pressure signals and limb acceleration signals during human movement, and combine them to obtain multi-channel timing signals. The fatigue level assessment module is configured to take multi-channel time-series signals as input and output fatigue level based on the human fatigue state assessment model.

[0016] Thirdly, the present invention also provides a computer device, comprising: Memory, used to store computer programs / instructions; A processor for executing the computer program / instructions to implement the human fatigue state assessment method based on multi-source sensor fusion as described in any of the first aspects.

[0017] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the human fatigue state assessment method based on multi-source sensor fusion as described in any of the first aspects.

[0018] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: This invention integrates physically complementary information from plantar pressure signals (dynamic information) and limb acceleration signals (kinematic information), enabling a more comprehensive characterization of gait changes under fatigue. Then, through local feature extraction, bi-branch attention enhancement, and multi-scale feature fusion, it fully explores the deep correlation information between multi-source signals in the temporal and spatial dimensions, improving the accuracy of fatigue grading and overcoming the limitations of existing single subjective fatigue scales. The multi-channel flexible piezoresistive sensor array of the present invention is integrated into the insole, which does not affect the user's normal movement and can monitor continuously for a long time. This makes it play an important role in monitoring the training load of athletes, formulating exercise prescriptions for rehabilitation patients, and assessing the daily activity ability of the elderly, and has broad application prospects. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall structure of a human fatigue state assessment system based on multi-source sensor fusion in one embodiment of the present invention. Figure 2 This is a schematic diagram of the layout of a multi-channel flexible piezoresistive sensor array on the sole of the foot in a human fatigue state assessment method based on multi-source sensor fusion in one embodiment of the present invention. Figure 3 This is a schematic diagram of multi-channel time-series signal acquisition for a human fatigue state assessment method based on multi-source sensor fusion in one embodiment of the present invention. Figure 4 This is a schematic diagram of the feature enhancement unit of the human fatigue state assessment model in a human fatigue state assessment method based on multi-source sensor fusion in one embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of the multi-scale feature fusion unit in the human fatigue state assessment model of the human fatigue state assessment method based on multi-source sensor fusion in one embodiment of the present invention. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0021] Example 1:

[0022] like Figure 1 As shown, this embodiment of the invention provides a method for assessing human fatigue state based on multi-source sensor fusion, including: The test acquires plantar pressure signals and limb acceleration signals during human movement. In this embodiment, the signals are measured when the subject is performing an incremental load running test on a treadmill.

[0023] Combination Figure 2 and Figure 3 The plantar pressure signal of the human foot includes plantar pressure signals from multiple anatomical regions of the foot. In this embodiment, the anatomical regions include the medial heel, lateral heel, medial arch, first metatarsal, the area between the second and third metatarsals, the area between the fourth and fifth metatarsals, the phalangeal region, and the lateral midfoot. The plantar pressure signal is acquired through a multi-channel flexible piezoresistive sensor array. In some embodiments, the multi-channel flexible piezoresistive sensor array is integrated into the insole, which includes multiple channels corresponding to each anatomical region.

[0024] The limb acceleration signals during human movement are acquired by an inertial measurement unit (IMU). In this embodiment, the IMU (Xsens DOT) is worn on the subject's waist, and the sampling frequency is set to 100Hz.

[0025] The human plantar pressure signal is amplified by the analog front end and then converted into a digital plantar pressure signal by the digital-to-analog converter. The digital plantar pressure signal and the limb acceleration signal during human movement are read separately and then wirelessly transmitted to the host computer after time synchronization.

[0026] Human plantar pressure signals and limb acceleration signals during human movement are filtered and denoised using a low-pass filter before gait cycle segmentation. Gait cycle segmentation is based on peak detection of the plantar pressure signal, specifically using the pressure peak at heel strike as the cycle start marker, and the interval between pressure peaks generated at adjacent heel strikes as a complete gait cycle. Cubic spline interpolation is then used to unify gait cycles of different lengths to the same length. Specifically, this includes: A target length L is set, and the gait cycle interval [0,T] is mapped to the normalized interval [0,1]. L resampling points are selected at equal intervals. Next, a cubic spline interpolation function satisfying the interpolation conditions and continuous first and second derivatives is constructed between adjacent original sampling points, and the piecewise polynomial coefficients are solved using natural boundary conditions. Finally, the interpolation function values ​​are calculated at each resampling point to obtain the gait cycle sequence. The gait cycle sequence is then time-aligned to obtain a multi-channel time-series signal.

[0027] Using multi-channel time-series signals as input, the fatigue level is obtained based on the human fatigue state assessment model. In this embodiment, the human fatigue state assessment model includes a local feature extraction unit, a feature enhancement unit, a multi-scale feature fusion unit, and a classifier connected in sequence. Specifically: The local feature extraction unit uses a two-dimensional convolutional neural network to extract local spatiotemporal features from multi-channel time-series signals.

[0028] Combination Figure 4 The feature enhancement unit enhances local spatiotemporal features by using a parallel dual-branch attention mechanism to enhance local spatiotemporal features in both the channel and spatial dimensions. The structure of the feature enhancement unit includes a channel attention branch and a spatial attention branch.

[0029] In the channel attention branch, local spatiotemporal features are convolved in one dimension to generate a first query feature, a first key feature, and a first value feature, respectively. Channel attention weights are determined based on the first query feature and the first key feature, and the first value feature is weighted based on the channel attention weights to obtain the channel attention feature. In the spatial attention branch, local spatiotemporal features are convolved in one dimension to generate a second query feature, a second key feature, and a second value feature, respectively. Spatial attention weights are determined based on the second query feature and the second key feature, and the first value feature is weighted based on the spatial attention weights to obtain the spatial attention feature.

[0030] Channel attention features and spatial attention features are processed sequentially by global average pooling, convolution, and normalization to obtain channel fusion weights and spatial fusion weights, respectively. Channel attention features are weighted based on channel fusion weights, and spatial attention features are weighted based on spatial fusion weights. The weighted channel attention features and the weighted spatial attention features are then fused to obtain enhanced local spatiotemporal features.

[0031] In this embodiment, one-dimensional convolution is used instead of matrix multiplication to generate the query, key, and value of the attention mechanism, which effectively reduces the computational complexity.

[0032] Combination Figure 5The multi-scale feature fusion unit comprises four parallel branches, three of which are composed of dilated convolutions with dilation rates of 1, 3, and 6, respectively. The branch with dilation rate r=1 extracts local detail features within a single gait cycle; the branch with dilation rate r=3 extracts regional contextual features between the metatarsal and arch regions; and the branch with dilation rate r=6 extracts global fatigue progression features across multiple consecutive gait cycles. A global average pooling branch extracts the overall stress distribution features across all gait cycles. The local detail features, regional contextual features, global fatigue progression features, and overall stress distribution features are fused to obtain a fused feature vector, which is then input into the classifier.

[0033] In the pre-training phase, the classifier first collects a large number of multi-source sensor signals from subjects under different fatigue states. After signal preprocessing, gait cycle segmentation, feature extraction and fusion, a fused feature vector sample with fatigue level labels is obtained, which is used to construct the training dataset.

[0034] The classifier takes the dataset as input and adopts a supervised learning approach. It uses the fused feature vector as input variable and the corresponding fatigue level label as output target. It calculates the error between the prediction result and the true label through forward propagation and uses the backpropagation algorithm to propagate the error back layer by layer. Combined with the optimizer, it iteratively updates the weight parameters and bias terms inside the classifier.

[0035] During training, the classifier continuously fits the complex nonlinear mapping relationship between input features and output labels by minimizing the loss function, gradually aligning the sample distribution in the feature space with the class boundaries in the fatigue level label space. After multiple rounds of iterative training until the loss function converges, this mapping relationship is solidified and stored in the form of parameter matrices and bias vectors of each layer within the classifier, thus completing the pre-construction of the mapping relationship.

[0036] In practical applications, after the fused feature vector is input into the classifier, it sequentially undergoes linear transformations and nonlinear activation functions in each fully connected layer to extract higher-level discriminative features. The final layer uses a Softmax function to normalize the output value to a probability distribution for each fatigue level. Each probability value is between 0 and 1, and their sum is 1, representing the likelihood of the input sample belonging to each fatigue level. The final classifier uses the maximum probability criterion, selecting the fatigue level with the highest probability value as the final discrimination result and outputting it. The entire discrimination process requires only one forward propagation, making it computationally efficient and meeting the application requirements of real-time fatigue monitoring.

[0037] In this embodiment, the fatigue level is divided into three levels, specifically defined as follows: Mild fatigue: Slightly increased heel impact, slightly decreased cadence, and basically normal arch support; Moderate fatigue: Decreased arch support, reduced peak force in the metatarsal region, and increased gait variability; Severe fatigue: Disordered distribution of plantar pressure and unstable gait rhythm.

[0038] Finally, the fatigue level is transmitted to a smartphone via Bluetooth Low Energy for visualization. For example, when moderate or higher fatigue levels are detected, a warning signal is sent to the smartphone via Bluetooth, prompting the user to reduce exercise intensity or stop exercising.

[0039] In some embodiments, this method can also be used for post-knee rehabilitation training. The patient undergoes flat-ground walking training under the guidance of a rehabilitation physician. The training duration is determined based on the patient's recovery stage, typically ranging from 20 to 40 minutes. During the training, plantar pressure signals and triaxial acceleration signals of the limbs during movement are collected.

[0040] In knee rehabilitation training, the principle of fatigue assessment is that as training time increases, the quadriceps and triceps surae muscles of the patient gradually become fatigued. In a state of fatigue, detectable changes occur in the patient's gait characteristics: Decreased arch support: The peak pressure in the medial arch area (channel 3) decreased significantly compared to the beginning of training; The force exerted during the push-off phase decreases: the peak pressure in the first metatarsal and the second to third metatarsal regions (channels 4 and 5) decreases; Decreased gait symmetry: Increased differences in left and right gait parameters (such as ground contact time and peak pressure); Increased oscillation phase variability: The peak-to-peak fluctuation of the acceleration signal increases.

[0041] The above changes are captured by a human fatigue assessment model, which outputs a mild, moderate, or severe fatigue level. When a moderate fatigue level is output, a prompt is sent to the rehabilitation physician, advising the patient to rest for 2-3 minutes. When a severe fatigue level is output, an audible and visual alarm is triggered, forcibly stopping training to prevent compensatory gait changes and secondary injuries caused by over-fatigue.

[0042] Example 2:

[0043] This embodiment also provides a human fatigue state assessment system based on multi-source sensor fusion, used to implement the human fatigue state assessment method based on multi-source sensor fusion as described in Embodiment 1, which includes: The multi-channel timing signal acquisition module is configured to acquire human plantar pressure signals and limb acceleration signals during human movement, and combine them to obtain multi-channel timing signals. The fatigue level assessment module is configured to take multi-channel time-series signals as input and output fatigue level based on the human fatigue state assessment model.

[0044] Example 3:

[0045] This embodiment also provides a computer device, including: Memory, used to store computer programs / instructions; A processor is used to execute the computer program / instructions for the human fatigue state assessment method based on multi-source sensor fusion as described in Example 1.

[0046] Example 4:

[0047] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the human fatigue state assessment method based on multi-source sensor fusion as described in Embodiment 1.

[0048] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for assessing human fatigue state based on multi-source sensor fusion, characterized in that, include: The plantar pressure signal and limb acceleration signal during human movement are acquired and combined to obtain a multi-channel time-series signal. Using multi-channel time-series signals as input, fatigue levels are obtained by outputting a human fatigue state assessment model. The human fatigue state assessment model includes a local feature extraction unit, a feature enhancement unit, a multi-scale feature fusion unit, and a classifier connected in sequence. The local feature extraction unit is used to extract local spatiotemporal features from multi-channel time-series signals; The feature enhancement unit is used to enhance local spatiotemporal features in the channel dimension and spatial dimension respectively through a parallel dual-branch attention mechanism to obtain enhanced local spatiotemporal features; The multi-scale feature fusion unit is used to extract gait features of different scales from enhanced local spatiotemporal features and fuse them to obtain a fused feature vector; The classifier is used to output the corresponding fatigue level based on the fused feature vector.

2. The method for assessing human fatigue state based on multi-source sensor fusion according to claim 1, characterized in that, The plantar pressure signal of the human foot includes plantar pressure signals of multiple anatomical regions of the human foot, including the medial side of the heel, the lateral side of the heel, the medial arch, the first metatarsal bone, the area between the second and third metatarsal bones, the area between the fourth and fifth metatarsal bones, the phalangeal region, and the lateral side of the midfoot.

3. The method for assessing human fatigue state based on multi-source sensor fusion according to claim 2, characterized in that, The human foot pressure signal is acquired by a multi-channel flexible piezoresistive sensor array integrated in the insole, which includes multiple channels corresponding to various anatomical regions; the limb acceleration signal during human movement is acquired by an inertial measurement unit.

4. The method for assessing human fatigue state based on multi-source sensor fusion according to claim 1, characterized in that, The method for acquiring the multi-channel timing signal includes: The preprocessed signal is obtained by filtering and denoising the human foot pressure signal and the limb acceleration signal during human movement. Gait periodicity segmentation is performed on the preprocessed signal to obtain a gait periodic sequence; A multi-channel timing signal is obtained by timing alignment of the gait periodic sequence.

5. The method for assessing human fatigue state based on multi-source sensor fusion according to claim 4, characterized in that, The step of performing gait period segmentation on the preprocessed signal to obtain a gait period sequence includes: Multiple gait cycles are obtained by dividing the interval between the pressure peaks generated at adjacent heel strike moments into a complete gait cycle; The gait cycle sequence is obtained by unifying gait cycles of different lengths to the same length using cubic spline interpolation.

6. The method for assessing human fatigue state based on multi-source sensor fusion according to claim 1, characterized in that, The feature enhancement unit includes a channel attention branch and a spatial attention branch; In the channel attention branch, the local spatiotemporal features are generated by one-dimensional convolution to form a first query feature, a first key feature, and a first value feature, respectively. The channel attention weights are determined based on the first query feature and the first key feature, and the first value feature is weighted based on the channel attention weights to obtain the channel attention features. In the spatial attention branch, the local spatiotemporal features are generated by one-dimensional convolution to generate a second query feature, a second key feature, and a second value feature, respectively. The spatial attention weight is determined based on the second query feature and the second key feature, and the first value feature is weighted based on the spatial attention weight to obtain the spatial attention feature. The channel attention features and spatial attention features are sequentially processed by global average pooling, convolution, and normalization to obtain the channel fusion weights and spatial fusion weights, respectively. Channel attention features are weighted based on channel fusion weights, spatial attention features are weighted based on spatial fusion weights, and the weighted channel attention features and the weighted spatial attention features are fused to obtain enhanced local spatiotemporal features.

7. The method for assessing human fatigue state based on multi-source sensor fusion according to claim 1, characterized in that, The multi-scale feature fusion unit includes four parallel branches, namely a global average pooling branch and three branches composed of dilated convolutions, and the dilation rates of the dilated convolutions in each branch are set to 1, 3 and 6, respectively. The branch corresponding to the dilation rate of the dilated convolution is 1, which is used to extract local detail features within a single gait cycle; the branch corresponding to the dilation rate of the dilated convolution is 3, which is used to extract regional context features between the metatarsal region and the arch region; the branch corresponding to the dilation rate of the dilated convolution is 6, which is used to extract global fatigue progressive features across multiple consecutive gait cycles; and the global average pooling branch is used to extract the overall pressure distribution features across all gait cycles. The local detail features, regional context features, global fatigue progressive features, and overall pressure distribution features are fused to obtain a fused feature vector.

8. A human fatigue state assessment system based on multi-source sensor fusion, characterized in that, The method for assessing human fatigue state based on multi-source sensor fusion as described in any one of claims 1 to 7 includes: The multi-channel timing signal acquisition module is configured to acquire human plantar pressure signals and limb acceleration signals during human movement, and combine them to obtain multi-channel timing signals. The fatigue level assessment module is configured to take multi-channel time-series signals as input and output fatigue level based on the human fatigue state assessment model.

9. A computer device, characterized in that, include: Memory, used to store computer programs / instructions; A processor for executing the computer program / instructions to implement the human fatigue state assessment method based on multi-source sensor fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the human fatigue state assessment method based on multi-source sensor fusion as described in any one of claims 1 to 7.

Images