Objective physical fatigue assessment method based on wearable sensing information fusion

Abstract

The invention discloses an objective physical fatigue assessment method based on wearable sensing information fusion. The method comprises the following steps: acquiring an IMU signal and an ECG signal of a target by using a wearable device; the IMU signal and the ECG signal are preprocessed and then input into a space-time fusion double-branch multi-mode network, a fatigue grade classification result is obtained, and the space-time fusion double-branch multi-mode network comprises a multi-mode feature extraction module, a space-time double-branch feature fusion module, a multi-mode feature fusion module and a fatigue grade classification module. The multi-modal feature extraction module is used for extracting spatial features and time sequence features; the space-time double-branch feature fusion module is used for carrying out interactive fusion on the time sequence features and the space features of the same type of sensors to obtain space-time feature joint representation; the multi-modal feature fusion module is used for capturing multi-modal feature joint representation of each heterogeneous sensor signal; and the fatigue grade classification module obtains a fatigue grade classification result. According to the invention, the accuracy of physical fatigue classification is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer science and technology, and more specifically, to an objective physical fatigue assessment method based on wearable sensing information fusion. Background Art

[0002] Physical fatigue refers to the decline in the working ability of muscles after long-term or high-intensity physical labor. It is a protective physiological response to prevent muscles and other systems from being over-fatigued or damaged. Medically, physical fatigue is characterized by a decline in muscle strength, coordination, and endurance, usually accompanied by a subjective sense of tiredness. Short-term fatigue may lead to physical discomfort and a decline in control ability, while long-term fatigue may trigger musculoskeletal and cardiovascular diseases. However, physical fatigue is essentially subjective, varying from person to person and fluctuating over time. Attributes such as subjectivity, individual differences, and time-variability pose great challenges to the objective quantification and assessment of physical fatigue. Current fatigue assessment methods can be divided into subjective and objective categories. Subjective methods rely on self-reported fatigue levels, such as the "Rating of Perceived Exertion" (RPE) scale widely used in sports science. These assessments are often influenced by personal perception, resulting in inconsistent results and may not accurately reflect the true fatigue level. Objective methods refer to the scientific assessment of an individual's physical fatigue level using measurable physiological data, eliminating the subjective bias inherent in self-reporting, and usually relying on quantifiable indicators collected by wearable sensors to track fatigue in real time. However, existing objective fatigue assessment methods still need to be improved in terms of signal input and model design.

[0003] Scientific exercise refers to a method of exercise that is designed according to an individual's physical condition and adjusted in real time to the exercise load, aiming to improve exercise effects, avoid injuries, enhance physical functions, and promote health. During exercise, due to physiological changes such as energy reserve consumption, accumulation of metabolic by-products (such as lactic acid, etc.), muscle damage, and nervous system stress, athletes will experience different degrees of fatigue. Physical fatigue caused by exercise is an important indicator reflecting the adaptive limit of the body to the exercise load. Moderate fatigue is a sign of training effect and helps the body to adapt and progress. However, continuous fatigue or over-fatigue is also a warning of overtraining and must be addressed through scientific recovery strategies, reasonable training plans, and adequate nutrition and rest to ensure that athletes maintain their best state. Therefore, the objective assessment of physical fatigue during exercise is an important indicator to ensure training safety and improve sports performance.

[0004] In recent years, wearable sensors and intelligent algorithms have made remarkable progress. Wearable devices such as smart bracelets, wearable sports electrocardiogram vests, and smart sports shoes have been increasingly accepted and applied by the public. The multi-dimensional motion and physiological information of the human body sensed by wearable sensors bring new solutions to fields such as exercise load monitoring, fatigue assessment, and health analysis. Exercise fatigue is a physiological response of the body to exercise load, and various physiological indicators of the human body change with the degree of fatigue. How to establish an objective physical fatigue assessment based on the response of physiological indicators to the degree of body fatigue is a key scientific problem that needs to be solved.

[0005] In the existing technology, the patent application CN202010660308.4 proposed a fatigue prediction method based on the Bayesian optimized XGBoost algorithm. This method collected various physiological indicators (heart rate, blood oxygen saturation, blood pressure, etc.) and eliminated abnormal data, and used the Bayesian optimized XGBoost algorithm to achieve the prediction of the fatigue value of the subject. However, this method fails to fully extract the features of various modal data, does not perform sufficient fusion of multi-modal features, and the accuracy of objective fatigue assessment is low. The patent application CN202010401609.5 provided a method and device for evaluating exercise fatigue. This method considered the differences in different altitude regions, but it lacked scientificity to convert exercise load using the exercise heart rate for several consecutive days, and it was only suitable for a rough estimate of physical fatigue. The patent application CN201810329785.5 proposed a method for monitoring exercise-induced fatigue based on multiple physiological parameters. Using the collected physiological signals as features, a fatigue prediction model based on a multi-layer feedforward neural network was established. However, this method only extracted features such as mean, standard deviation, difference between maximum and minimum values, mean of the first-order difference signal, and standard deviation of the first-order difference signal in the time domain, and did not consider the features in the frequency domain and non-linear domain.

[0006] In summary, the existing physical fatigue assessment methods based on manually crafted features require a cumbersome and complex feature selection process, have a limited number of features that can be extracted, and require rich expert experience. Moreover, the process of extracting manually crafted features is usually independent of the subsequent classifier, which may lead to poor performance. In addition, the existing physical fatigue assessment methods ignore the extraction of spatial information (i.e., complex patterns across signal channels) from wearable sensor signals. And in terms of feature fusion, single-modal methods rely heavily on single-modal sensor signals, while multi-modal methods lack complementary feature fusion between heterogeneous modalities and only perform simple feature splicing, affecting the accuracy of fatigue assessment. Summary of the Invention

[0007] The object of the present invention is to overcome the defects of the above-mentioned existing technologies and provide an objective physical fatigue assessment method based on the fusion of wearable sensing information. The method includes the following steps:

[0008] Collect the IMU signal and ECG signal of the target using a wearable device, where the wearable device includes an IMU sensor and an electrocardiogram detection sensor;

[0009] Preprocess the IMU signal and ECG signal, and then input them into a trained spatio-temporal fusion double-branch multimodal network to obtain the fatigue level classification result of the target;

[0010] Among them, the spatio-temporal fusion double-branch multimodal network includes a multimodal feature extraction module, a spatio-temporal double-branch feature fusion module, a multimodal feature fusion module, and a fatigue level classification module. The multimodal feature extraction module includes a spatial branch and a temporal branch. The spatial branch is used to extract the correlation relationship between different channels, and the temporal branch is used to capture the time variation information of the input signal; the spatio-temporal double-branch feature fusion module is used to interactively fuse the signal temporal features and spatial features from sensors of the same category to obtain a joint spatio-temporal feature representation; the multimodal feature fusion module is used to capture the joint multimodal feature representation of heterogeneous sensor signals; the fatigue level classification module is used to obtain the fatigue level classification result based on the joint multimodal feature representation.

[0011] Compared with the prior art, the advantages of the present invention are that a spatio-temporal fusion double-branch multimodal network for objective physical strength assessment is designed, and multi-source wearable sensor data is used for physical fatigue assessment without manual feature screening. The designed network model includes a spatial branch for extracting complex patterns across sensor channels and a temporal branch for extracting time series information independent of sensor channels. Furthermore, spatio-temporal branch fusion and multimodal feature fusion are gradually carried out in the intermediate layer. In addition, based on the mutual attention of Transformer Tokens, the model can capture the long-term dependence information between each block in the signal, improving the classification accuracy of objective physical fatigue assessment.

[0012] Through the following detailed description of the exemplary embodiments of the present invention with reference to the accompanying drawings, other features and advantages of the present invention will become clear. Brief Description of the Drawings

[0013] The accompanying drawings incorporated in the specification and constituting a part of the specification illustrate embodiments of the present invention and, together with the description, are used to explain the principles of the present invention.

[0014] Figure 1 is a flowchart of an objective physical fatigue assessment method based on wearable sensing information fusion according to an embodiment of the present invention;

[0015] Figure 2 is a flowchart of data preprocessing according to an embodiment of the present invention;

[0016] Figure 3 Schematic diagram of a spatio-temporal fusion dual-branch multi-modal network according to an embodiment of the present invention;

[0017] Figure 4 Schematic diagram of an ROC curve obtained by different methods according to an embodiment of the present invention. Detailed implementation manners

[0018] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present invention.

[0019] The following description of at least one exemplary embodiment is merely illustrative in nature and in no way limits the present invention or its application or use.

[0020] Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and devices should be regarded as part of the specification.

[0021] In all the examples shown and discussed herein, any specific values should be construed as merely exemplary and not as limitations. Thus, other examples of the exemplary embodiments may have different values.

[0022] It should be noted that: like reference numerals and letters denote like items in the following drawings, and thus, once an item is defined in one drawing, further discussion thereof is not required in subsequent drawings.

[0023] Sports physical fatigue is a physiological response of the body to exercise load, and various physiological indexes of the human body will change with the change of the body's fatigue state. The present invention aims to establish an objective evaluation method for sports physical fatigue based on the physiological response of sports physical fatigue. For example, an objective and non-invasive quantitative evaluation method for exercise fatigue is designed based on physiological information such as electrocardiogram continuously collected by wearable devices in real time and motion information such as acceleration and angular velocity. By integrating multi-source wearable sensing information, it aims to comprehensively evaluate the fatigue state of athletes from the multi-dimensional perspectives of cardiac activity and body movement.

[0024] Generally speaking, the present invention mainly aims at quantifying and evaluating the physical fatigue degree of athletes in the sports health scenario, and uses an electrocardiogram monitoring device and an inertial measurement unit to collect the physiological information and motion information of the athlete during running respectively. Through steps such as collecting, preprocessing, feature extraction, spatio-temporal dual-branch feature fusion, multi-modal feature fusion, and physical fatigue degree classification modeling of multi-source wearable sensing signals, the quantification and evaluation of exercise-induced fatigue are realized. The model proposed by the present invention can capture the spatial information and temporal information of the same type of sensors, and interactively fuse the multi-modal complementary information of different types of sensors. By fusing the complementary features of sensor data of different types, the accuracy of physical fatigue classification is improved.

[0025] Specifically, referring to Figure 1 as shown, the provided objective physical fatigue assessment method based on wearable sensing information fusion includes the following steps:

[0026] Step S110, using a wearable device to collect multi-source data.

[0027] For data collection, a data collection device using wearable sensors is used to objectively evaluate physical fatigue. The participant wears three wearable inertial sensors and a 12-lead electrocardiogram monitoring device to capture the physiological signals and motion signals of the training personnel when running on a running platform, while performing a preset modified Bruce incremental load experiment. The 12-lead electrocardiogram is attached to the chest according to the standard, and the sampling rate is 200 Hz. For example, the electrocardiogram device used is the CardioSoft cardiac test system of GE Healthcare Systems Information Technology Company. The three inertial sensors are worn on the left wrist, waist, and left calf respectively, and the sampling rate is 100 Hz. The inertial sensor used is the Shimmer3 IMU. The modified Bruce treadmill test procedure is shown in Table 1. The speed of the treadmill is increased every 5 minutes until the subject reaches a state of exhaustion from the initial relaxed state. The participant reports their subjective fatigue feeling using the Borg Rating of Perceived Exertion scale at the end of each exercise stage, as shown in Table 2. The coarse-grained fatigue level divides fatigue into three levels: no fatigue, moderate fatigue, and severe fatigue. The fine-grained fatigue level divides fatigue into five levels: no fatigue, mild fatigue, moderate fatigue, severe fatigue, and extreme fatigue.

[0028] Table 1: Modified Bruce incremental load experiment

[0029]

[0030] Table 2: Borg Rating of Perceived Exertion scale and two methods of fatigue degree classification

[0031]

[0032] Step S120: Preprocess the collected multi-source data and then construct a sample data set.

[0033] In one embodiment, to reduce noise and increase the dimension of information, the signal is preprocessed and the data is segmented into samples, as shown in Figure 2 For IMU data, in the signal filtering stage, a Butterworth low-pass filter with a cut-off frequency of 10 Hz and a Wiener filter with a window length of 7 are applied. In the amplitude extraction stage, the amplitudes of the accelerometer signal and the gyroscope signal are extracted. The extracted amplitudes are added as additional channels to the 6-axis original IMU signal. For ECG data, the signal is filtered using a 50 Hz Butterworth low-pass filter, and wavelet decomposition and reconstruction are performed using an 8th-order Daubechies wavelet. To balance the computational cost and prediction accuracy of the model, non-overlapping sliding windows of 1 minute are selected for data segmentation. After data segmentation, 3044 data slices are obtained. The size of each IMU slice vector is 8×6000 (6-axis original signal and 2-channel amplitudes), and the size of each ECG slice vector is 12×12000 (12-lead signal).

[0034] After preprocessing the collected multi-source data, a sample data set for training the subsequent model is constructed. The sample data set reflects the correspondence between the characteristics of IMU data and ECG data and the fatigue level.

[0035] Step S130: Use the sample data set to train a spatio-temporal fusion dual-branch multimodal network.

[0036] As shown in Figure 3 The constructed spatio-temporal fusion dual-branch multimodal network generally includes a multimodal feature extraction module, a spatio-temporal dual-branch feature fusion module, a multimodal feature fusion module, and a physical fatigue level classification module. The multimodal feature extraction module includes a spatial branch and a temporal branch. The multimodal feature extraction module is used to extract multimodal features from the input data, including demographic features, physiological features of electrocardiogram signals, and motion features of inertial signals. The spatio-temporal dual-branch feature fusion module is used to interactively fuse the temporal features and spatial features from the same category of sensors to obtain a spatio-temporal feature joint representation of this category of sensors. The multimodal feature fusion module is used to capture the multimodal feature joint representation of different heterogeneous sensor signals through a multimodal feature merging attention module. The physical fatigue level classification is used to accurately classify the fatigue level based on the multimodal feature joint representation.

[0037] Specifically, the implementation process of using the spatio-temporal fusion dual-branch multimodal network for objective physical fatigue assessment is as follows:

[0038] Input: IMU chunks ECG Chunk and the feature chunks extracted from demographic information

[0039] Output: The predicted probability y for each fatigue category.

[0040] Multimodal Feature Extraction

[0041] Calculate the spatial and temporal feature embeddings of the IMU based on Formulas 1 and 2 respectively;

[0042] Calculate the spatial and temporal feature embeddings of the ECG based on Formulas 5 and 6 respectively

[0043] Calculate the feature embedding of demographic information (DI) based on Formula 9

[0044] Extract the spatial and temporal features of the IMU based on Formulas 3 and 4 respectively

[0045] Extract the spatial and temporal features of the ECG based on Formulas 7 and 8 respectively

[0046] Extract the features of demographic information based on Formula 10

[0047] Spatio-Temporal Dual-Branch Feature Fusion

[0048] Calculate the spatio-temporal fusion feature embedding of the IMU based on Formula 11

[0049] Calculate the spatio-temporal fusion feature embedding of the ECG based on Formula 13

[0050] Extract the spatio-temporal fusion feature of the IMU based on Formula 12

[0051] Extract the spatio-temporal fusion feature of the ECG based on Formula 14

[0052] Multimodal Feature Fusion

[0053] Calculate the multimodal fusion feature embedding based on Formula 15

[0054] Extract the multimodal fusion feature based on Formula 16

[0055] Calculate and return the predicted probability y for physical fatigue level classification.

[0056]

[0057]

[0058] In Equation (1) is the spatial feature embedding of the 0th layer of the IMU. The superscript s represents spatial, and the subscript 0 represents the initial layer 0. is the feature representation of the Nth chunk of the IMU. The superscript N represents the Nth chunk, and the subscript p represents the patch. I is the feature embedding matrix of the IMU chunk. is the position embedding matrix of the IMU spatial feature. The superscript s represents spatial, and the subscript pos represents position. In Equation (2) is the temporal feature embedding of the 0th layer of the cth channel of the IMU. The superscript t represents temporal, c represents the signal channel, and the subscript 0 represents the initial layer 0. is also the feature representation of the IMU chunk. The superscript represents the chunk number, and the subscript p represents the patch. is the position embedding matrix of the IMU temporal feature. The superscript t represents temporal, and the subscript pos represents position. In Equation (3) is the spatial feature extracted from the lth layer of the IMU. In Equation (4) is the temporal feature extracted from the lth layer of the IMU.

[0059] Similar to the above spatio-temporal dual-branch feature extraction process of the IMU, in Equation (5) is the spatial feature embedding of the 0th layer of the ECG. In Equation (6) is the temporal feature embedding of the 0th layer of the cth channel of the ECG. In Equation (7) is the spatial feature extracted from the lth layer of the ECG. In Equation 8 is the temporal feature extracted from the lth layer of the ECG.

[0060] In Equation (9) is the feature embedding of the 0th layer of the demographic information (DI). In Equation (10) is the feature representation extracted from the lth layer of the DI. In Equation (11) is the feature embedding of the 0th layer of the IMU after spatio-temporal dual-branch feature fusion. In Equation (12) is the spatio-temporal fusion feature of the IMU extracted from the lth layer. Similar to the spatio-temporal dual-branch feature fusion process of the IMU, in Equation (13) is the feature embedding of the 0th layer of the ECG after spatio-temporal dual-branch feature fusion. In Equation (14) is the spatio-temporal fusion feature of the ECG extracted from the lth layer. In Equation (15) is the feature embedding of the 0th layer after multi-modal feature fusion, x class is the class embedding in the Transformer, M i , M e and M d are the class embedding matrices of the three modalities of IMU, ECG, and DI respectively, M posis the position embedding matrix of the multi-modal fusion features. In Equation (16), is the multi-modal fusion feature extracted from the l-th layer. In Equation (17), y is the calculated fatigue level classification probability.

[0061] 1) Multi-modal feature extraction

[0062] The model captures multi-modal complementary information from multi-source sensor signals for a more comprehensive representation of the physiological state of the subject. It not only extracts the temporal features that previous methods often focus on, but also extracts the spatial features that are often overlooked. The model regards the combination of all channel signals in the sensor as a single-channel grayscale image, with the slice window length as the image length and the number of channels as the image width. The spatial branch of the model learns the correlation between different channels, can discover complex patterns across channels in the sensor, and finally improves the model's understanding ability of the data globally and the model's prediction ability. Different from the spatial branch, the temporal branch is used to capture the rich time-varying information contained in the sensor. To prevent the Transformer from generating cross-channel interactive attention when extracting temporal features, the temporal branch feeds the sensor data channel by channel into the Transformer encoding module with shared weights.

[0063] 2) Spatio-temporal dual-branch feature fusion

[0064] The time branch independently learns the time series features of each channel signal, and the spatial branch performs mutual attention on the local chunks across signal channels. The model performs interactive fusion of the temporal features and spatial features from the same category of sensors to obtain a joint spatio-temporal feature representation of this category of sensors. The spatio-temporal dual-branch network structure designed in the present invention captures both the rich time-varying information contained in the signal and the complex patterns across signal channels. Mining the spatial information of multi-channels of sensor signals helps to discover complex patterns. Each channel of the sensor signal represents information with different meanings and corresponds to different sub-states of the human body. The complex patterns of physiological signals and motion signals are often jointly represented by multi-channel features. The activity pattern of the heart needs to be observed by multi-lead electrocardiogram for its distribution and changes around the heart. The motion pattern of the limb needs to be jointly represented by the triaxial acceleration signal and angular velocity signal of the inertial measurement unit.

[0065] 3) Multi-modal feature fusion

[0066] The multi-modal feature interaction module is used to capture the joint representation of multi-modal features of heterogeneous sensor signals. The features of heterogeneous sensors are correlated and mutually attended to in the multi-modal feature interaction module. Subsequently, a Bi-LSTM is used to extract the high-level contextual semantic information between local chunks. The fusion of ECG and IMU signals is crucial for evaluating the degree of motor physical fatigue. ECG provides valuable information about heart rate variability, which is a reliable indicator of physiological stress. At the same time, IMU sensors can capture motion-related data, such as posture changes and body acceleration, which reflect the level of physical activity. By combining these two types of signals, the model can gain a more comprehensive understanding of the physiological and physical states of the body during physical fatigue.

[0067] 4) Classification of physical fatigue levels

[0068] During model training, cross-entropy loss can be used, and the self-assessed fatigue level of the subject at each exercise stage can be used as a reference label. The learnable class embedding tokens are extracted and passed through the multi-layer perceptron head for the classification of physical fatigue levels.

[0069] In one embodiment, the Transformer encoder is stacked by a multi-head self-attention (MSA) module, a multi-layer perceptron (MLP) module, and residual connections. The spatial features of the IMU are embedded through the chunk embedding matrix and the position embedding matrix into The temporal features of the IMU are embedded through the chunk embedding matrix and the position embedding matrix into The spatio-temporal feature embedding process of the ECG is similar to that of the IMU, but the sequence length of the ECG is twice that of the IMU, and the number of channels of the ECG is half that of the IMU. The features of demographic information are embedded through the chunk embedding matrix and the position embedding matrix into The multi-modal feature embedding module additionally adds a class embedding and a modality class embedding M datatype (i.e., and ). The number of channels is C = 1, the number of chunks is N = HW / (P H P W ) = 192, the chunk resolution of the IMU and the ECG is (P H = 1, P W = 750) and is mapped to V = 750 dimensions, and the chunk resolution of demographic information is and is mapped to V d = 125 dimensions.

[0070] In step S140, multi-source data is collected in real time for the target, and the trained spatio-temporal fusion dual-branch multi-modal network is used to obtain the fatigue level classification result.

[0071] After the spatio-temporal fusion dual-branch multi-modal network is trained, optimized model parameters such as weights and biases are obtained. In the model application stage, multi-source data is collected in real time for the target, and after preprocessing, it is input into the trained spatio-temporal fusion dual-branch multi-modal network to obtain the fatigue level classification result. The model application process is basically similar to the training process and will not be elaborated here.

[0072] To further verify the effect of the present invention, experimental simulations were carried out.

[0073] 65 healthy and physically capable volunteers were recruited for the experiment. The demographic information of the volunteers is shown in Table 3. During the data collection process, the subjects performed an incremental load exercise test on a treadmill and stopped the treadmill test when they felt extremely fatigued or physically uncomfortable. During the test, wearable devices were used to collect data from the subjects, and at the same time, the self-assessed fatigue level at each exercise stage was used as the classification label.

[0074] Table 3: Demographic information of the volunteers

[0075]

[0076] The experiment adopted a five-fold cross-validation strategy and evaluated the spatio-temporal fusion dual-branch multi-modal network (DBMNet) proposed by the present invention in both the coarse-grained fatigue level classification and the fine-grained fatigue level classification tasks. The final results were the average of the five experimental results. All the collected data was evenly divided into five groups, and one of the groups was taken as the test set in turn, and the remaining four groups were used as the training set. The scikit-learn library was used to calculate the multi-class classification performance metrics with sample weights. The six performance metrics were accuracy (ACC), precision (PRE), recall (REC), F1-Score (F1), Matthews correlation coefficient (MCC), and the area under the receiver operating characteristic curve (AUC). The experimental results are shown in Tables 4, 5 and Figure 4 as shown, where Figure 4 (a) is the coarse-grained fatigue classification, Figure 4 (b) is the fine-grained fatigue classification. In Tables 4 and 5, D represents demographic information, H represents heart rate signal, I represents IMU signal, and E represents ECG signal.

[0077] Table 4 Coarse-grained fatigue level classification results

[0078]

[0079] Table 5: Classification Results of Fine-Grained Fatigue Levels

[0080]

[0081] In summary, the experimental results show that the provided objective physical fatigue assessment method based on wearable sensing information fusion combines the physiological information of the 12-lead electrocardiogram (ECG) signal and the motion information of the multi-site inertial sensor (IMU) signal for the quantification and assessment of physical fatigue in sports health, captures the spatial information and temporal information of the same type of sensor, and interactively fuses the multi-modal complementary information of different types of sensors, ultimately improving the classification accuracy of physical fatigue degree by 4 to 5 percentage points.

[0082] It should be noted that the training process of the network model involved in the present invention can be carried out offline on a server or in the cloud, and the trained model can be embedded in an electronic device to achieve real-time assessment of the physical fatigue state. The electronic device can be a terminal device or a server. The terminal device includes any terminal device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, a smart wearable device (smart watch, virtual reality glasses, virtual reality helmet, etc.). The server includes but is not limited to an application server or a Web server, and can be an independent server or a cluster server or a cloud server, etc.

[0083] In summary, compared with the prior art, the present invention has the following advantages:

[0084] 1) In terms of feature extraction, the present invention designs a spatio-temporal dual-branch network structure. The time branch independently learns the time series features of each channel signal, and the space branch learns the correlation relationship between different channels to discover complex patterns across channels in the sensor. In this way, both the rich time-varying information contained in the signal and the complex patterns across signal channels are captured. Mining the spatial information of multiple channels of sensor signals helps to discover complex patterns, and ultimately improves the model's ability to understand the data globally and its prediction ability.

[0085] 2) In terms of feature fusion, for homogeneous sensors, the present invention uses a spatio-temporal dual-branch feature interaction module to interactively fuse the temporal features and spatial features from the same type of sensor to obtain a joint spatio-temporal feature representation of this type of sensor; for heterogeneous sensors, a multi-modal feature interaction module is used to capture the joint multi-modal feature representation of the signals of different heterogeneous sensors. By integrating multi-source wearable sensing information, the complementary features in wearable heterogeneous sensors can be fully captured, and a more comprehensive assessment of the fatigue state of athletes can be carried out from the multi-dimensional perspectives of cardiac activity and body movement.

[0086] 3) The present invention does not require a cumbersome and complex manual feature extraction and screening process. The designed network model can achieve end-to-end real-time physical fatigue assessment, providing a new solution for accurate and reliable exercise fatigue assessment, and contributing to promoting the scientific process of sports health management and personalized training.

[0087] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0088] The computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not construed as an instantaneous signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0089] The computer-readable program instructions described herein can be downloaded from the computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0090] The computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, Python, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or alternatively, may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present invention.

[0091] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer - readable program instructions.

[0092] These computer - readable program instructions can be provided to a processor of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data - processing apparatus, create a means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, which causes a computer, a programmable data - processing apparatus, and / or other devices to operate in a particular manner, so that the computer - readable medium storing the instructions comprises a manufacture, which includes instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0093] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0094] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions. As will be apparent to those of ordinary skill in the art, implementations in hardware, in software, and in a combination of software and hardware are all equivalent.

[0095] The embodiments of the present invention have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of technologies in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. An objective physical fatigue assessment method based on wearable sensor information fusion, comprising the following steps: Using a wearable device to collect an IMU signal and an ECG signal of a target, wherein the wearable device includes an IMU sensor and an electrocardiogram detection sensor; Performing data preprocessing on the IMU signal and the ECG signal, and then inputting them into a trained spatiotemporal fusion dual-branch multimodal network to obtain a fatigue level classification result of the target; Among them, the spatiotemporal fusion dual-branch multimodal network includes a multimodal feature extraction module, a spatiotemporal dual-branch feature fusion module, a multimodal feature fusion module and a fatigue level classification module. The multimodal feature extraction module includes a spatial branch and a temporal branch. The spatial branch is used to extract the correlation relationship between different channels, and the temporal branch is used to capture the time change information of the input signal; the spatiotemporal dual-branch feature fusion module is used to interactively fuse the signal temporal features and spatial features from the same category of sensors to obtain a joint representation of spatiotemporal features; the multimodal feature fusion module is used to capture the multimodal feature joint representation of each heterogeneous sensor signal; the fatigue level classification module is used to obtain a fatigue level classification result based on the multimodal feature joint representation.

2. The method according to claim 1, characterized in that The multimodal feature extraction module performs the following steps: The spatial feature embedding and temporal feature embedding of the IMU signal are calculated based on the following formulas: The spatial feature embedding and temporal feature embedding of ECG signals are calculated based on the following formulas: The feature embedding of demographic information is calculated based on the following formula: The spatial and temporal features of the IMU signal are extracted based on the following formulas: The spatial and temporal features of ECG signals are extracted based on the following formulas: Extract features of demographic information based on the following formula: in, It is the spatial feature embedding of the 0th layer of the IMU signal. The superscript s represents the space, and the subscript 0 represents the initial layer 0. is the feature representation of the Nth slice of the IMU signal, the superscript N represents the Nth slice, the subscript p represents the slice patch, and I is the feature embedding matrix of the IMU signal slice. It is the position embedding matrix of the spatial features of the IMU signal. The subscript pos indicates the position. It is the temporal feature embedding of the cth channel and the 0th layer of the IMU signal. The superscript t represents the timing, c represents the signal channel, and the subscript 0 represents the initial layer 0. is the block feature representation of the IMU signal, is the position embedding matrix of the IMU signal timing features, is the spatial feature extracted from the lth layer of the IMU signal, is the temporal feature extracted from the lth layer of the IMU signal, where is the spatial feature embedding of the ECG signal at layer 0, is the temporal feature embedding of the cth channel and the 0th layer of the ECG signal, is the spatial feature extracted from the ECG signal at layer l, is the temporal feature extracted from the lth layer of the ECG signal, is the feature embedding of the 0th layer of demographic information, It is the feature representation extracted from the lth layer of demographic information.

3. The method according to claim 2, characterized in that The spatiotemporal dual-branch feature fusion module performs the following steps: The spatiotemporal fusion feature embedding of the IMU signal is calculated based on the following formula: The spatiotemporal fusion feature embedding of ECG signals is calculated based on the following formula: The spatiotemporal fusion features of the IMU signal are extracted based on the following formula: The spatiotemporal fusion features of ECG signals are extracted based on the following formula: in, It is the 0th layer feature embedding of the IMU signal after the spatiotemporal dual-branch feature fusion. is the spatiotemporal fusion feature of the IMU signal extracted by the lth layer, It is the 0th layer feature embedding of the ECG signal after the spatiotemporal dual-branch feature fusion. It is the ECG spatiotemporal fusion feature extracted from the lth layer.

4. The method according to claim 1, characterized in that: The multimodal feature fusion module performs the following steps: The multimodal fusion feature embedding is calculated based on the following formula: < img src='' class="img-anchor" img-id="FDA0005209557710000031" / > The multimodal fusion features are extracted based on the following formula: Calculate and return the predicted probability y of the physical fatigue level classification: in, is the 0th layer feature embedding after multimodal feature fusion, M i , M e and M d They are the category embedding matrices of the three modalities: IMU signal, ECG signal and demographic information, M pos is the position embedding matrix of multimodal fusion features, is the multimodal fusion feature extracted from the lth layer, and y is the calculated fatigue level classification probability.

5. The method according to claim 1, characterized in that The wearable device includes an inertial sensor and an electrocardiogram monitoring device, wherein the inertial sensor is used to collect IMU data, and the electrocardiogram monitoring device is used to collect ECG signals.

6. The method according to claim 1, characterized in that The data preprocessing process includes: The IMU signal is subjected to signal filtering, amplitude extraction, and data segmentation in sequence. In the signal filtering stage, a Butterworth low-pass filter with a cutoff frequency of 10 Hz and a Wiener filter with a window length of 7 are applied. In the amplitude extraction stage, the amplitudes of the accelerometer signal and the gyroscope signal are calculated, and the extracted amplitudes are added as additional channels to the 6-axis original IMU signal. The ECG signal is subjected to signal filtering, wavelet decomposition and reconstruction, and data segmentation in sequence, wherein a 50 Hz Butterworth low-pass filter is used to filter the signal, and an 8th-order Daubechies wavelet is used for wavelet decomposition and reconstruction; Among them, a non-overlapping sliding window of 1 minute is selected for data segmentation. After data segmentation, 3044 data slices are obtained. The size of each IMU slice vector is 8×6000, and the size of each ECG slice vector is 12×12000.

7. The method according to claim 1, characterized in that The fatigue level classification results include five levels: no fatigue, mild fatigue, moderate fatigue, severe fatigue and extreme fatigue.

8. The method according to claim 1, characterized in that The spatiotemporal fusion dual-branch multimodal network is trained using cross entropy loss.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images