Gait monitoring method, electronic equipment and wearable equipment

Abstract

The invention is suitable for the field of wearable equipment, and provides a gait monitoring method, electronic equipment and wearable equipment. The gait monitoring method comprises the following steps: acquiring IMU data and sEMG data of a wearable device worn on a leg; the IMU data and the sEMG data are input into a fusion model, and target muscle strength and target KAM output by the fusion model are obtained; wherein the fusion model is used for performing feature extraction on the IMU data and the sEMG data, and fusing the extracted features based on a cross-modal attention mechanism, so that the correlation between the IMU data and the sEMG data can be more fully mined, the IMU data and the sEMG data are better fused, and the prediction performance is further improved; and thus, the calibration work when the wearable device is used can be reduced.

Description

Technical Field

[0001] The present application relates to the field of wearable devices, and in particular to gait monitoring methods, electronic devices, and wearable devices. Background Art

[0002] Knee osteoarthritis (KOA) is one of the most common degenerative joint diseases, with symptoms including pain, stiffness, limited joint movement, and muscle weakness. It is estimated that the number of people suffering from KOA worldwide in 2020 was 86.7 million, and the prevalence of KOA in people aged 40 and above was 22.9%. As a degenerative disease, KOA currently has no good treatment. In clinical practice, severe patients who cannot tolerate knee pain may request knee replacement surgery. However, this treatment is invasive and may lead to problems such as postoperative infection. In contrast, conservative treatments such as physical therapy do not require intervention, can relieve patients' pain, and improve their quality of life, and are therefore more advantageous.

[0003] Gait retraining uses conservative treatment to reduce the load on the knee joint. Specifically, patients can be guided to adjust their gait through several strategies, such as toe in / out, knee in / out, or widening the stride, thereby reducing the load on the knee joint and relieving the patient's pain. Studies have shown that 70% of people have reduced the load on the knee joint by at least 5% after adjusting the foot progression angle (FPA) using personalized training methods. In addition, the way muscles are coordinated during walking also affects the load on the knee joint. Simulation results show that the load on the knee joint can be reduced by "avoiding the gastrocnemius muscle" gait pattern. Therefore, gait retraining is an effective way to slow the progression of the disease and relieve pain in patients with KOA.

[0004] Current gait retraining methods are mostly fixed strategies, but personalized gait retraining can better reduce knee joint load and muscle control. Personalized gait retraining strategies based on biomechanical indicators such as knee joint load and muscle strength have the potential to improve rehabilitation outcomes and have shown promise. If knee joint load and muscle strength can be monitored during gait retraining, personalized gait retraining can be achieved.

[0005] Accurately monitoring these metrics requires an advanced gait analysis system. However, due to its complex setup and high cost, this system is not widely used. Currently, some wearable device solutions can estimate biomechanical metrics related to the knee joint, but these solutions only achieve good performance after calibration using the annotated data provided by the gait analysis system for each new user, making them inconvenient for users. Summary of the Invention

[0006] In view of this, embodiments of the present application provide a gait monitoring method, an electronic device, and a wearable device, which can reduce calibration work when using the wearable device.

[0007] A first aspect of an embodiment of the present application provides a gait monitoring method, comprising: acquiring IMU data and sEMG data from a wearable device worn on a leg;

[0008] The IMU data and the sEMG data are input into a fusion model to obtain the target muscle strength and target KAM output by the fusion model; wherein the fusion model is used to extract features from the IMU data and the sEMG data, and fuse the extracted features based on a cross-modal attention mechanism.

[0009] In one embodiment, inputting the IMU data and the sEMG data into a fusion model to obtain the target muscle strength and target KAM output by the fusion model includes:

[0010] Performing feature extraction on the IMU data to obtain a first muscle strength feature and a first KAM feature;

[0011] Performing feature extraction on the sEMG data to obtain a second muscle strength feature and a second KAM feature;

[0012] fusing the first muscle strength feature and the second muscle strength feature based on a cross-modal attention mechanism to obtain a target muscle strength;

[0013] The first KAM feature and the second KAM feature are fused based on the cross-modal attention mechanism to obtain the target KAM.

[0014] In one embodiment, the fusing of the first KAM feature and the second KAM feature based on the cross-modal attention mechanism to obtain a target KAM includes:

[0015] The first muscle strength feature and the second KAM feature are fused based on a cross-modal attention mechanism to obtain a KAM splicing feature;

[0016] The KAM concatenation feature and the first KAM feature are fused based on a cross-modal attention mechanism to obtain the target KAM.

[0017] In one embodiment, the muscle strength includes QF muscle strength, BF muscle strength and GAS muscle strength.

[0018] In one embodiment, the fusing of the first muscle strength feature and the second KAM feature based on the cross-modal attention mechanism to obtain the KAM splicing feature includes:

[0019] The first QF muscle strength feature and the second KAM feature are fused based on the cross-modal attention mechanism to obtain the first KAM fusion feature;

[0020] The first BF muscle strength feature and the second KAM feature are fused based on the cross-modal attention mechanism to obtain the second KAM fusion feature;

[0021] The first GAS muscle strength feature and the second KAM feature are fused based on the cross-modal attention mechanism to obtain the third KAM fusion feature;

[0022] The first KAM fusion feature, the second KAM fusion feature and the third KAM fusion feature are spliced ​​to obtain the KAM splicing feature.

[0023] In one embodiment, extracting features from the IMU data to obtain a first muscle strength feature and a first KAM feature includes:

[0024] Get the first IMU data of the thigh;

[0025] Get the second IMU data of the calf;

[0026] Performing feature extraction on the first IMU data and the second IMU data based on a cross attention mechanism to obtain IMU features;

[0027] The IMU features are sequentially input into the first Transformer encoder layer and the second Transformer encoder layer to obtain the first muscle strength features and the first KAM features.

[0028] In one embodiment, extracting features from the sEMG data to obtain a second muscle strength feature and a second KAM feature includes:

[0029] Acquire multiple channels of sEMG data collected during the gait cycle;

[0030] Performing feature extraction on the sEMG data of the multiple channels based on a channel attention mechanism to obtain sEMG features;

[0031] The sEMG feature is input into the third Transformer encoder layer to obtain the second muscle strength feature and the second KAM feature.

[0032] In one embodiment, the fusion framework is trained based on a multi-task learning method.

[0033] A second aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the gait monitoring method as described in the first aspect above is implemented.

[0034] A third aspect of an embodiment of the present application provides a wearable device, including an IMU sensor, a multi-channel sEMG sensor, and the electronic device as described in the second aspect above.

[0035] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the gait monitoring method as described in the first aspect above is implemented.

[0036] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the gait monitoring method described in any one of the first aspects above.

[0037] Compared with the prior art, the beneficial effects of the embodiments of the present application are as follows: by obtaining the IMU data and sEMG data of the wearable device worn on the legs, the IMU data and sEMG data are input into the fusion model to obtain the target muscle strength and target KAM output by the fusion model. Since the target muscle strength and target KAM are obtained by combining the IMU data and sEMG data, more comprehensive leg movement information can be obtained, thereby improving the prediction performance. At the same time, since the fusion model is used to extract features from the IMU data and sEMG data, and the extracted features are fused based on the cross-modal attention mechanism, the correlation between the IMU data and the sEMG data can be more fully explored, and the IMU data and the sEMG data can be better fused, thereby further improving the prediction performance, and thus reducing the calibration work when using the wearable device. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art.

[0039] Figure 1 The KAM curve diagram provided for a healthy person during a gait cycle;

[0040] Figure 2 This is the KAM curve diagram of KOA patients in one gait cycle;

[0041] Figure 3 Schematic diagram of forward dynamics and inverse dynamics in biomechanical calculations;

[0042] Figure 4A schematic diagram of a wearable device provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the implementation flow of the gait monitoring method provided in an embodiment of the present application;

[0044] Figure 6 A framework diagram for extracting features from sEMG data from multiple channels based on a channel attention mechanism provided in an embodiment of the present application;

[0045] Figure 7 This is an architectural diagram of the channel attention module provided in an embodiment of the present application;

[0046] Figure 8 This is an architectural diagram of a linear encoding module provided in an embodiment of the present application;

[0047] Figure 9 A framework diagram for feature extraction of IMU data based on the cross-attention mechanism provided in an embodiment of the present application;

[0048] Figure 10 A framework diagram for fusing IMU features and sEMG features provided in an embodiment of the present application;

[0049] Figure 11 A schematic diagram of a gait monitoring system provided in an embodiment of the present application;

[0050] Figure 12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0052] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0053] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0054] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0055] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0056] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0057] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0058] Biomechanical indicators such as knee joint load and muscle strength play an important role in personalized gait retraining.

[0059] Knee adduction moment (KAM) is used to characterize knee joint load. KAM refers to the moment acting on the front of the knee joint, which affects knee joint load during walking. Typically, during the stance phase, when one leg supports the entire body, KAM during gait is a curve with two peaks. During the swing phase, when there is no force on the leg, KAM is zero. Figure 1 is the KAM curve of a healthy person in a gait cycle, Figure 2 Figure 2 shows the KAM curve for a KOA patient during a gait cycle. To account for weight differences among subjects, the figure shows the weight-normalized KAM curve. The horizontal axis represents the percentage of a gait cycle, and the vertical axis represents KAM in N*m / kg. The primary goal of gait retraining is to reduce peak KAM through various strategies, thereby reducing the load on the knee joint.

[0060] Lower limb muscle strength includes the quadriceps femoris (QF), biceps femoris (BF), and gastrocnemius (GAS). As the most powerful knee extensor, the quadriceps femoris (QF) is crucial to the knee joint. It comprises four major muscles on the anterior thigh: the rectus femoris, vastus lateralis, vastus medialis, and vastus medius. The biceps femoris (BF), also known as the hamstrings, is a muscle located on the posterior thigh and consists of two heads: the long head and the short head. The biceps femoris plays a crucial role in knee abduction and internal or external rotation. The gastrocnemius (GAS) is a superficial muscle on the posterior calf, extending from its two heads, the medial and lateral heads, just above the knee to the heel. Together with the muscle head originating from the femur, it helps stabilize the knee. During walking, the gastrocnemius intervenes to prevent the tibia from translating forward, while the quadriceps extend the knee.

[0061] For example, to calculate KAM and lower limb muscle strength, modeling can be performed through forward dynamics and inverse dynamics.

[0062] Figure 3 This is a schematic diagram of forward dynamics and inverse dynamics in biomechanics calculations. The motion equation for inverse dynamics is:

[0063]

[0064] Where q represents the joint angle, Indicates speed, represents acceleration, M represents inertia matrix, C represents Coriolis force / centrifugal force, G represents gravity, τ represents internal torque, J F represents the contact force (i.e., ground reaction force), represents the torsional moment matrix.

[0065] It can be understood that according to the positions of various body parts and the measured ground reaction force, the joint torque can be calculated according to the above-mentioned inverse dynamics motion equation, and the muscle force components can be further analyzed.

[0066] In forward dynamics, starting from the nerve's command to muscle control, and then analyzing muscle activation dynamics and muscle contraction dynamics, muscle strength can be obtained. Among them, the calculation of muscle contraction dynamics is based on the Hill muscle model:

[0067]

[0068] Where a represents the degree of muscle activation, b represents the force-related discontinuity function, represents the normalized muscle fiber velocity, represents the normalized muscle fiber length, represents the activated muscle force. It can be seen that muscle fiber velocity and muscle fiber length are both correlated with muscle force. The changing patterns of muscle fiber length and muscle fiber velocity can be inferred from the kinematic model of limb movement. After obtaining the muscle force, the joint torque is calculated using forward dynamics and formula (3).

[0069] Among them F i Indicates muscle strength, r i Represents the moment arm.

[0070] Based on the above calculation principle, knee joint load and muscle strength can be calculated in combination with a gait analysis system. The gait analysis system includes optical motion capture (Mocap), ground reaction force (GRF) measurement, and biomechanical modeling. This system is complex and expensive to set up, making it unsuitable for both clinical and everyday applications.

[0071] In one embodiment, knee joint load and muscle strength can be inferred by wearing a wearable device. One method is to wear an inertial measurement unit (IMU) sensor on the lower limbs, using the IMU sensor to collect motion information and predict knee joint load and muscle strength. Another method is to wear a surface electromyography (sEMG) sensor on the lower limbs and estimate knee joint load and muscle strength based on muscle activation information collected by the sEMG sensor during walking.

[0072] IMU-based methods are generally based on the principle of forward dynamics and lack information about ground reaction forces. sEMG-based methods are generally based on the principle of inverse dynamics and cannot provide the kinematic information required for muscle contraction dynamics and torque calculations, so all of the above methods require calibration of labeled data for new users. Specifically, for new users, it is first necessary to use a gait analysis system to collect the user's gait data, and use this data to retrain or adjust the model before monitoring the knee joint load and muscle strength of the new user. Since advanced gait analysis systems are complex to set up and expensive, these methods are not practical in actual applications.

[0073] Therefore, wearable devices with only separate sEMG sensors or IMU sensors can only provide muscle-related information or body geometry-related information and cannot perform accurate model predictions.

[0074] In one embodiment, IMU sensors and sEMG sensors can be worn simultaneously on the lower limbs. The data collected by the sEMG and IMU sensors can be fused to comprehensively retrieve biomechanical information about muscle activity and movement, thereby calculating knee joint load and muscle force. The inclusion of IMU sensors provides kinematic information that is missing from sEMG sensors. For example, during walking, muscle contraction geometry and musculoskeletal geometry can be inferred from data collected by the IMU sensors.

[0075] However, simple data fusion has poor prediction performance. For example, directly feeding data collected by sEMG and IMU sensors into a machine learning model for prediction fails to fully exploit the biomechanical correlation between the two modalities, and prediction performance remains poor without calibration.

[0076] Because the data collected by sEMG sensors and IMU sensors have completely different semantics in biomechanical modeling, and the corresponding biomechanical modeling is nonlinear and complex, mining the biomechanical correlation between the two is a daunting task. Furthermore, muscles and joints must satisfy torque equations and implicit balance requirements, leading to hidden correlations between muscle forces and KAMs, resulting in hidden biomechanical correlations between different prediction tasks. Furthermore, the biomechanical modeling schemes of the two modalities are also different, making fusion even more challenging.

[0077] To this end, the present application provides a wearable device, including an IMU sensor, a multi-channel sEMG sensor and an electronic device. The IMU sensor and the multi-channel sEMG sensor are worn on the user's legs, and the electronic device performs a gait monitoring method.

[0078] For example, Figure 4 As shown, the wearable device includes a pair of leg straps worn on the thigh and calf, respectively. Each leg strap integrates an IMU sensor and a multi-channel sEMG sensor. The number of sEMG channels of the leg straps on the thigh and calf can be 8 and 6, respectively. Each channel includes a pair of vertically placed dual electrodes and a reference electrode, which is used to collect standard sEMG signals. The sEMG sensors are arranged near the inner side of the leg, and the IMU sensor is set on the outer side. The IMU sensor and reference electrode are placed on muscle-free body parts to avoid artifacts. For example, the IMU sensor is placed on the iliotibial band of the thigh and the tibia of the calf to minimize the influence of muscle movement. The reference electrode of the thigh leg strap is located on the iliotibial bone. The iliotibial band is a thick band of fascial tissue extending along the outside of the leg, while the reference electrode of the calf leg strap is located on the tibia (i.e., the shin bone).

[0079] Exemplarily, each leg strap includes an IMU sensor, a pair of dry electrodes for acquiring sEMG signals (i.e., sEMG sensors), a shielded wire for transmitting sEMG signals, a PCB board for collecting IMU signals and sEMG signals from the user, a controller, and elastic cloth.

[0080] The gait monitoring method adopts a fusion model to extract features from the IMU data collected by the IMU sensor and the sEMG data collected by the sEMG sensor, and fuses the extracted features based on the cross-modal attention mechanism, so as to obtain more comprehensive leg movement information, more fully explore the correlation between the IMU data and the sEMG data, better fuse the IMU data and the sEMG data, improve the prediction performance, and thus reduce the calibration work when using wearable devices. At the same time, in the feature extraction process, the present application extracts the features of the sEMG data based on the channel-attention mechanism to dynamically weigh the multi-channel sEMG data and improve the quality of the collected signal, so that the sEMG sensor does not need to be placed in a precise position, which is convenient for wearing. The present application extracts the features of the IMU data based on the cross-attention mechanism to effectively extract motion information from the IMU data and complete different prediction tasks.

[0081] The following is an exemplary description of the gait monitoring method provided in this application.

[0082] like Figure 5 As shown, the gait monitoring method provided by one embodiment of the present application includes:

[0083] S101: Obtain IMU data and sEMG data from a wearable device worn on the leg.

[0084] The IMU data is collected by an IMU sensor. The wearable device may include two IMU sensors, one for wearing on the thigh and one for wearing on the calf. The sEMG data includes data collected by each sEMG sensor in a multi-channel sEMG sensor. The multiple sEMG sensors in the multi-channel sEMG sensor are arranged in an array. For example, the multi-channel sEMG sensor includes two sEMG sensor arrays.

[0085] S102: Input the IMU data and the sEMG data into a fusion model to obtain the target muscle strength and target KAM output by the fusion model.

[0086] Among them, the fusion model is trained based on a fusion framework inspired by biomechanics. The fusion model is used to extract features from IMU data and sEMG data, and fuse the extracted features based on the cross-modal attention mechanism.

[0087] The data processing process of the fusion model is introduced in detail below.

[0088] For ease of wearing, the sEMG sensor may not be at the precise position to be measured, and the sEMG data is the sEMG signal near the position to be measured. After acquiring the sEMG data, the characteristics of the sEMG signal at the position to be measured can be determined based on the signal attenuation corresponding to different positions and multiple sEMG data near the position to be measured. The traditional blind source separation (BSS) method cannot be directly used for feature extraction. To this end, the present application performs feature extraction on sEMG data of multiple channels based on the channel attention mechanism, so that different attention can be given to sEMG signals of different channels to achieve better feature extraction.

[0089] For example, Figure 6 This diagram shows a framework for feature extraction from multi-channel sEMG data based on a channel-attention mechanism. During a gait cycle, 14 channels of sEMG signals are collected. The channels associated with each muscle group can be determined based on sEMG localization and prior knowledge. A short-time Fourier transform (STFT) is performed on each channel's sEMG signal. Feature extraction is performed on the transformed signal through a convolutional layer, resulting in vectorized KAM features, vectorized QF muscle strength features, vectorized BF muscle strength features, and vectorized GAS muscle strength features. These vectorized KAM features, vectorized QF muscle strength features, vectorized BF muscle strength features, and vectorized GAS muscle strength features are then fed into the corresponding channel-attention module for feature extraction. The channel-attention module is a convolutional module with enhanced channel attention. Feature extraction through the channel-attention module squeezes the current features onto the temporal channel dimension and generates an attention map to better focus the signal for feature extraction. Since the time granularity is small, the kernel size of the time dimension can be set to 15, the kernel size of the channel dimension can be set to 3, and circular padding can be performed. For example, Figure 7 As shown in the figure, the channel attention module includes a compression module, an attention module and a normalization (Softmax) module. The features of the input compression module are extracted by the compression module, the attention module and the normalization module in sequence. The extracted features are then fused with the features of the input compression module. The obtained features are the features output by the channel attention module. The features output by the channel attention module are then input into the corresponding linear encoding module for feature extraction to obtain the sEMG features extracted from the sEMG signal. For example, Figure 8As shown in the figure, the linear encoding module includes a multilayer perceptron (MLP), a norm layer, and a positional encoding layer. The features output by the channel attention module pass through the MLP, norm layer, and positional encoding layer in sequence. The obtained features are then residually connected (Add) with the features output by the norm layer to obtain the features output by the linear encoding module, namely the sEMG features. The activation function between each layer in the feature extraction framework can be SELU.

[0090] Feature extraction of sEMG data based on the channel attention mechanism can obtain robust features from time and channel analysis, and the sEMG signals of all channels can be used for overall analysis of the lower limbs.

[0091] Figure 9This is a framework diagram for feature extraction of IMU data based on the cross-attention mechanism. The first IMU data collected by the IMU worn on the thigh and the second IMU data collected by the IMU worn on the shank are input into the cross-attention module to obtain the first motion feature and the second motion feature, respectively. The first motion feature and the second motion feature are fused and then input into the convolutional feature extractor and the IMU feature embedding layer in sequence to obtain the IMU features extracted from the IMU data. Among them, the cross-attention module includes a multi-head attention module, a feedforward layer, two residual connections and a layer normalization (Add&Norm) module corresponding to the first IMU data, and also includes a multi-head attention module, a feedforward layer, two residual connections and a layer normalization (Add&Norm) module corresponding to the second IMU data. The IMU data input to the multi-head attention module includes the query (Query, Q), key (Key, K), and value (Value, V) corresponding to the first IMU data, as well as the Q value, K value, and V value corresponding to the second IMU data. For the first IMU data and the second IMU data, when the Q value, K value, and V value are input into the corresponding multi-head attention module, the V value of the first IMU data and the V value of the second IMU data are replaced. Afterwards, the output value of the multi-head attention module corresponding to the first IMU data and the first IMU data are synchronously input into a residual connection and layer normalization module, and the output data is then processed by a feedforward layer and another residual connection and layer normalization module to obtain the first motion feature. The output value of the multi-head attention module corresponding to the second IMU data and the second IMU data are synchronously input into a residual connection and layer normalization module, and the output data is then processed by a feedforward layer and a residual connection and layer normalization module to obtain the second motion feature. The above-mentioned cross-attention module is also called a cross-attention enhanced Transformer encoder layer, which is a method of introducing a cross-attention mechanism into a Transformer model to enhance encoder performance. For example, the attention-enhanced Transformer encoder layer has a hidden layer dimension of 32 and a feedforward layer dimension of 144, equipped with four attention heads. The convolutional feature extractor includes a convolution module with a kernel size of 5 and a max pooling layer to align the signal length and extract all channel features. To preserve important sequence information, a learnable positional encoding can also be added to each IMU feature embedding layer. The activation function between each layer in the feature extraction framework can be SELU.

[0092] Since the changes in knee joint movement and the geometry of related muscle contraction are not determined solely by the thigh or calf, but by both the thigh and the calf, by processing the IMU data through the above-mentioned multi-head attention module, it is possible to calculate attention not only based on a single IMU during the feature extraction process, but also focus on the important time position of the other IMU, thereby extracting the correlation between the first IMU data and the second IMU data, thereby improving the accuracy of subsequent calculations.

[0093] I understand. Figure 6 and Figure 9 The following diagrams outline the framework for extracting features from sEMG data and IMU data, respectively. The feature extraction process also includes other data preprocessing steps. For example, extracting features from sEMG data involves gait cycle segmentation and sEMG data filtering, while extracting features from IMU data involves IMU complementary filtering, STFT, IMU supplementary filtering, and IMU normalization.

[0094] In the above embodiment, feature extraction of sEMG data using four parallel encoder layers can capture temporal information in the sEMG data. Feature extraction of IMU data using the first and second Transformer encoder layers can extract features related to muscle fiber geometry and body kinematics from the IMU.

[0095] Figure 10 This diagram shows the framework for fusing IMU features with sEMG features. Specifically, after feature extraction of the IMU data, the extracted IMU features are sequentially input into the first and second Transformer Encoder layers. The second Transformer Encoder layer consists of three parallel encoder layers, one for QF, one for BF, and one for GAS. The outputs of these three encoder layers are then fed into three IMU Embed layers, one for QF, one for BF, and one for GAS. The outputs of these three IMU Embed layers are the first QF muscle force feature, the first BF muscle force feature, and the first GAS muscle force feature, respectively, representing the first muscle force feature. The output of the first Transformer Encoder layer is also fed into an IMU Embed layer corresponding to the KAM (Kinematic Acuity Modulation) to produce the first KAM feature. The embedding layer of the first Transformer Encoder supervises the learning of kinematic features, while the embedding layer of the second Transformer Encoder supervises the learning of the muscle fiber geometry of the corresponding muscle.

[0096] After feature extraction of the sEMG data, the extracted sEMG features are fed into the third Transformer encoder layer, which consists of four parallel encoder layers, one for QF, one for BF, one for GAS, and one for KAM. The output data from these four encoder layers is fed into four sEMG embedding layers, each corresponding to QF, BF, GAS, and KAM. The output data from these four sEMG embedding layers are the second QF muscle strength feature, the second BF muscle strength feature, the second GAS muscle strength feature, and the second KAM feature. The second QF muscle strength feature, the second BF muscle strength feature, and the second GAS muscle strength feature constitute the second muscle strength feature.

[0097] The first QF muscle strength feature and the second QF muscle strength feature are input into the first cross-modal attention (CMA) module. The data processing process of the CMA module is the same as that of the cross-attention module. The first CMA module includes two Cross Transformer Layers. The data input into one Cross Transformer Layer includes the Q value, K value, and V value corresponding to the first QF muscle strength feature, and the data input into the other Cross Transformer Layer includes the Q value, K value, and V value corresponding to the second QF muscle strength feature. When the Q value, K value, and V value are input into the corresponding Cross Transformer Layer, the V value of the first QF muscle strength feature is replaced with the V value of the second QF muscle strength feature, and then the data output by the two Cross Transformer Layers are spliced ​​to obtain the QF fusion feature.

[0098] The first BF muscle strength feature and the second BF muscle strength feature are input into the second CMA module. The second CMA module includes two Cross Transformer Layers. The data input into one Cross Transformer Layer includes the Q value, K value, and V value corresponding to the first BF muscle strength feature, and the data input into the other Cross Transformer Layer includes the Q value, K value, and V value corresponding to the second BF muscle strength feature. When the Q value, K value, and V value are input into the corresponding Cross Transformer Layer, the V value of the first BF muscle strength feature is replaced with the V value of the second BF muscle strength feature, and then the data output by the two Cross Transformer Layers are spliced ​​to obtain the BF fusion feature.

[0099] The first GAS muscle strength feature and the second GAS muscle strength feature are input into the third CMA module. The third CMA module includes two Cross Transformer Layers. The data input into one Cross Transformer Layer includes the Q value, K value, and V value corresponding to the first GAS muscle strength feature, and the data input into the other Cross Transformer Layer includes the Q value, K value, and V value corresponding to the second GAS muscle strength feature. When the Q value, K value, and V value are input into the corresponding Cross Transformer Layer, the V value of the first GAS muscle strength feature is replaced with the V value of the second GAS muscle strength feature, and then the data output by the two Cross Transformer Layers are spliced ​​to obtain the GAS fusion feature.

[0100] The first QF muscle strength feature and the second KAM feature are input into the fourth CMA module. The fourth CMA module includes two Cross Transformer Layers. The data input into one Cross Transformer Layer includes the Q value, K value, and V value corresponding to the first QF muscle strength feature, and the data input into the other Cross Transformer Layer includes the Q value, K value, and V value corresponding to the second KAM feature. When the Q value, K value, and V value are input into the corresponding Cross Transformer Layer, the V value of the first QF muscle strength feature is replaced with the V value of the second KAM feature. The data output from the two Cross Transformer Layers are then spliced ​​to obtain the first KAM fusion feature.

[0101] The first BF muscle strength feature and the second KAM feature are input into the fifth CMA module, which includes two Cross Transformer Layers. The data input into one Cross Transformer Layer includes the Q value, K value, and V value corresponding to the first BF muscle strength feature, and the data input into the other Cross Transformer Layer includes the Q value, K value, and V value corresponding to the second KAM feature. When the Q value, K value, and V value are input into the corresponding Cross Transformer Layer, the V value of the first BF muscle strength feature is replaced with the V value of the second KAM feature, and then the data output from the two Cross Transformer Layers are spliced ​​to obtain the second KAM fusion feature.

[0102] The first GAS muscle strength feature and the second KAM feature are input into the sixth CMA module, which includes two Cross Transformer Layers. The data input into one Cross Transformer Layer includes the Q value, K value, and V value corresponding to the first GAS muscle strength feature, and the data input into the other Cross Transformer Layer includes the Q value, K value, and V value corresponding to the second KAM feature. When the Q value, K value, and V value are input into the corresponding Cross Transformer Layer, the V value of the first GAS muscle strength feature is replaced with the V value of the second KAM feature, and then the data output from the two Cross Transformer Layers are spliced ​​to obtain the third KAM fusion feature.

[0103] The first KAM fusion feature, the second KAM fusion feature, and the third KAM fusion feature are spliced ​​to obtain a KAM spliced ​​feature. The KAM spliced ​​feature and the first KAM feature are input into the seventh CMA module. The seventh CMA module includes two Cross Transformer Layers. The data input into one Cross Transformer Layer includes the Q value, K value, and V value corresponding to the KAM spliced ​​feature, and the data input into the other Cross Transformer Layer includes the Q value, K value, and V value corresponding to the first KAM feature. When the Q value, K value, and V value are input into the corresponding Cross Transformer Layer, the V value of the KAM spliced ​​feature is replaced with the V value of the first KAM feature, and then the data output from the two Cross Transformer Layers are spliced ​​to obtain the KAM fusion feature.

[0104] The QF, BF, GAS, and KAM fusion features are fed into the fourth Transformer encoder layer, which consists of four parallel encoder layers, one for QF, one for BF, one for GAS, and one for KAM. The outputs of these four encoder layers are fed into four multilayer perceptrons (MLPs), one for QF, one for BF, one for GAS, and one for KAM. The outputs of these four MLPs are the predicted target QF muscle force, target BF muscle force, target GAS muscle force, and target KAM, respectively.

[0105] Since the position information of IMU data and sEMG data will change during the cross-modal fusion process, inputting QF fusion features, BF fusion features, GAS fusion features and KAM fusion features into the fourth Transformer encoder layer can enable the model to learn the temporal information of the target.

[0106] In the above embodiment, by using a cross-modal attention module to process IMU data and sEMG data, an attention map based on one modality can be calculated and used in another modality. At the same time, information can be exchanged between modalities, and the feature space (semantics) of the current modality will not change. At the same time, different cross-modal fusion schemes are used for muscle strength and KAM to cope with the differences in biomechanical calculations. While fully retrieving biomechanical information of muscle activity and movement information, sEMG data and IMU data can be better integrated to improve prediction performance and reduce calibration work.

[0107] In one embodiment, a multi-task training scheme is adopted during the training process of the biomechanically inspired fusion framework. Specifically, Figure 10 As shown in FIG, after obtaining the first QF muscle strength feature, the first BF muscle strength feature, the first GAS muscle strength feature, and the first KAM feature, the first QF muscle strength feature, the first BF muscle strength feature, the first GAS muscle strength feature, and the first KAM feature are input into the multi-task learning module. Specifically, the first QF muscle strength feature, the first BF muscle strength feature, the first GAS muscle strength feature, and the first KAM feature are respectively input into the corresponding MLP to obtain the muscle contraction geometry (MCG) corresponding to QF, the MCG corresponding to BF, the MCG corresponding to GAS, and the motion information corresponding to KAM. Subsequently, the parameters of the biomechanically inspired fusion framework are optimized based on the MCG corresponding to QF, the MCG corresponding to BF, the MCG corresponding to GAS, and the motion information corresponding to KAM. Among them, MCG includes muscle fiber length and velocity. The motion information can be related joint angle change information, such as hip adduction, hip rotation, knee abduction, and ankle angle change. The multi-task learning scheme can help the model focus on the common features between different tasks before the task regressor is separated, and focus on the specific features of the task after separation. Using a multi-task learning scheme in the prediction of muscle strength and KAM can improve the feature extraction ability and modeling ability of the main tasks.

[0108] During the training process of the biomechanics-inspired fusion framework, the loss function is:

[0109]

[0110] Here, y represents the target metric sequence, f(x) represents the predicted value, and δ represents a hyperparameter, such as 1.0.

[0111] To balance different tasks in the multi-task framework, we enforce the loss rule by dynamically weighting task i with the inverse of the current loss term (denoted as αi), expressed as:

[0112] in, No gradient backpropagation is performed.

[0113] The fusion model provided in this application is obtained by training with a data set. For example, by collecting gait data from 21 KOA patients and 17 healthy subjects, a large gait data set of 9686 gait cycles with different speed settings is formed. Among them, the data set can be used for six common gait retraining strategies. Specifically, the subject is located in a room equipped with nine Qualisys Miqus cameras and an additional Qualisys Miqus camera for motion capture, and walks on a Bertec split-belt treadmill. Based on the information collected by the camera, the subject's motion can be tracked to obtain a motion trajectory. The Bertec split-belt treadmill can be used to measure the subject's GRF. Based on the motion trajectory and GRF, muscle strength can be calculated. At the same time, KAM can be calculated using existing technology.

[0114] The model was trained using the above dataset and its output data was analyzed. The average normalized root mean square error (NRMSE) for KAM estimation was 9.94%, and the average NRMSE for the estimation of the three major lower limb muscle strength was 8.64%, which is comparable to the performance of models that require calibration in the existing technology.

[0115]

[0116] Table 1 shows the comparison results of muscle strength, KAM and other schemes predicted by the wearable device (Knee Guard) of the present application. The prediction results are expressed by the NRMSE corresponding to KAM, QF, BF, and GAS and the corresponding average values. Other schemes include a scheme that uses a convolutional neural network (CNN) to fuse IMU data and sEMG data (IMU&sEMG(CNN)), a scheme that uses a long short-term memory network (LSTM) to extract features from IMU data (IMU(CNN-LSTM)), a scheme that uses an MLP scheme to extract features from IMU data (IMU(MLP)), a scheme that uses CNN to extract features from sEMG data (sEMG(CNN)), and a scheme that uses LSTM to extract features from sEMG data (sEMG(LSTM)).

[0117] It can be seen that compared with solutions using only IMU data or only sEMG data, the wearable device provided by this application has significant improvements in both KAM and muscle force prediction, indicating that IMU data and sEMG data can provide complementary information for monitoring tasks. In particular, compared with solutions using only sEMG data, our system reduces the error by approximately 50%, demonstrating the design effectiveness of the sEMG array in feature extraction. In contrast, in these tasks, the wearable device provided by this application can fully utilize the potential of IMU and sEMG modality fusion, showing excellent performance consistent with the calibration results.

[0118] In one embodiment, if Figure 11As shown, the present application also provides a gait monitoring system, including a sEMG feature extraction module, an IMU feature extraction module and a feature fusion module. The sEMG feature extraction module is used to extract features from sEMG data to obtain sEMG features. The IMU feature extraction module is used to extract features from IMU data to obtain IMU features. The feature fusion module is used to determine a first muscle force feature and a first KAM feature based on the IMU feature, the first muscle force feature including a first QF muscle strength feature, a first BF muscle strength feature and a first GAS muscle strength feature, and to determine a second muscle force feature and a second KAM feature based on the sEMG feature. The second muscle force feature includes a second QF muscle strength feature, a second BF muscle strength feature and a second GAS muscle strength feature. The feature fusion module is also used to fuse the first muscle strength feature and the second muscle strength feature based on a cross-modal attention mechanism to obtain a target muscle strength. The target muscle strength includes a target QF muscle strength, a target BF muscle strength and a target GAS muscle strength. The feature fusion module is also used to fuse the first muscle strength feature and the second KAM feature based on the cross-modal attention mechanism to obtain the KAM splicing feature, and then fuse the KAM splicing feature with the first KAM feature based on the cross-modal attention mechanism to obtain the target KAM. During the training of the feature fusion module, the MCG is determined based on the first QF muscle strength feature, the first BF muscle strength feature, and the first GAS muscle strength feature, and the motion information is determined based on the first KAM feature, thereby performing multi-task learning.

[0119] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0120] Figure 12 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 12 As shown, the electronic device of this embodiment includes: a processor 11, a memory 12, and a computer program 13 stored in the memory 12 and executable on the processor 11. When the processor 11 executes the computer program 13, the steps in the above-mentioned gait monitoring method embodiment are implemented, such as Figure 1 Steps S101 to S102 are shown.

[0121] Exemplarily, the computer program 13 may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 11 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 13 in the electronic device.

[0122] Those skilled in the art will understand that Figure 12 These are merely examples of electronic devices and do not constitute a limitation of the electronic device. The electronic device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0123] The processor 11 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0124] The memory 12 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. The memory 12 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Furthermore, the memory 12 may also include both an internal storage unit of the electronic device and an external storage device. The memory 12 is used to store the computer program and other programs and data required by the electronic device. The memory 12 may also be used to temporarily store data that has been output or is to be output.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0126] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0127] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0129] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0130] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A gait monitoring method, characterized in that: include: Obtain IMU data and sEMG data from the wearable device worn on the leg; The IMU data and the sEMG data are input into a fusion model to obtain the target muscle strength and target KAM output by the fusion model; wherein the fusion model is used to extract features from the IMU data and the sEMG data, and fuse the extracted features based on a cross-modal attention mechanism.

2. The method according to claim 1, characterized in that Inputting the IMU data and the sEMG data into a fusion model to obtain a target muscle strength and a target KAM output by the fusion model includes: Performing feature extraction on the IMU data to obtain a first muscle strength feature and a first KAM feature; Performing feature extraction on the sEMG data to obtain a second muscle strength feature and a second KAM feature; fusing the first muscle strength feature and the second muscle strength feature based on a cross-modal attention mechanism to obtain a target muscle strength; The first KAM feature and the second KAM feature are fused based on the cross-modal attention mechanism to obtain the target KAM.

3. The method according to claim 2, characterized in that The first KAM feature and the second KAM feature are fused based on the cross-modal attention mechanism to obtain a target KAM, including: The first muscle strength feature and the second KAM feature are fused based on a cross-modal attention mechanism to obtain a KAM splicing feature; The KAM concatenation feature and the first KAM feature are fused based on a cross-modal attention mechanism to obtain the target KAM.

4. The method according to claim 3, characterized in that The muscle strength includes QF muscle strength, BF muscle strength and GAS muscle strength.

5. The method according to claim 4, characterized in that The first muscle strength feature and the second KAM feature are fused based on the cross-modal attention mechanism to obtain a KAM splicing feature, including: The first QF muscle strength feature and the second KAM feature are fused based on the cross-modal attention mechanism to obtain the first KAM fusion feature; The first BF muscle strength feature and the second KAM feature are fused based on the cross-modal attention mechanism to obtain the second KAM fusion feature; The first GAS muscle strength feature and the second KAM feature are fused based on the cross-modal attention mechanism to obtain the third KAM fusion feature; The first KAM fusion feature, the second KAM fusion feature and the third KAM fusion feature are spliced ​​to obtain the KAM splicing feature.

6. The method according to claim 2, characterized in that The extracting features of the IMU data to obtain a first muscle strength feature and a first KAM feature includes: Get the first IMU data of the thigh; Get the second IMU data of the calf; Performing feature extraction on the first IMU data and the second IMU data based on a cross attention mechanism to obtain IMU features; The IMU features are sequentially input into the first Transformer encoder layer and the second Transformer encoder layer to obtain the first muscle strength features and the first KAM features.

7. The method according to claim 2, characterized in that The feature extraction of the sEMG data to obtain a second muscle strength feature and a second KAM feature includes: Acquire multiple channels of sEMG data collected during the gait cycle; Performing feature extraction on the sEMG data of the multiple channels based on a channel attention mechanism to obtain sEMG features; The sEMG feature is input into the third Transformer encoder layer to obtain the second muscle strength feature and the second KAM feature.

8. The method according to claim 1, characterized in that The fusion framework is trained based on a multi-task learning method.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

10. A wearable device, characterized in that: The device comprises an IMU sensor, a multi-channel sEMG sensor, and the electronic device as claimed in claim 9.

Images