An exoskeleton motion intention recognition method based on multi-modal feature fusion and deep reversible consistency learning

Abstract

The application discloses a kind of based on multi-modal feature fusion and deep reversible consistency learning's exoskeleton motion intention recognition method, belong to exoskeleton robot and intelligent interaction control field, including: collecting foot bottom pressure and joint kinematics data, calculate foot bottom pressure center and its derivative feature, construct multi-modal dataset;Respectively construct encoder for each mode, select optimal transformation weight matrix as public priori benchmark by selective priori learning;Based on reversible semantic consistency learning, semantic label is mapped to modal invariant anchor point feature, and the representation learning of encoder is constrained;After constraint, the dynamic weighted fusion of multi-modal feature is carried out, and fusion feature is generated;Joint optimization training is carried out using the total loss function formed by focal loss function and DRCL loss function, and motion intention recognition result is output.The application effectively fills the multi-modal heterogeneity gap, realizes the high-precision robust identification of complex live working motion mode.

Description

Technical Field

[0001] This invention belongs to the field of exoskeleton robot and intelligent interactive control technology, and in particular relates to an exoskeleton motion intention recognition method based on multimodal feature fusion and deep reversible consistency learning. Background Technology

[0002] Currently, exoskeleton robots, as an effective means to enhance human function and reduce exercise load, are gradually expanding their application into special operations. In live-line work on power distribution lines, operators typically need to wear foot straps to climb poles, a process that demands high levels of lower limb muscle strength and joint stability, easily leading to muscle strain and knee injuries. To improve these working conditions, introducing exoskeleton robots into live-line work scenarios and building human-machine collaborative work modes has become an important technological approach. Within this framework, accurately recognizing the operator's movement intentions is a prerequisite for achieving precise exoskeleton assistance. In existing technologies, motion pattern recognition methods based on deep learning have been extensively studied. For example, using convolutional neural networks (CNNs) or long short-term memory networks (LSTMs) to extract and classify features from sensor-collected motion data can, to some extent, recognize routine movements such as walking and climbing slopes, providing fundamental support for exoskeleton control.

[0003] However, directly applying the existing methods to motion intent recognition on exoskeletons used in live-line work in power distribution networks still suffers from the following key technical shortcomings. First, live-line work involves the collaborative discrimination of multimodal signals such as plantar pressure and joint kinematics. There are inherent heterogeneity gaps between different modal data, and traditional CNN or LSTM models have limited ability to fuse multimodal features and lack a dynamic evaluation mechanism for the contribution of each modality. This easily leads to insufficient utilization of key modal information or feature averaging, affecting recognition accuracy. Second, in actual live-line work scenarios, the data distribution of different motion modes exhibits extreme imbalance: basic modes such as walking on flat ground and climbing poles have a high proportion of samples, while transitional states such as standing upright and preparing to climb poles have very few samples. Traditional recognition methods often use the cross-entropy loss function for model optimization. This loss, under class imbalance conditions, easily biases the model towards the majority class samples, resulting in a significant decrease in the recognition rate of minority classes and transitional movements, making it difficult to meet the actual needs for high-precision and robust motion intent recognition in complex high-altitude work scenarios. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes an exoskeleton motion intent recognition method based on multimodal feature fusion and deep reversible consistency learning, thereby resolving the issues present in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for recognizing exoskeleton motion intent based on multimodal feature fusion and deep reversible consistency learning, comprising: S1: Collect multimodal data of the operator wearing the exoskeleton to perform live electrical work. The multimodal data includes plantar pressure data and joint kinematic data. Calculate the plantar pressure center and its derived features based on the plantar pressure data, and combine the plantar pressure center and its derived features with the joint kinematic data to construct a multimodal feature dataset. S2: Construct encoders for each modal data to extract features, and select the optimal matrix from the transformation weight matrices generated by each encoder as a common prior benchmark through selective prior learning; S3: Based on reversible semantic consistency learning, semantic labels are mapped to modality-invariant anchor features using the common prior benchmark, and the representation learning of each encoder is constrained. S4: Dynamically weight and fuse the constrained modal features to generate fused features; S5: Input the fused features into the classifier to predict motion patterns, and use the total loss function composed of the focus loss function and the DRCL loss function to jointly optimize and train the model, and output the motion intention recognition result.

[0006] Preferably, the plantar pressure center and its derived features include the sum of plantar pressure centers, the difference of plantar pressure centers, and the symmetry of plantar pressure centers; wherein, the plantar pressure center is calculated based on the position coordinates and pressure values ​​of each sensor on the sole of the foot.

[0007] Preferably, the formula for calculating the center of plantar pressure is: ; ; in, Let i be the coordinates of sensor i. To measure pressure, It is a non-zero minimum constant.

[0008] Preferably, in the selective prior learning, the prior discrimination loss includes a hardness adjustment factor, which increases with the number of training rounds; the optimal matrix is ​​a transformation weight matrix selected based on the quality score evaluation.

[0009] Preferably, the reversible semantic consistency learning includes: introducing a Mixup feature enhancement mechanism and performing a modality-invariant representation reshaping mechanism; the modality-invariant representation reshaping mechanism uses the generalized inverse of a common prior benchmark to recast the semantic label into a modality-invariant representation.

[0010] Preferably, the DRCL loss function comprises three components: the first component is the mean squared error loss between the constrained intramodal sample features and the modality-invariant representation; the second component is the cosine similarity loss that amplifies the inter-class feature differences; and the third component is the classification loss that maintains the semantic consistency between the sample representation and the true label.

[0011] Preferably, the dynamic weighted fusion includes: stacking the features of each modality to form a feature tensor, calculating the attention weights of each modality through a learnable linear transformation layer and a Softmax activation function, and performing a weighted summation of the features of each modality.

[0012] Preferably, the formula for calculating the attention weight is: ,in It is a weight matrix. It is a bias vector. It is the batch size. This indicates that the expansion starts from the dimension indicated in parentheses, and subsequent dimensions are converted to one dimension. d is the number of modalities, and d is the feature dimension; The formula for calculating the fusion feature is: , Let the attention weights be for the k-th modality. This represents the feature of the k-th mode.

[0013] Preferably, the expression for the focus loss function is: ,in γ represents the probability estimated by the model for the true label, and γ is the focusing parameter; the total loss function is the sum of the focus loss function and the DRCL loss function.

[0014] In a second aspect, the present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: This invention, through reversible semantic consistency learning in step S3, maps semantic labels to modality-invariant anchor features using a common prior benchmark and constrains the representation learning of each encoder. This enhances the compactness of intra-class sample features and expands the differences in inter-class sample features. Combined with the dynamic weighted fusion of constrained modal features in step S4, adaptive integration of multimodal information is achieved. The synergistic effect of these features effectively overcomes the heterogeneity gap between multimodal data, improving the efficiency and robustness of feature fusion.

[0016] This invention utilizes a combined loss function, comprised of the focus loss function and the DRCL loss function, in step S5 to jointly optimize and train the model. The focus loss function dynamically reduces the loss weight of easily classified samples, allowing model training to focus on difficult-to-classify and low-sample motion patterns. The DRCL loss function further enhances the feature discriminative power within and between modes. This combined optimization significantly improves the recognition performance of transitional states and minority class actions under conditions of extreme data imbalance.

[0017] In step S1, this invention collects multimodal data including plantar pressure data and joint kinematic data, and calculates the plantar pressure center and its derived features, providing discriminative information on center of gravity changes and balance states for work postures such as high-altitude climbing and pole holding. Combined with the selective prior learning in step S2 to establish a common prior benchmark, the semantic consistency constraints and dynamic fusion in steps S3 and S4, and the joint optimization training in step S5, the overall solution achieves high-precision and robust recognition of multiple motion modes in live-line work, providing reliable technical support for the precise assistance of exoskeleton devices. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of a multimodal data acquisition and pattern recognition algorithm according to an embodiment of the present invention; wherein, (a) is a data acquisition experiment, (b) is data preprocessing and data decomposition, and (c) is a pattern recognition algorithm; Figure 2 This is a schematic diagram showing the distribution points of the plantar pressure sensors [0-15] of the upper and lower limbs exoskeleton device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of model convergence and overall performance evaluation in an embodiment of the present invention; (a) is the accuracy of the training set, test set, and validation set; (b) is the loss of the training set, test set, and validation set. Figure 4 This is a schematic diagram of the mode confusion matrix in an embodiment of the present invention, wherein (a) is the confusion matrix of the complete model; (b) is the confusion matrix after removing focal loss in the ablation experiment; (c) is the confusion matrix after removing SPL in the ablation experiment; and (d) is the confusion matrix after removing the multimodal attention mechanism in the ablation experiment. Figure 5The following are confusion matrices for imbalanced class samples using different methods according to embodiments of the present invention; wherein, (a) a hybrid model of DRCL and Focal loss is used; (b) an LSTM method is used; (c) a Transformer method is used; (d) a GRU method is used; (e) a BiLSTM method is used; and (f) a CNN1D-LSTM method is used. Figure 6 The following are ROC curves for different methods used on imbalanced class samples in embodiments of the present invention: (a) using a hybrid model of DRCL and Focal loss; (b) using the LSTM method; (c) using the BiLSTM method; (d) using the CNN1D-LSTM method; (e) using the Transformer method; and (f) using the GRU method. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0021] Example 1 like Figure 1 As shown, this embodiment provides a method for recognizing exoskeleton motion intentions based on multimodal feature fusion and deep reversible consistency learning, including: S1: Collect multimodal data of the operator wearing the exoskeleton to perform live electrical work. The multimodal data includes plantar pressure data and joint kinematic data. Calculate the plantar pressure center and its derived features based on the plantar pressure data, and combine the plantar pressure center and its derived features with the joint kinematic data to construct a multimodal feature dataset. Furthermore, the plantar pressure center and its derived features include the sum of plantar pressure centers, the difference of plantar pressure centers, and the symmetry of plantar pressure centers; wherein, the plantar pressure center is calculated based on the position coordinates and pressure values ​​of each sensor on the sole of the foot.

[0022] Specifically, multimodal data was collected from the operator wearing the exoskeleton while performing live-line work tasks. The tasks included five basic modes: walking on flat ground (LW), climbing up a pole (CUP), climbing down a pole (CDP), pole holding (KWP), and walking with foot straps (WFB), as well as two transitional states: standing upright (SU) and preparing to climb up a pole (PBCUP). The collected data included plantar pressure information for both feet (32 channels in total), knee and hip joint position information, and knee and hip joint velocity information.

[0023] Establish a plantar coordinate system and calculate the center of plantar pressure (CoP) data and its derived characteristics. The CoP calculation formula is as follows: (1) (2) Where the X-axis represents the inward / outward direction, the Y-axis represents the forward / backward direction, m represents the number of plantar sensors, and the position coordinates of each sensor i are... The pressure measured at the current time is , To prevent singularly small constants.

[0024] Considering that the instantaneous center of gravity coordinates reflected by the raw Cop value may have limitations in some static models, we introduce a sum of Cop values ​​to reflect the overall center of gravity position, a Cop difference value to reflect the balance, and a Cop symmetry value to reflect the weight-bearing ratio of the left and right feet. The specific definitions are as follows: (3) (4) (5) (6) (7) (8) in, and These represent the Cop values ​​for the left and right feet along the X-axis, respectively. and These represent the Cop values ​​for the left and right feet along the Y-axis, respectively. and This represents the Cop and values ​​along the X and Y axes. and This represents the difference in Cop along the X and Y axes. and This indicates the Cop symmetry along the X and Y axes.

[0025] S2: Construct encoders for each modal data to extract features, and select the optimal matrix from the transformation weight matrices generated by each encoder as a common prior benchmark through selective prior learning; Furthermore, in the selective prior learning, the prior discrimination loss includes a hardness adjustment factor, which increases with the number of training rounds; the optimal matrix is ​​a transformation weight matrix selected based on the quality score evaluation.

[0026] Specifically, we first define a dataset with K modalities as follows: (9) in, For the sample set of the Kth mode, For the corresponding category label set, Represented as the Kth mode of the Kth mode One sample, Indicates sample The tag, This represents the number of samples in the Kth mode. For the number of categories. Furthermore, if Belongs to the kind, ,but The The elements are Otherwise The k-th modality-specific encoder that projects the samples into the common subspace is... , For the public subspace dimension.

[0027] The matrix with the best quality score among the changed weight matrices trained on all modalities is selected and used as a prior to guide representation learning. For the Kth modality, the loss is defined as follows: (10) in, It is the first In the mode, the th The feature representation of each sample in the common subspace yes The Column vector, As a hardness adjustment factor, it helps the model focus on difficult-to-align samples first, and then focus on overall performance, as the number of epochs increases from 0.01 to 1 during training.

[0028] By comparing the quality scores of each weight matrix, the best weight matrix is ​​selected to guide subsequent learning. The Kth modality matrix... The mass fraction is: (11) S3: Based on reversible semantic consistency learning, semantic labels are mapped to modality-invariant anchor features using the common prior benchmark, and the representation learning of each encoder is constrained. Furthermore, the reversible semantic consistency learning includes: introducing a Mixup feature enhancement mechanism and performing a modality-invariant representation reshaping mechanism; the modality-invariant representation reshaping mechanism uses the generalized inverse of a common prior benchmark to recast the semantic labels into modality-invariant representations.

[0029] Introducing feature enhancement mechanisms to enable the model to learn on a wider range of data distributions can be described as follows: (12) (13) in It is a mixing factor.

[0030] To address the common unavailability of paired samples from different modalities, a modality-invariant representation reshaping mechanism (MRR) is proposed. This mechanism utilizes a learned prior matrix W to bridge representation learning across different modalities in a common subspace, thereby obtaining a modality-invariant representation. Specifically, under the assumption that the labels are mapped perfectly accurately, we use the prior W to recast a modality-invariant representation for each semantic category, such that... Approaching 0, it can be represented as: (14) (15) (16) (17) For convenience, Recorded as Then define the loss function for the k-th mode: (18) Considering Since the loss function cannot effectively enhance the similarity and distinguishability between intra-class sample representations, we have developed the following loss function to improve this: (19) in, express and Cosine similarity between them.

[0031] In addition, to maintain semantic consistency between the sample representation and the real labels in the label space, a method is adopted that... The same loss function is shown below: (20) in, yes The c-th column vector, yes The c-th element. Finally, by combining the above three loss functions, the DRCL loss function is as follows: (twenty one) in, and It is a hyperparameter that controls the contribution of different loss function components.

[0032] S4: Dynamically weight and fuse the constrained modal features to generate fused features; Furthermore, the dynamic weighted fusion includes: stacking the features of each modality to form a feature tensor, calculating the attention weights of each modality through a learnable linear transformation layer and a Softmax activation function, and performing a weighted summation of the features of each modality.

[0033] Specifically, to adapt to the complex and ever-changing pole-climbing motion patterns of the exoskeleton, we introduced a multimodal attention fusion mechanism in the RSC stage of DRCL. This mechanism is applied to the motion patterns of exoskeletons through various modal encoders. Features extracted and constrained by DRCL loss We first stack the features of all modalities to form a feature tensor. Subsequently, attention weights for each modality are calculated using a learnable linear transformation layer and a softmax activation function: (twenty two) in It is a weight matrix. It is a bias vector. It is the batch size. This indicates that the expansion starts from the dimension indicated in parentheses, and subsequent dimensions are converted to one dimension. d is the number of modalities, and d is the feature dimension. Feature fusion. We obtain the following by weighted summation of the features of each modality: (twenty three) in Let the attention weights be for the k-th modality. This represents the feature of the k-th mode.

[0034] S5: Input the fused features into the classifier to predict motion patterns, and use the total loss function composed of the focus loss function and the DRCL loss function to jointly optimize and train the model, and output the motion intention recognition result.

[0035] Furthermore, the DRCL loss function comprises three components: the first component is the mean squared error loss between the constrained intramodal sample features and the modality-invariant representation; the second component is the cosine similarity loss that amplifies the inter-class feature differences; and the third component is the classification loss that maintains the semantic consistency between the sample representation and the true label.

[0036] Specifically, the focus loss function is an effective way to handle the problem of extremely imbalanced pattern data during exoskeleton pole climbing. Generally, we first define the cross-entropy loss for binary classification: (twenty four) in The real category was specified. It is the probability estimated by the model for the label when y=1.

[0037] For simplicity, define : (25) Then rewrite Therefore, based on the above, an adjustment factor is added to the cross-entropy loss. Among them, the focusing parameters Adjustable, we define focus loss as: (26) To verify the effectiveness, accuracy, and robustness of the method proposed in this invention, multimodal data acquisition and pattern recognition experiments were conducted on exoskeleton live-line operations in real-world scenarios. The specific analysis is as follows: 1. Experimental setup and data acquisition: Combining Figure 1 The overall flow of the multimodal data acquisition and pattern recognition algorithm shown in the figure begins with data acquisition. Figure 1 a) Preprocessing and solution ( Figure 1 b), and finally fed into the pattern recognition algorithm ( Figure 1 c). In this process, the system depends on Figure 2 The foot pressure sensor [0-15] distribution points shown accurately acquire the foot reaction force, providing a reliable hardware data source for subsequent CoP and its derived features calculation.

[0038] 2. Model Convergence and Overall Performance Evaluation: The constructed multimodal feature dataset is input into the hybrid model of this invention for training. Figure 3As shown in (a) the accuracy curve and (b) the loss curve, the model learns rapidly in the early stages of training and tends to converge after a limited number of iterations (epochs). The final accuracy on both the test and validation sets is stable at around 98.8%, and the loss curve is smooth without significant fluctuations, indicating that the model has excellent fitting ability, without overfitting or underfitting, and has the generalization conditions for deployment in actual control systems.

[0039] 3. Ablation Experiments and Module Effectiveness Verification: To isolate and verify the contribution of each innovative module to the overall system performance, ablation experiments were conducted. The results are as follows: Figure 4 The pattern confusion matrix is ​​shown. A comparison reveals that when using the complete model ( Figure 4 In the case of 'a', the vast majority of samples were correctly classified; however, when Focal loss was removed... Figure 4 In case b), severe class confusion occurred when facing the "KWP (Keeping on Pole)" mode with very few samples; removing the SPL module ( Figure 4 c) or removing the multimodal attention mechanism ( Figure 4 Following step d), the model also exhibited a significant increase in misclassification rate when handling similar transitional actions. This fully demonstrates the irreplaceable necessity of the synergistic work of deep reversible consistency learning, multimodal attention, and focus loss in processing complex heterogeneous data and mitigating class imbalance.

[0040] 4. Comparative Experiments and Verification of Advanced Features: To fully demonstrate the advanced features of the hybrid model of this invention, it was compared with classic deep learning temporal models (such as LSTM, Transformer, GRU, BiLSTM, and CNN1D-LSTM). Figure 5 The confusion matrices of different methods used for imbalanced class samples show that traditional time-series methods ( Figure 5 In dealing with complex high-altitude operation modes, the hybrid model (bf) in this invention commonly makes numerous misjudgments between "climbing the pole (CUP)" and "holding the pole (KWP)," indicating that a single modeling method cannot fully utilize multimodal information. In contrast, the hybrid model of this invention (bf) Figure 6 a) Effectively suppressed the number of misclassifications in minority and difficult-to-classify categories. Furthermore, such as... Figure 6 As shown in the ROC curve, the model of this invention ( Figure 6 In example a), the classification performance curve of model a) is closest to the top left corner across all categories and exhibits stable performance across different thresholds. Its overall AUC (area under the curve) is significantly better than other comparative models. Figure 6 (bf in the middle).

[0041] Example 2 This embodiment also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0042] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for recognizing exoskeleton motion intent based on multimodal feature fusion and deep reversible consistency learning, characterized in that, Includes the following steps: S1: Collect multimodal data of the operator wearing the exoskeleton to perform live electrical work. The multimodal data includes plantar pressure data and joint kinematic data. Calculate the plantar pressure center and its derived features based on the plantar pressure data, and combine the plantar pressure center and its derived features with the joint kinematic data to construct a multimodal feature dataset. S2: Construct encoders for each modal data to extract features, and select the optimal matrix from the transformation weight matrices generated by each encoder as a common prior benchmark through selective prior learning; S3: Based on reversible semantic consistency learning, semantic labels are mapped to modality-invariant anchor features using the common prior benchmark, and the representation learning of each encoder is constrained. S4: Dynamically weight and fuse the constrained modal features to generate fused features; S5: Input the fused features into the classifier to predict motion patterns, and use the total loss function composed of the focus loss function and the DRCL loss function to jointly optimize and train the model, and output the motion intention recognition result.

2. The method according to claim 1, characterized in that, The plantar pressure center and its derived features include the sum of plantar pressure centers, the difference of plantar pressure centers, and the symmetry of plantar pressure centers; wherein, the plantar pressure center is calculated based on the position coordinates and pressure values ​​of each sensor on the sole of the foot.

3. The method according to claim 2, characterized in that, The formula for calculating the center of pressure on the foot is: ； ； in, Let i be the coordinates of sensor i. To measure pressure, It is a non-zero minimum constant.

4. The method according to claim 1, characterized in that, In the selective prior learning, the prior discrimination loss includes a hardness adjustment factor, which increases with the number of training rounds; the optimal matrix is ​​a transformation weight matrix selected based on the quality score evaluation.

5. The method according to claim 1, characterized in that, The reversible semantic consistency learning includes: introducing a Mixup feature enhancement mechanism and performing a modality-invariant representation reshaping mechanism; the modality-invariant representation reshaping mechanism uses the generalized inverse of a common prior benchmark to recast semantic labels into modality-invariant representations.

6. The method according to claim 5, characterized in that, The DRCL loss function comprises three components: the first component is the mean squared error loss between the constrained intramodal sample features and the modality-invariant representation; the second component is the cosine similarity loss that amplifies the inter-class feature differences; and the third component is the classification loss that maintains the semantic consistency between the sample representation and the true label.

7. The method according to claim 1, characterized in that, The dynamic weighted fusion includes: stacking the features of each modality to form a feature tensor, calculating the attention weights of each modality through a learnable linear transformation layer and a Softmax activation function, and performing a weighted summation of the features of each modality.

8. The method according to claim 7, characterized in that, The formula for calculating the attention weight is: ,in It is a weight matrix. It is a bias vector. It is the batch size. This indicates that the expansion starts from the dimension indicated in parentheses, and subsequent dimensions are converted to one dimension. d is the number of modalities, and d is the feature dimension; The formula for calculating the fusion feature is: , Let the attention weights be for the k-th modality. This represents the feature of the k-th mode.

9. The method according to claim 1, characterized in that, The expression for the focus loss function is: ,in γ represents the probability estimated by the model for the true label, and γ is the focusing parameter; the total loss function is the sum of the focus loss function and the DRCL loss function.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-9.

Images