An exoskeleton gait feature recognition method based on multi-modal information fusion representation

Abstract

The present disclosure relates to a multi-modal information fusion representation exoskeleton gait feature recognition method and device, electronic equipment and storage medium. The method comprises: calibrating and initializing the sensor of the exoskeleton wearer coupling system; collecting exoskeleton sensor signals, calculating multi-order derivatives and storing as time series; constructing a multi-layer stack denoising sparse auto-encoding neural network model; constructing a parallel neural network model for space-time domain feature hybrid extraction; based on a genetic-particle swarm hybrid optimization method, the weight vector of the cross-entropy function in the above model is optimized and searched, the mode search of the gait feature label vector predicted by the model in the previous n time is carried out, and the abnormal value is filtered and recognized as the gait feature evaluation output of post-processing. The present disclosure realizes that the recognition accuracy of the label reaches the target optimal value, effectively avoids the oscillation step caused by the recognition abnormal value, and effectively improves the recognition accuracy, robustness and generalization ability of the exoskeleton for gait features.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of robots, in particular, to a multi-modal information fusion representation exoskeleton gait feature recognition method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] With the rapid development of robot technology, exoskeleton robots have been widely applied to various application scenarios. Exoskeletons not only need to provide function enhancement for wearers under different limb movements in power output, but also need to provide perception prediction for wearers under different movement scenarios in intention decision. How the exoskeleton recognizes different gait features of the current wearer according to the multi-modal movement interaction sensing information of the human-machine system reaction is directly related to whether the exoskeleton robot can effectively perceive the gait mode and gait phase of the wearer currently in, and further affects whether the exoskeleton can smoothly and accurately switch to the corresponding controller model to assist the wearer to complete the movement assistance in the corresponding task scenario.

[0003] Gait phase recognition mainly depends on the signal distribution of the foot pressure. At present, most foot detection sensors have few signal nodes. Although the pressure detection based on single-point stress sensors is simple and reliable, its structure is large in size, the space contact is weak, and when the palm of the foot is bent, the signal reaction is not sensitive due to insufficient contact. At present, there is a lack of continuous detection and multi-dimensional characteristic extraction analysis of the overall palm stress distribution, and the threshold judgment method is easy to produce oscillation and jitter when switching, which is the research focus of gait phase recognition. The multi-modal information possessed by the exoskeleton itself in gait mode recognition is sufficient to fully reflect the kinematic characteristics and dynamic properties of the current system. How to fully utilize the current time information of the signal source to quickly extract features, accurately fuse information, and efficiently represent is the research focus of gait mode recognition. If too many sensor nodes are added, it will lead to low data acquisition frequency and slow signal transmission speed, affecting the real-time performance of recognition. How to reasonably use the typical amplitude-frequency characteristic interaction signal of the exoskeleton itself to obtain low response delay and high-dimensional difference feature information, design a dimension reduction feature expression method for complex input, improve the noise anti-interference ability, and filter the step jump in abnormal prediction, so as to realize efficient fusion representation and accurate recognition switching of multi-modal information is a problem to be solved. At present, there are few exoskeleton gait feature recognition methods in the field of robot movement intention perception that can adapt to smooth and efficient prediction switching between continuous different gait features.

[0004] Therefore, one or more methods are needed to solve the above problems.

[0005] It should be noted that the information disclosed in the above BACKGROUND section is only for the purpose of strengthening the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The purpose of the present disclosure is to provide a multi-modal information fusion representation exoskeleton gait feature recognition method, device, electronic equipment and computer readable storage medium, thereby at least to some extent overcoming one or more problems caused by the limitations and defects of the related art.

[0007] According to one aspect of the present disclosure, a multi-modal information fusion representation exoskeleton gait feature recognition method is provided, comprising:

[0008] Calibration and initialization processing of control programs of a back IMU sensor, joint absolute encoder, wearer foot bottom flexible film array insole, exoskeleton foot bottom contact force sensor of an exoskeleton wearer coupling system;

[0009] Based on the back IMU sensor, joint absolute encoder, wearer foot bottom flexible film array insole, exoskeleton foot bottom contact force sensor, exoskeleton sensor signals are collected, multi-order derivatives are calculated and stored as time series, and a multi-modal fusion information expansion matrix representing space-time domain features is constructed;

[0010] Based on the parameterized representation characteristic trend of KL divergence analysis, a loss function is designed and a sparsification correction term is introduced, a random denoising controller is designed for hierarchical nodes, and a multi-layer stack denoising sparse auto-encoding neural network model is constructed;

[0011] A node activation function based on CNN spatial domain feature extraction is designed, a node transfer structure unit based on GRU time domain feature extraction is designed, a multi-layer feature mixing connection method and a feature fusion function are designed, and a parallel neural network model for spatial-time domain feature mixing extraction is constructed;

[0012] Based on the genetic-particle swarm hybrid optimization method, the weight vector of the cross-entropy function in the multi-layer stack denoising sparse auto-encoding neural network model and the spatial-time domain feature mixing extraction parallel neural network model is optimized and searched to make the accuracy of the recognition model reach a preset value, the gait feature label vector predicted by the model at the first n time is searched for the mode, and the recognition outliers are filtered as the post-processing gait feature evaluation output.

[0013] In an exemplary embodiment of the present disclosure, the method further comprises:

[0014] Initializing, initial bias extraction, reference zero value calibration processing of the control program of the back IMU sensor, joint absolute encoder, wearer foot bottom flexible film array insole, exoskeleton foot bottom contact force sensor of the exoskeleton wearer coupling system.

[0015] In an exemplary embodiment of the present disclosure, the method further comprises:

[0016] Based on the exoskeleton sensor signals collected by the back IMU sensor, joint absolute encoder, wearer foot bottom flexible film array insole, exoskeleton foot bottom contact force sensor;

[0017] Based on the exoskeleton sensor signals, calculate the multi-order derivative and store the multi-order derivative as a time series;

[0018] Construct a multi-modal information feature vector and perform image compilation and expansion;

[0019] Collect the contact state matrix of the wearer foot bottom flexible array insole, perform data expansion on the discrete sparse point array based on Gaussian radial basis function nonlinear interpolation, and construct a multi-modal fusion information expansion matrix representing space-time features.

[0020] In an exemplary embodiment of the present disclosure, the method further comprises:

[0021] After normalizing and two-dimensional copying and expanding the multi-modal information feature vector, perform image compilation and expansion to generate an image matrix.

[0022] In an exemplary embodiment of the present disclosure, the method further comprises:

[0023] Based on the autoencoder, train the multi-layer stack denoising sparse autoencoding neural network model in stack order;

[0024] Train the training output of the autoencoder as the input of the final classifier;

[0025] Integrate and adjust the training output of the final classifier based on the weight to complete the classification training of the data set based on the multi-layer stack denoising sparse autoencoding neural network model.

[0026] In an exemplary embodiment of the present disclosure, the method further comprises:

[0027] Using a wearer foot bottom flexible film array insole based on a flexible circuit board to collect the pressure change trend along the foot width and foot length between the wearer's foot bottom and the exoskeleton, and generate a pressure change point array;

[0028] Perform data expansion on the pressure change point array based on a two-dimensional nonlinear interpolation method of Gaussian radial basis function.

[0029] In one exemplary embodiment of this disclosure, the method further includes:

[0030] The generalization ability of the trained model is tested on non-training samples, and the gradient of the SPM resultant force is compared with the gradient threshold. The model is optimized based on the judgment of whether the gait phase at the current moment is in the true forefoot take-off phase.

[0031] In one aspect of this disclosure, an exoskeleton gait feature recognition device based on multimodal information fusion representation is provided, comprising:

[0032] The sensor initialization module is used for calibration and initialization of the control program of the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor of the exoskeleton wearer coupling system.

[0033] The data matrix construction module is used to collect exoskeleton sensor signals based on back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor, calculate multi-order derivatives and store them as time series, and construct a multimodal fusion information extension matrix that represents spatial-temporal features;

[0034] The first modeling module is used to design a loss function and introduce a sparsity correction term based on the trend of parameterized representation characteristics of KL discreteness, design a random denoising controller for hierarchical nodes, and construct a multi-layer stacked denoising sparse autoencoder neural network model.

[0035] The second modeling module is used to design node activation functions based on CNN spatial domain feature extraction, design node transmission structure units based on GRU temporal domain feature extraction, design multi-layer feature hybrid connection methods and feature fusion functions, and construct a parallel neural network model for spatial-temporal feature hybrid extraction.

[0036] The gait feature evaluation module is used to optimize the weight vector of the cross-entropy function in a multi-layer stacked denoising sparse autoencoder neural network model and a parallel neural network model with spatial-temporal feature hybrid extraction based on the genetic-particle swarm optimization method, so as to make the accuracy of the identification model reach a preset value. The module performs a mode search on the gait feature label vectors predicted by the model in the first n time steps and filters out the identification outliers as the gait feature evaluation output for post-processing.

[0037] In one aspect of this disclosure, an electronic device is provided, comprising:

[0038] Processor; and

[0039] A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of the preceding claims.

[0040] In one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the preceding claims.

[0041] An exemplary embodiment of this disclosure discloses a method for identifying exoskeleton gait features using multimodal information fusion representation. The method includes: calibration and initialization processing of the control program for the back IMU sensor, joint absolute encoder, wearer's flexible thin-film array insole, and exoskeleton foot contact force sensor of the exoskeleton wearer coupling system; acquiring exoskeleton sensor signals based on the back IMU sensor, joint absolute encoder, wearer's flexible thin-film array insole, and exoskeleton foot contact force sensor, calculating multi-order derivatives and storing them as time series to construct a multimodal fusion information extension matrix representing spatial-temporal features; and designing a loss function and introducing a sparsity correction term based on the analysis of the parametric representation characteristics trend of KL dispersion, and performing randomization on the hierarchical nodes. This paper presents a denoising controller design, constructing a multi-layer stacked denoising sparse autoencoder neural network model. It designs node activation functions based on CNN spatial domain feature extraction, node propagation structures based on GRU temporal feature extraction, and multi-layer feature hybrid connection methods and feature fusion functions to construct a parallel neural network model for spatial-temporal feature hybrid extraction. Based on a genetic-particle swarm optimization method, the weight vectors of the cross-entropy function in both the multi-layer stacked denoising sparse autoencoder neural network model and the parallel neural network model for spatial-temporal feature hybrid extraction are optimized to achieve a preset accuracy. A mode search is performed on the gait feature label vectors predicted by the model in the first n time steps to filter out identification outliers as the gait feature evaluation output for post-processing. This disclosure achieves optimal recognition accuracy for each category of labels while effectively avoiding oscillating steps caused by identification outliers, thus significantly improving the accuracy, robustness, and generalization ability of the exoskeleton for gait feature recognition.

[0042] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0043] The above and other features and advantages of this disclosure will become more apparent from the detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0044] Figure 1 A flowchart is shown below illustrating an exoskeleton gait feature identification method based on a multimodal information fusion representation according to an exemplary embodiment of the present disclosure;

[0045] Figure 2 A human-machine system sensor distribution diagram is shown for an exoskeleton gait feature identification method based on a multimodal information fusion representation according to an exemplary embodiment of the present disclosure.

[0046] Figure 3 A schematic diagram of the multimodal information fusion representation algorithm is shown, illustrating the gait pattern identification process of an exoskeleton gait feature identification method according to an exemplary embodiment of the present disclosure.

[0047] Figure 4 The diagram illustrates a denoising-sparse-stacked autoencoder neural network model structure for gait pattern recognition, which is an exoskeleton gait feature recognition method based on multimodal information fusion representation according to an exemplary embodiment of the present disclosure.

[0048] Figure 5 A schematic diagram of the multimodal information fusion representation algorithm is shown, illustrating the gait phase identification process of an exoskeleton gait feature identification method according to an exemplary embodiment of the present disclosure.

[0049] Figure 6 The diagram illustrates a parallel neural model structure for gait phase identification based on spatial-temporal feature hybrid extraction, which is an exoskeleton gait feature identification method for multimodal information fusion representation according to an exemplary embodiment of the present disclosure.

[0050] Figure 7 A structural block diagram of an exoskeleton gait feature identification device based on a multimodal information fusion representation according to an exemplary embodiment of the present disclosure is shown.

[0051] Figure 8 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown; and

[0052] Figure 9 The illustration shows a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0053] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0054] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details described, or other methods, components, materials, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, materials, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0055] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.

[0056] In this example embodiment, a method for identifying exoskeleton gait features based on multimodal information fusion is first provided; reference Figure 1 As shown, the exoskeleton gait feature identification method based on multimodal information fusion representation may include the following steps:

[0057] Step S110 involves the calibration and initialization of the control program for the back IMU sensor, joint absolute encoder, wearer's flexible thin-film array insole, and exoskeleton foot contact force sensor of the exoskeleton wearer coupling system.

[0058] Step S120: Based on the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor, the exoskeleton sensor signals are collected, multi-order derivatives are calculated and stored as time series, and a multimodal fusion information extension matrix representing spatial-temporal features is constructed.

[0059] Step S130: Based on the analysis of the parameterized characterization trend of KL discreteness, design a loss function and introduce a sparsity correction term, design a stochastic denoising controller for hierarchical nodes, and construct a multi-layer stacked denoising sparse autoencoder neural network model.

[0060] Step S140: Design a node activation function based on CNN spatial domain feature extraction, design a node propagation structure unit based on GRU temporal domain feature extraction, design a multi-layer feature hybrid connection method and feature fusion function, and construct a parallel neural network model for spatial-temporal feature hybrid extraction.

[0061] Step S150: Based on the genetic-particle swarm optimization method, the weight vector of the cross-entropy function in the multi-layer stacked denoising sparse autoencoder neural network model and the parallel neural network model with spatial-temporal feature hybrid extraction is optimized and searched to make the accuracy of the identification model reach the preset value. The mode search is performed on the gait feature label vectors predicted by the model in the first n time steps, and outliers are filtered out as the gait feature evaluation output for post-processing.

[0062] An exemplary embodiment of this disclosure discloses a method for identifying exoskeleton gait features using multimodal information fusion representation. The method includes: calibration and initialization processing of the control program for the back IMU sensor, joint absolute encoder, wearer's flexible thin-film array insole, and exoskeleton foot contact force sensor of the exoskeleton wearer coupling system; acquiring exoskeleton sensor signals based on the back IMU sensor, joint absolute encoder, wearer's flexible thin-film array insole, and exoskeleton foot contact force sensor, calculating multi-order derivatives and storing them as time series to construct a multimodal fusion information extension matrix representing spatial-temporal features; and designing a loss function and introducing a sparsity correction term based on the analysis of the parametric representation characteristics trend of KL dispersion, and performing randomization on the hierarchical nodes. This paper presents a denoising controller design, constructing a multi-layer stacked denoising sparse autoencoder neural network model. It designs node activation functions based on CNN spatial domain feature extraction, node propagation structures based on GRU temporal feature extraction, and multi-layer feature hybrid connection methods and feature fusion functions to construct a parallel neural network model for spatial-temporal feature hybrid extraction. Based on a genetic-particle swarm optimization method, the weight vectors of the cross-entropy function in both the multi-layer stacked denoising sparse autoencoder neural network model and the parallel neural network model for spatial-temporal feature hybrid extraction are optimized to achieve a preset accuracy. A mode search is performed on the gait feature label vectors predicted by the model in the first n time steps to filter out identification outliers as the gait feature evaluation output for post-processing. This disclosure achieves optimal recognition accuracy for each category of labels while effectively avoiding oscillating steps caused by identification outliers, thus significantly improving the accuracy, robustness, and generalization ability of the exoskeleton for gait feature recognition.

[0063] The following will further explain a method for identifying exoskeleton gait features based on multimodal information fusion representation in this example embodiment.

[0064] Example 1:

[0065] In step S110, calibration and initialization processing can be performed on the control program of the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor of the exoskeleton wearer coupling system.

[0066] In this example embodiment, the method further includes:

[0067] The control programs for the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor of the exoskeleton wearer coupling system are initialized, initial bias is extracted, and reference zero value is calibrated.

[0068] In step S120, exoskeleton sensor signals can be collected based on the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor. Multi-order derivatives are calculated and stored as time series to construct a multimodal fusion information extension matrix that characterizes spatial-temporal features.

[0069] In this example embodiment, exoskeleton sensor signals are acquired based on the back IMU sensor, joint absolute encoder, wearer's flexible thin-film array insole, and exoskeleton foot contact force sensor.

[0070] Based on the exoskeleton sensor signals, multi-order derivatives are calculated and stored as time series.

[0071] Construct multimodal information feature vectors and perform image-based compilation and expansion;

[0072] The contact state matrix of the flexible array insole on the sole of the wearer's foot is collected. The discrete sparse lattice is extended by nonlinear interpolation based on Gaussian radial basis function to construct a multimodal fusion information extension matrix that represents spatial-temporal features.

[0073] In this example embodiment, the method further includes:

[0074] After normalizing and performing two-dimensional copying and expansion on the multimodal information feature vectors, image-based compilation and expansion are performed to generate an image matrix.

[0075] In step S130, based on the analysis of the parameterized characterization trend of KL discreteness, a loss function can be designed and a sparsity correction term can be introduced. A stochastic denoising controller can be designed for the hierarchical nodes to construct a multi-layer stacked denoising sparse autoencoder neural network model.

[0076] In this example embodiment, the method further includes:

[0077] The multi-layer stacked denoising sparse autoencoder neural network model is trained according to the stack order based on the autoencoder;

[0078] The training output of the autoencoder is used as the input to the final classifier for training;

[0079] The training output of the final classifier is integrated and adjusted based on the weights to complete the classification training of the dataset based on the multi-layer stacked denoising sparse autoencoder neural network model.

[0080] In step S140, a node activation function based on CNN spatial domain feature extraction can be designed, a node transfer structure unit based on GRU temporal domain feature extraction can be designed, a multi-layer feature hybrid connection method and feature fusion function can be designed, and a parallel neural network model for spatial-temporal feature hybrid extraction can be constructed.

[0081] In this example embodiment, the method further includes:

[0082] A flexible thin-film array insole based on a flexible circuit board was used to collect the pressure change trend between the wearer's foot and exoskeleton along the foot width and foot length, and generate a pressure change dot matrix.

[0083] The pressure variation lattice is expanded using a two-dimensional nonlinear interpolation method based on Gaussian radial basis functions.

[0084] In step S150, the weight vector of the cross-entropy function in the multi-layer stacked denoising sparse autoencoder neural network model and the parallel neural network model with spatial-temporal feature hybrid extraction can be optimized and searched based on the genetic-particle swarm optimization method so that the accuracy of the identification model reaches the preset value. The mode search is performed on the gait feature label vectors predicted by the model in the first n time steps, and outliers are filtered out as the gait feature evaluation output for post-processing.

[0085] In this example embodiment, the method further includes:

[0086] The generalization ability of the trained model is tested on non-training samples, and the gradient of the SPM resultant force is compared with the gradient threshold. The model is optimized based on the judgment of whether the gait phase at the current moment is in the true forefoot take-off phase.

[0087] Example 2:

[0088] In the embodiments of this example, as shown in the appendix Figure 2As shown, the gait feature recognition method of the wearer-exoskeleton human-machine coupling system involves sensors including a back IMU Ⅰ, a hip joint encoder Ⅱ, a knee joint encoder Ⅲ, a plantar contact force sensor Ⅳ, and a flexible thin-film array insole Ⅴ. The plantar contact force sensor Ⅳ primarily measures the contact stress between the wearer's shoe sole and the exoskeleton's foot sole, while the flexible thin-film array insole Ⅴ primarily measures the contact state between the wearer's foot sole and the wearer's shoe sole. Because the actual contact state between the wearer's foot and the ground is affected by the material properties of the shoes and the exoskeleton's foot sole, as well as the rigidity of the stress sensor, the plantar contact force sensor Ⅳ, while reflecting the current actual contact force magnitude, exhibits a phase lag in its sensitivity and response speed compared to the wearer's true intention. Therefore, placing the flexible thin-film array insole Ⅴ on the wearer's foot sole can effectively detect the wearer's foot movement state and movement intention.

[0089] A method for identifying exoskeleton gait features based on multimodal information fusion representation, such as Figure 3 The method described includes:

[0090] Step S110: Complete the initialization of the control program for the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor of the exoskeleton-wearer coupling system, extract the initial bias, and calibrate the reference zero value.

[0091] Step S120: Collect signals from multiple sensors on the exoskeleton, calculate multi-order derivatives and store them as time series, construct a multimodal information feature vector X and perform image-based compilation and expansion, collect the contact state matrix of the wearer's flexible array insole, and expand the discrete sparse lattice data based on Gaussian radial basis function nonlinear interpolation.

[0092] Step S130: Analyze the parameterized representation characteristics trend of KL discreteness, design the loss function and introduce a sparsity correction term, design a stochastic denoising controller for the hierarchical nodes, and construct a multi-layer stacked denoising sparse autoencoder neural network model.

[0093] Step S140: Design a node activation function based on CNN spatial domain feature extraction, design a node propagation structure unit based on GRU temporal domain feature extraction, design a multi-layer feature hybrid connection method and feature fusion function, and construct a parallel neural network model for spatial-temporal feature hybrid extraction;

[0094] Step S150: The genetic-particle swarm optimization method (hybrid GA-PSO, HGP) is used to optimize the weight vector of the cross-entropy function in the neural network model obtained in the third and fourth steps, so that the accuracy of the identification model reaches the optimal value. The mode search is performed on the gait feature label vectors predicted by the model in the first n time steps, and outliers are filtered out as the gait feature evaluation output for post-processing.

[0095] In step S120, this method selects interactive signals that are independent of physical binding and force and have typical amplitude-frequency characteristic differences, including joint angles θ of knee extension / flexion, hip extension / flexion, and hip internal / external rotation of the exoskeleton based on Hall magnetic field rotation detection. Joint and joint angular velocity The exoskeleton back linear acceleration a detected by the inertial measurement unit (IMU) Back Pitch / roll attitude angle ζ IMU and attitude angular velocity And the ground reaction force pressure signal F between the wearer's shoe sole and the exoskeleton shoe sole based on elastomer strain detection. GRF Sensor arrangement as follows Figure 2 As shown, the interaction signals are stored as a multimodal information feature vector X = {x} in the following order. i}

[0096]

[0097] To avoid abnormal network weights due to the excessively large magnitude of some input variables during training, it is necessary to process the data with large magnitude differences across the value range to obtain normalized values.

[0098]

[0099] In the formula ——Interactive Information X i Corresponding basic bias

[0100] —X after bias adjustment i The corresponding lower limit of the range

[0101] —X after bias adjustment i The corresponding lower limit of the range

[0102] To enhance the data dimensionality and feature information of the samples, the multimodal information feature vector X is expanded by two-dimensional copying and then visualized into an image matrix.

[0103]

[0104] After the above preprocessing, the multimodal information feature vector consists of 20 feature data points: 6 joint angles, 6 joint angular velocities, 2 attitude angles, 2 attitude angular velocities, 2 linear accelerations, and 2 plantar pressures. To avoid excessively large preset image information leading to low training time and recognition efficiency of the recognition model, X... ExpIt is set as a low-dimensional image matrix. In summary, N is selected. Sensor =20, N Image =28, then the multimodal information feature samples of the human-computer coupling system used for motion pattern model recognition training are single-layer image numerical matrices with a size of 28×28 and a value range of [-1,1].

[0105] In this example embodiment, the method designs an autoencoder network model using random noise reduction, multi-layer concatenation, and sparsification operations to achieve layer-by-layer feature extraction and effectively remove redundant information, reducing the dimensionality of data features while preserving the original data's identifying features to the greatest extent possible. The resulting network model has a simple structure and can achieve hidden layer feature extraction and parameter learning under unsupervised learning, enabling dimensionality reduction feature representation of complex inputs. It maintains good noise resistance, especially for sensor signals with noise interference. During training, the autoencoder (AE) is trained sequentially according to a stack. After the first AE is trained, its output is used as the input to the second AE, and so on until the output of the last AE is obtained. This output is then used as the input to the final classifier, and after weight integration and adjustment, the dataset classification training is completed. The goal of the stacked autoencoder is to concatenate the dimensionality reduction feature representations learned by each AE, utilizing the core features of the data for rapid classification.

[0106] To measure the average activation output of hidden node j at level l+1 The similarity between the sparsity ρ and the set sparsity is used to introduce KL discreteness for parameterized characterization.

[0107]

[0108] analyze With ρ and The changing trend of ρ is determined to establish the monotonicity on both sides of the extreme boundary and inflection point, and an appropriate value of ρ is selected. In order to ensure... To minimize the penalty term by approximating the preset ρ, divergence is introduced into the loss function. This is applied to m pre-classified labeled datasets. Batch gradient descent is used to train the neural network model.

[0109] In order to make By approximating the predefined ρ to minimize the penalty term, divergence is introduced into the loss function, thereby obtaining a loss function that sparsifies the model.

[0110]

[0111] In the formula h W,b (·)——Network neuron output mapping function

[0112] —The connection weight between node i at level l and node j at level l+1

[0113] λ — Weight decay parameter

[0114] β — Weight of the sparse penalty term

[0115] s l —Number of nodes in layer l

[0116] n l --Number of network layers

[0117] To prevent overfitting during training, reduce the joint fitness between neurons, and enhance the model's generalization ability, the weights of some hidden layer nodes are randomly disabled during model training. This is achieved by introducing dropout, which resets the input layer data to zero with a certain probability to obtain the output.

[0118]

[0119] In the formula — Zeroing probability vector elements

[0120] b—Bias of the network neuron node output mapping function

[0121] x — Input to the network neuron node output mapping function

[0122] — Output of network neuron node mapping function

[0123] In step S140, this method employs a flexible dot matrix pressure insole based on a flexible printed circuit (FPC). Leveraging its soft material and ability to conform to the foot's shape, it collects the pressure variation trends between the wearer's sole and exoskeleton along the sole width (SW) and sole length (SL). Sole pressure matrix (SPM) information is then collected. The dimensions are 14×7, totaling N. SPM = 98 data points, each data point is a dimensionless digital pressure representing the degree of contact, with a value range of [0, 255], where 0 represents complete separation and 255 represents complete contact.

[0124]

[0125] To obtain plantar pressure dot matrix images with richer features and smoother edges, a two-dimensional nonlinear interpolation method based on the Gaussian radial basis function (GRBF) was used to expand the dot matrix data. The interpolation approximation assumptions were set as follows:

[0126]

[0127] In the formula Gaussian radial basis functions

[0128] r—Interpolation point position vector

[0129] r j —Radial basis function support point position vector

[0130] Search for weight coefficients by applying interpolation conditions. Thus, the interpolation polynomial coefficient matrix can be solved. Regarding the target fitting vector linear equations

[0131]

[0132] In this example embodiment, this method, compared to commonly used two-dimensional linear interpolation, two-dimensional cubic convolutional interpolation, and two-dimensional fifth-order polynomial interpolation, expands the three-dimensional feature F of the matrix. Exp ={ i,j f Exp The feature matrix obtained based on nonlinear function interpolation is smoother at the element boundaries. As the order of the nonlinear function increases, the numerical fluctuations of the feature matrix elements become increasingly drastic, resulting in complex curvature oscillations below the zero-value plane. The feature matrix based on GRBF, while achieving continuous element interpolation, can maintain a smoother surface curvature, and the extended features are closer to the original data.

[0133] This study utilizes convolutional neural networks to extract structural features of plantar pressure in the foot space, primarily consisting of three steps: convolution, activation function, and pooling. First, N... Channel ×N Width ×N HeightConvolutional kernels perform convolutional feature calculations on the input node data, and then activation functions are used for nonlinear mapping to simulate more subtle changes between input and output, increasing the nonlinear expressive power of the network. Analysis of commonly used activation functions and their gradients reveals that for the conventional ReLU activation function, when the bias term is too small, causing the input activation function's value range to be negative, gradient vanishing during backpropagation will prevent the corresponding weights and bias parameters from being updated, leading to neuron failure at that point. For LReLU and Sigmoid, the inability to achieve one-sided saturation means convergence is not always optimal. Furthermore, consistently non-negative output values ​​can cause jagged gradients during parameter updates. An activation function is considered ideal when the following two conditions are met: the output distribution is zero-mean, thus accelerating training; and the activation function is one-sidedly saturated, thus achieving better convergence. Therefore, to avoid the aforementioned problems during backpropagation, the ELU activation function is used instead of the conventional ReLU activation function to assign gradients to negative regions, resulting in node outputs that are...

[0134]

[0135]

[0136] In the formula f ELU (*)——ELU node activation function

[0137] x i —Input of node j at level l+1

[0138] — Kernel bias between node j in layer l and node j in layer l+1

[0139] — Convolution kernel weights of node i in layer l and node j in layer l+1

[0140] α — Gradient hyperparameter of the activation function

[0141] After Conv2D+ELU feature extraction, a max-pooling2D operation is performed, and the feature layer matrix is ​​resized to (P... w ,P h The ZeroPadding2D padding operation fills the matrix boundary with zeros, enabling edge filtering after convolution and controlling the output size h'×w' of the feature map.

[0142]

[0143] Repeat the above feature extraction operation. When extracting features in the last convolutional layer, perform convolution calculation on the output of the previous convolutional operation and perform activation operation using the same activation function. Finally, perform global max pooling (GlobalMaxPooling2D) on the output value to obtain spatial structure features.

[0144] GRU, as a variant of LSTM, is also a special type of chain-like recurrent neural network (RNN). Closely related to sequential and list data, it is a natural neural network architecture for data with spatiotemporal patterns, capable of learning long-term dependencies. The cell structure of a GRU network mainly contains two gate structure operational units and one recall operation operational unit. The current memory state output value h... <t>< / t> The input x at the current time <t>< / t> The output value h of the memory state at the previous moment <t-1>< / t-1> The decision

[0145]

[0146] In the formula —Reset the gate and update the output value of the gate's arithmetic unit. σ * W is the activation function. * For connection weights.

[0147] —Recall the output value of the operation,

[0148] ——Hadamard product operation

[0149] The calculation method for the recall operation unit is basically the same as that for the gate structure operation unit, except that the recall operation unit selects tanh for activation. The input operation of the network node has the memory state output value h from the previous time step. <t-1>< / t-1> The reset adjustment, compared to LSTM, has a simpler structure. When the reset gate's output approaches 0, the intermediate memory state is forced to ignore the previous memory state and only uses the current input for reset. This effectively allows the intermediate memory state to delete any information irrelevant to the future, achieving a more compact information representation. On the other hand, the update gate controls how much information from the previous memory state will be passed to the intermediate memory state. This operation is similar to the storage unit in an LSTM network, helping the RNN remember long-term information. After multi-layer GRU feature extraction, the output of the last GRU is activated by ReLU and fully connected, and then concatenated with CNN features to obtain coupled features. Finally, feature fusion connection, activation output, and Softmax classification are performed sequentially to obtain gait phase probability labels.

[0150] During movement, the wearer's legs constantly switch between swing and support phases, and the soles of the feet successively experience gait phase changes: soldangling (SD), heel strike (HS), forefoot strike (FS), heel lift (HL), forefoot lift (FL), and soldangling (SD).

[0151] Specifically, when ascending or descending stairs, if the stairs are narrow, the wearer tends to land on their toes first. During phase identification, it is necessary to post-process and identify the interference state where the forefoot lands before the heel in the swaying phase, thus avoiding false triggering of gait phase states. Dynamic analysis of plantar pressure changes reveals that the sum of plantar pressure decreases during the FL phase compared to FS. Since plantar pressure values ​​exhibit sensitive high-frequency changes with the contact degree between the pressure insole and the wearer's foot, a tracking differentiator (TD) algorithm is introduced to accurately extract the differential signal of the sum of plantar pressure in the current control cycle and reduce signal hysteresis. This avoids noise interference and phase delay introduced by differential calculations. For extracting multi-order differential signals from discontinuous signals with random noise, variable-step TD outperforms traditional differential calculations and Kalman filtering methods. When the sampled signal is noisy and the sampling time varies, using variable-step TD to extract the filtered tracking signal and its first-order differential tracking signal shows good vibration suppression during value range discrimination. Based on the TD formula, the filtered tracking signal and its first-order differential tracking signal of the resultant force of the original plantar pressure matrix can be obtained.

[0152]

[0153] In the formula —Signal tracking output

[0154] —Signal Differential Tracking Output

[0155] T – Integral step size

[0156] r is the filter coefficient

[0157] —Characteristic second-order signal, and input signal u k Related to forecast step size h

[0158] The trained model is tested for generalization ability on non-training samples, and the results are analyzed based on the gradient of the SPM resultant force and the gradient threshold u. Δ+Compare and determine the gait phase at time k. Is it in true forefoot lift-off phase (FL) or false forefoot lift-off phase (FL)? + That is, the gradient of the current resultant force and the positive or negative gradient threshold u. Δ+ u Δ- To determine whether it is FL or FL based on the relationship between them + After gradient discrimination and post-processing, the gait phase was correctly identified, avoiding false triggering of the phase transition from the support phase to the swing phase.

[0159]

[0160] Cross-entropy loss, used in deep learning frameworks, describes the similarity between predicted labels and ideal targets. A smaller value indicates that the model's predictions are closer to the real samples. Cross-entropy is a fusion of the normalized exponential function LogSoftmax and the negative likelihood log loss NLLLoss, based on probability and target orientation. When the input is a log probability vector and a target label, it can compress a multidimensional vector into the range (0,1) with a sum of 1. Due to the complex transition states between the gait patterns and gait phases of exoskeletons, similarities in motion at segmentation boundaries can easily occur during training, leading to significant discrepancies between the recognition accuracy of some classification labels and the ideal accuracy after training. Therefore, a weighted correction is applied to the cross-entropy function, adjusting the weights of the cross-entropy loss function for error-prone classification labels and their associated labels.

[0161]

[0162] In the formula —Probability labels predicted by the network after training

[0163] w — the weight vector of the ideal label category, w = [w1, w2, ..., w M ]

[0164] λ k — Vector index of the ideal classification label

[0165] like Figure 4 , 5As shown, after determining the cross-entropy loss function for model training, it is necessary to analyze the impact of weight vector hyperparameters on the model's recognition accuracy, thereby selecting appropriate model parameters to improve the recognition accuracy and efficiency of the trained network model. This method employs a hybrid genetic-particle swarm optimization (HGP) approach to optimize the weight vectors corresponding to each label category in both the denoising-sparse-stack-based autoencoder neural network model for gait pattern recognition and the parallel neural model based on spatial-temporal feature hybrid extraction for gait phase recognition. This effectively improves the recognition accuracy for each label category.

[0166]

[0167] like Figure 6 As shown, due to the limitations of neural network models in generalizing to untrained data, a few abnormal predictions may occur during the identification process. To avoid phase switching jitter caused by abnormal prediction jumps, a mode search is introduced to post-process the prediction results, namely, a label vector composed of the model's predicted gait features at the current time step and the model's predicted gait features from the previous n time steps. Perform a mode search to obtain the gait phase estimate for the current moment.

[0168]

[0169] In this example embodiment, to address the need for identification and switching between continuous and different modalities, an interaction signal with typical amplitude-frequency characteristics, independent of physical binding and force, is used. A graphical compilation extension method is employed to achieve feature fusion representation of multimodal information. For the identification model, a hidden layer feature extraction and parameter learning method based on unsupervised learning is used to achieve dimensionality reduction of complex input features. Furthermore, random denoising, multi-layer cascading, and sparsification operations are used to optimize the autoencoder network model, enabling layer-by-layer feature extraction and effective removal of redundant information. This reduces the dimensionality of data features while preserving the original data's identification features to the greatest extent possible, and maintains good noise resistance.

[0170] In this example embodiment, to address the need for identification and switching between multiple consecutive gait phases, a continuous and accurate description of high-dimensional features is achieved by nonlinearly interpolating and extending the features of the low-dimensional hardware sensor contact array. By extracting features from the spatial and temporal domains and fusing them, the feature dimension, richness, and continuity of the plantar contact information are improved while ensuring the signal acquisition frequency. This effectively improves the accuracy of the neural network model in identifying consecutive similar gait phases. Furthermore, to address the misidentification that is prone to occur when the step width is small and the wearer tends to land on their toes first, threshold analysis and judgment of the gradient of the resultant force of the plantar contact matrix are used to effectively avoid incorrect judgment of gait phase in the case of false foot lift.

[0171] In the embodiments of this example, the present invention can improve the autonomous perception and efficient prediction of the wearer's target gait intention by exoskeleton robots to a certain extent. In particular, the switching of gait phase and gait mode between different motion scenarios of the exoskeleton robot is optimized by using a hybrid particle swarm optimization-genetic optimization algorithm to optimize the weight vector of each label in the loss function. This achieves the target optimal value for the identification accuracy of each category of gait phase and gait mode. The introduction of mode search for the predicted label time-series vector effectively avoids the oscillation step caused by outliers, and has good robustness and generalization ability. The exoskeleton's ability to identify overall gait features is greatly improved.

[0172] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0173] Furthermore, in this example embodiment, a device for identifying exoskeleton gait features based on multimodal information fusion is also provided. (Refer to...) Figure 7 As shown, the exoskeleton gait feature identification device 200, which uses multimodal information fusion to represent gait characteristics, may include: a sensor initialization module 210, a data matrix construction module 220, a first model modeling module 230, a second model modeling module 240, and a gait feature evaluation module 250. Wherein:

[0174] The sensor initialization module 210 is used for calibration and initialization of the control program of the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor of the exoskeleton wearer coupling system.

[0175] The data matrix construction module 220 is used to collect exoskeleton sensor signals based on the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor, calculate multi-order derivatives and store them as time series, and construct a multimodal fusion information extension matrix that represents spatial-temporal features.

[0176] The first modeling module 230 is used to design a loss function and introduce a sparsity correction term based on the parameterized representation characteristics trend of KL discreteness, design a random denoising controller for hierarchical nodes, and construct a multi-layer stacked denoising sparse autoencoder neural network model.

[0177] The second modeling module 240 is used to design node activation functions based on CNN spatial domain feature extraction, design node transmission structure units based on GRU temporal domain feature extraction, design multi-layer feature hybrid connection methods and feature fusion functions, and construct a parallel neural network model for spatial-temporal feature hybrid extraction.

[0178] The gait feature evaluation module 250 is used to optimize the weight vector of the cross-entropy function in the multi-layer stacked denoising sparse autoencoder neural network model and the parallel neural network model with spatial-temporal feature hybrid extraction based on the genetic-particle swarm optimization method, so as to make the accuracy of the identification model reach a preset value. The module performs mode search on the gait feature label vectors predicted by the model in the first n time steps and filters out the identification outliers as the gait feature evaluation output for post-processing.

[0179] The specific details of the exoskeleton gait feature identification device modules of the above-mentioned multimodal information fusion representation have been described in detail in the corresponding exoskeleton gait feature identification method of multimodal information fusion representation, so they will not be repeated here.

[0180] It should be noted that although several modules or units of the exoskeleton gait feature recognition device 200 characterized by multimodal information fusion have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0181] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0182] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented as entirely hardware embodiments, entirely software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.”

[0183] The following reference Figure 8 To describe an electronic device 300 according to such an embodiment of the present invention. Figure 8 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0184] like Figure 8 As shown, the electronic device 300 is presented in the form of a general-purpose computing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different system components (including storage unit 320 and processing unit 310), and a display unit 340.

[0185] The storage unit stores program code that can be executed by the processing unit 310, causing the processing unit 310 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 310 can perform actions such as... Figure 1 Steps S110 to S150 are shown in the diagram.

[0186] Storage unit 320 may include readable media in the form of volatile storage units, such as random access memory (RAM) 3201 and / or cache memory 3202, and may further include read-only memory (ROM) 3203.

[0187] Storage unit 320 may also include a program / utility 3204 having a set (at least one) program module 3205, such program module 3205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0188] Bus 330 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0189] Electronic device 300 can also communicate with one or more external devices 370 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 300, and / or with any device that enables electronic device 300 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 350. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 360. As shown, network adapter 360 communicates with other modules of electronic device 300 via bus 330. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0190] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0191] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section above.

[0192] refer to Figure 9 As shown, a program product 400 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0193] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0194] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0195] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0196] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0197] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0198] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0199] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for identifying exoskeleton gait features based on multimodal information fusion representation, characterized in that, The method includes: Calibration and initialization processing of the control program for the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor of the exoskeleton wearer coupling system; Based on the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor, the exoskeleton sensor signals are collected, multi-order derivatives are calculated and stored as time series, and a multimodal fusion information extension matrix representing spatial-temporal features is constructed. Based on the analysis of the parameterized characterization trend of KL discreteness, a loss function is designed and a sparsity correction term is introduced. A stochastic denoising controller is designed for the hierarchical nodes, and a multi-layer stacked denoising sparse autoencoder neural network model is constructed. Design a node activation function based on CNN spatial domain feature extraction, design a node transmission structure unit based on GRU temporal domain feature extraction, design a multi-layer feature hybrid connection method and feature fusion function, and construct a parallel neural network model for spatial-temporal feature hybrid extraction; Based on the genetic-particle swarm optimization method, the weight vector of the cross-entropy function in the multi-layer stacked denoising sparse autoencoder neural network model and the parallel neural network model with spatial-temporal feature hybrid extraction is optimized to make the accuracy of the identification model reach the preset value. The mode search is performed on the gait feature label vectors predicted by the model in the first n time steps, and outliers are filtered out as the gait feature evaluation output for post-processing.

2. The method as described in claim 1, characterized in that, The method further includes: The control programs for the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor of the exoskeleton wearer coupling system are initialized, initial bias is extracted, and reference zero value is calibrated.

3. The method as described in claim 1, characterized in that, The method further includes: The exoskeleton sensor signals are acquired based on the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor. Based on the exoskeleton sensor signals, multi-order derivatives are calculated and stored as time series. Construct multimodal information feature vectors and perform image-based compilation and expansion; The contact state matrix of the flexible array insole on the sole of the wearer's foot is collected. The discrete sparse lattice is extended by nonlinear interpolation based on Gaussian radial basis function to construct a multimodal fusion information extension matrix that represents spatial-temporal features.

4. The method as described in claim 3, characterized in that, The method further includes: After normalizing and performing two-dimensional copying and expansion on the multimodal information feature vectors, image-based compilation and expansion are performed to generate an image matrix.

5. The method as described in claim 1, characterized in that, The method further includes: The multi-layer stacked denoising sparse autoencoder neural network model is trained according to the stack order based on the autoencoder; The training output of the autoencoder is used as the input to the final classifier for training; The training output of the final classifier is integrated and adjusted based on the weights to complete the classification training of the dataset based on the multi-layer stacked denoising sparse autoencoder neural network model.

6. The method as described in claim 1, characterized in that, The method further includes: A flexible thin-film array insole based on a flexible circuit board was used to collect the pressure change trend between the wearer's foot and exoskeleton along the foot width and foot length, and generate a pressure change dot matrix. The pressure variation lattice is expanded using a two-dimensional nonlinear interpolation method based on Gaussian radial basis functions.

7. The method as described in claim 1, characterized in that, The method further includes: The generalization ability of the trained model is tested on non-training samples, and the gradient of the SPM resultant force is compared with the gradient threshold. The model is optimized based on the judgment of whether the gait phase at the current moment is in the true forefoot take-off phase.

8. A device for identifying exoskeleton gait features based on multimodal information fusion representation, characterized in that, The device includes: The sensor initialization module is used for calibration and initialization of the control program of the back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor of the exoskeleton wearer coupling system. The data matrix construction module is used to collect exoskeleton sensor signals based on back IMU sensor, joint absolute encoder, wearer's flexible thin film array insole, and exoskeleton foot contact force sensor, calculate multi-order derivatives and store them as time series, and construct a multimodal fusion information extension matrix that represents spatial-temporal features; The first modeling module is used to design a loss function and introduce a sparsity correction term based on the trend of parameterized representation characteristics of KL discreteness, design a random denoising controller for hierarchical nodes, and construct a multi-layer stacked denoising sparse autoencoder neural network model. The second modeling module is used to design node activation functions based on CNN spatial domain feature extraction, design node transmission structure units based on GRU temporal domain feature extraction, design multi-layer feature hybrid connection methods and feature fusion functions, and construct a parallel neural network model for spatial-temporal feature hybrid extraction. The gait feature evaluation module is used to optimize the weight vector of the cross-entropy function in a multi-layer stacked denoising sparse autoencoder neural network model and a parallel neural network model with spatial-temporal feature hybrid extraction based on the genetic-particle swarm optimization method, so as to make the accuracy of the identification model reach a preset value. The module performs a mode search on the gait feature label vectors predicted by the model in the first n time steps and filters out the identification outliers as the gait feature evaluation output for post-processing.

9. An electronic device, characterized in that, include Processor; and A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images