Hip exoskeleton motion pattern recognition method and system based on deep learning

Abstract

The invention discloses a hip exoskeleton motion pattern recognition method and system based on deep learning. The method comprises the following steps: data acquisition and processing: acquiring heterogeneous sensor data through an inertial measurement unit and a joint motor of a hip exoskeleton; performing off-line / real-time calibration and time synchronization on the data, and constructing a data set in combination with a motion label; model construction and training: splicing multi-modal sensor data as input, constructing a deep learning network, fusing heterogeneous features to recognize terrain and motion modes, training a model and storing an optimal model; and lightweight deployment: optimizing sensor configuration through an ablation experiment, compressing model parameters by adopting channel pruning, and generating a lightweight model in combination with an INT8 quantification technology. The system comprises a data acquisition module, a mode identification module and a quantitative deployment module. According to the method, the problems of data fusion and real-time deployment of heterogeneous sensors are solved, and the precision and efficiency of exoskeleton motion recognition are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of exoskeleton intelligent control, and in particular relates to a hip exoskeleton motion pattern recognition method and system based on deep learning. Background Art

[0002] For people with limited or weak lower limb function, traditional assistive methods are currently limited in functionality, and relying on professional rehabilitation physicians faces challenges such as manpower shortages, uneven results, and low efficiency. Therefore, innovative technologies are urgently needed to address rehabilitation needs. Exoskeleton technology offers a highly promising solution to these issues. Currently, traditional exoskeletons lack intelligence and rely on preset rules to provide assistance, which restricts their application in complex motion scenarios. Artificial intelligence-based exoskeletons can sense human-machine status and provide active assistance, effectively assisting daily activities and promoting rehabilitation training. Accurate gait and terrain recognition significantly improve the core performance and safety of exoskeletons.

[0003] The sensor systems in the existing technology are overly complex. Chinese patent application document CN120093566A discloses a hybrid ankle exoskeleton system that integrates functional electrical stimulation. It requires precise attachment of electrodes to the tibialis anterior and soleus muscles, making it difficult for patients to operate independently and limiting the universality of the system. Chinese patent application document CN119839839A relies on seven types of sensor devices, including CMOS, radar, GPS-RTK, two-stage inertial sensors, and foot pressure sensors. The hardware integration is difficult, which greatly increases the manufacturing cost and maintenance burden. At the same time, the large amount of sensor data requires high computing power of the equipment, and real-time performance cannot be guaranteed on a scarce computing power platform. Summary of the Invention

[0004] The purpose of the present invention is to provide a hip exoskeleton motion pattern recognition method based on deep learning to solve the technical problems of heterogeneous sensor data fusion and real-time deployment, thereby significantly improving the accuracy and efficiency of exoskeleton motion recognition.

[0005] To achieve the purpose of the present invention, on the one hand, the present invention provides a hip exoskeleton motion pattern recognition method based on deep learning, comprising the following steps:

[0006] Step 1: Use the sensors on the hip exoskeleton to capture the wearer's real-time motion data, collect and save the motion data to obtain a data set, generate motion label true values ​​for the data set through the corresponding host computer, and establish a motion pattern recognition data set;

[0007] Step 2: constructing a network model structure based on the motion pattern recognition dataset, realizing feature fusion of heterogeneous sensor multimodal data, outputting terrain and motion pattern recognition results, and training the network model on the motion pattern recognition dataset to achieve convergence, thereby obtaining a motion pattern recognition network model;

[0008] Step 3: Based on the motion pattern recognition network model, after optimizing the sensor configuration through ablation experiments, network pruning is used to compress the model parameters, and at the same time, INT8 quantization technology is combined to build a lightweight motion pattern recognition model.

[0009] On the other hand, the present invention also provides a system for hip exoskeleton motion pattern recognition based on deep learning, comprising the following modules:

[0010] The hip exoskeleton data acquisition module is used to collect the wearer's motion data and combine it with external input motion labels to construct a motion pattern recognition dataset;

[0011] A motion pattern recognition module, which develops and trains a hip exoskeleton motion pattern recognition network model based on the motion pattern recognition dataset, extracts motion features, and recognizes terrain and wearer motion patterns;

[0012] The quantized deployment module is used to lightweight the motion pattern recognition network model and deploy it to the embedded single-chip microcomputer on the hip exoskeleton device, so that the hip exoskeleton can recognize the wearer's motion pattern in real time.

[0013] Compared with the existing technology, the significant progress of the present invention lies in: (1) The present invention uses the motion data information of the inertial sensor installed on the waist of the exoskeleton and the motor encoders on both sides of the hip, which greatly simplifies the complexity and cost of the sensor system; (2) The present invention combines artificial intelligence technology to efficiently extract motion data features and achieve accurate gait and terrain recognition capabilities; (3) The hip exoskeleton system constructed by the present invention, through the self-developed hip exoskeleton device equipped with various sensors and the exoskeleton motion pattern recognition method based on deep learning, achieves a classification accuracy of more than 98% in five types of gait recognition tasks, including walking, climbing up and downhill, and sitting up. At the same time, after quantization, the model has a parameter volume of only 20KB. The actual inference time measured on the low-power embedded single-chip terminal is only 13ms.

[0014] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0016] Figure 1 This is a block diagram of the exoskeleton motion pattern recognition method based on deep learning of the present invention;

[0017] Figure 2This is a diagram of the hip exoskeleton data acquisition module of the present invention;

[0018] Figure 3 is a specific layout diagram of the exoskeleton sensor of the present invention;

[0019] Figure 4 is a visualization image diagram of different gait motion data of the exoskeleton sensor of the present invention;

[0020] Figure 5 This is a diagram showing the structure of the hip exoskeleton motion pattern recognition network model of the present invention;

[0021] Figure 6 This is a comparison diagram of the parameters of the hip exoskeleton motion pattern recognition model before and after quantization. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] The present invention is a hip exoskeleton motion pattern recognition method based on deep learning, combined with Figure 1 , including the following steps:

[0024] Step 1: Use the sensors on the hip exoskeleton to capture the wearer's real-time motion data, collect and save the motion data to obtain a data set, generate motion label true values ​​for the data set through the corresponding host computer, and establish a motion pattern recognition data set;

[0025] Step 2: constructing a network structure based on the motion pattern recognition dataset to achieve feature fusion of multimodal data from heterogeneous sensors, outputting terrain and motion pattern recognition results, and training the network model on the motion pattern recognition dataset to achieve convergence, thereby obtaining a motion pattern recognition network model;

[0026] Step 3. Based on the motion pattern recognition network model, after optimizing the sensor configuration through ablation experiments, while ensuring the recognition accuracy, reduce the number of sensors used to reduce the model complexity and optimize the computational overhead. Use network pruning to compress the model parameters. At the same time, combine the INT8 quantization technology to quantize the model weight parameters to reduce the computational complexity and improve the inference efficiency. Build a lightweight motion pattern recognition model, and deploy the lightweight motion pattern recognition model to the MCU installed on the exoskeleton to enable the exoskeleton to recognize the wearer's motion pattern.

[0027] The step 1 specifically includes the following steps:

[0028] Step 1-1, collecting sensor data from the waist inertial measurement unit, the left thigh inertial measurement unit, the right thigh inertial measurement unit and the joint motor, the sensor data of the waist inertial measurement unit including the acceleration information and angular velocity information of the wearer's waist root node, the sensor data of the left thigh inertial measurement unit including the acceleration information and angular velocity information of the wearer's left thigh near the knee joint, the sensor data of the right thigh inertial measurement unit including the acceleration information, angular velocity information and posture information of the wearer's right thigh near the knee joint, and the joint motor sensor data including the angle information and angular velocity information of the wearer's thigh swing;

[0029] The hip exoskeleton data acquisition module of this embodiment is as follows Figure 2 As shown, the STM32F407IGT6 is selected as the main control unit MCU (microcontroller unit); the joint motors installed on the left and right hips of the hip exoskeleton device contain motor built-in magnet encoders, which are connected to the MCU via CAN (Controller Area Network) lines to record the angle and angular velocity information of the wearer's thigh swing; inertial measurement units (IMUs) are installed at the waist position of the exoskeleton and the left and right thighs near the knee joints. The IMUs communicate with the MCU via IIC (Inter-Integrated Circuit) bus to record the acceleration, angular velocity and posture information of the wearer's waist root node and left and right thighs;

[0030] The motor receives information through a query-based method. In this exoskeleton system, we set the control cycle to 10ms and periodically query the motor joint angles and angular velocities in a 10ms timer interrupt, resulting in a motor message frequency of 100Hz. For the IMU, we set the ODR (output data register) update frequency to 500Hz. In the MCU, we set the IIC clock frequency to 100kHz and set a 2ms timer interrupt to read the IMU raw data, resulting in an IMU message frequency of 500Hz.

[0031] Step 1-2: calibrate the above heterogeneous sensor data;

[0032] Steps 1-3: Time synchronization of two heterogeneous sensors based on the calibrated sensor data;

[0033] Steps 1-4: Record and save the data collected from the heterogeneous sensors after time synchronization is completed;

[0034] Steps 1-5: The simultaneously recorded heterogeneous sensor data are imported and synchronously saved into each frame of heterogeneous sensor data to obtain a motion pattern recognition dataset.

[0035] The motion tagging process is as follows: the wearer or assistant connects to the exoskeleton's Bluetooth via an electronic information device (such as a mobile phone, tablet, or laptop) to receive sensor messages and issue motion tags. Taking the mobile app as an example, when a participant performs different exercise modes, they click different movement buttons on the app, and the motion tags are sent to the exoskeleton to annotate their current motion status. The motion tag occupies one byte, with the start of collection being 0, standing being 1, walking being 2, going upstairs being 3, going downstairs being 4, switching motion being 5, going uphill being 6, going downhill being 7, and running being 8. The motion tags are sent to the exoskeleton and populated into the corresponding frame of heterogeneous sensor data. All data frames are continuously saved to construct a complete motion pattern dataset, which is then used for subsequent model building and testing.

[0036] The steps 1-2 specifically include:

[0037] First, perform offline internal calibration of the IMU: Place the IMU horizontally for about 15 minutes. Theoretically, the accelerometer of the IMU will measure the for , g is the acceleration of gravity, and the angular velocity of rotation is measured is zero; but the actual IMU has zero bias, statistics and , collect accelerometer and gyroscope measurements, calculate the average measurement value to calibrate the accelerometer and gyroscope zero bias:

[0038] ;

[0039] ;

[0040] Where N is the total number of frames collected by the inertial measurement unit, represents the k-th frame measurement, and are the raw measurements of the IMU's accelerometer and gyroscope, g is the acceleration of gravity, and It is the zero bias of offline calibration;

[0041] Then perform real-time calibration on the collected raw data of the inertial measurement unit:

[0042] ;

[0043] ;

[0044] in, and This is the data after calibration of the inertial measurement unit;

[0045] During the installation of the joint motors to the exoskeleton, there are differences in the installation rotation angles, resulting in inconsistent mechanical zero positions of the motors of different devices. Therefore, it is stipulated that the exoskeleton should be worn and started in a standing position, and the sensor data of the joint motors should be calibrated during startup to achieve consistency in the zero positions of the motors of different devices. At the same time, in order to avoid the motor angle jumping from 180° to -180° during normal movement, the standing state angle is set to 0° in the initialization phase through software settings, taking into account the actual range of the human joints that can lift the legs forward and backward from the standing state. At the same time, by adding circle detection, it is stipulated that the left and right leg lifting is the positive direction. Through the above operations, during normal movement, the angles of the left and right leg motors have opposite change trends, and the data distribution difference is large, which is conducive to model convergence.

[0046] Calibrate the sensor data of the joint motor:

[0047] ;

[0048] ;

[0049] in, and is the original angle of the joint motor, N is the number of turns, clockwise is the positive turn, and counterclockwise is the negative turn; and The joint motor angle measurement after calibration.

[0050] In actual use, for multiple sets of exoskeletons, there is no need to additionally set the mechanical zero point and positive rotation direction of the motor, which greatly reduces the workload.

[0051] The steps 1-3 are specifically as follows: Since the IMU and joint motors are not designed to trigger acquisition synchronously at the hardware bottom layer, the collected IMU data and joint motor data are inconsistent in timestamps. The IMU and motor hardware are connected to the same MCU, and the two sensors use the same clock source time; the IMU data is linearly interpolated to the motor measurement timestamp to achieve time synchronization of heterogeneous sensor data acquisition. In this embodiment, by setting a 2ms high-frequency timing interrupt at the MCU bottom layer, the IMU register is read in the timing interrupt and the real physical measurement value is obtained, and the MCU time is recorded; the motor information is read in a query-type manner, and a 10ms timing is set to send the motor query instruction, and the motor data time is recorded in the receiving interrupt. ;

[0052] Calibrated IMU sensor data and Perform linear interpolation to obtain the joint motor data acquisition time The sensor data of the inertial measurement unit is used to complete the time synchronization of the collected data of the two heterogeneous sensors:

[0053] ;

[0054] ;

[0055] in, 、 Joint motor data time Left and right adjacent inertial measurement unit data frames; and are the times corresponding to the data frames respectively; and This is the measurement after the IMU and joint motor data acquisition time is synchronized. Since the sampling frequency set by the inertial measurement unit is very high, the time synchronization error between the two heterogeneous sensors, IMU and motor, is small and negligible through the interpolation approximation method.

[0056] The steps 1-4 are specifically as follows: In this embodiment, each measurement value is of type int16 and occupies 2 bytes. Therefore, the length of one frame of IMU data is 12 bytes. The data length of integrating waist IMU, left thigh IMU, and right thigh IMU measurements into one frame is 36 bytes.

[0057] Save the accelerometer data of the three inertial measurement units and the left and right two joint motors in the xyz three-axis synchronization And the gyroscope's three-axis data in xyz ; For the angle and angular velocity information of the joint motor, scale it and save it;

[0058] In this embodiment, the angle θ and the rotation speed The float type occupies four bytes. To reduce the data size, it is scaled to uint16, which occupies 2 bytes. Determine the scaling factor. , After scaling, the length of the left and right motor angle and angular velocity data is 8 bytes. Specifically, the maximum and minimum values ​​of the motor angles are determined and scaled based on the exoskeleton device motor and leg rod structure and the human body joint limit. When standing upright, the motor angle is 180°; when lifting the leg backward, the motor angle decreases, and the minimum value is ; Lift your legs upward, the motor angle increases, the maximum value The joint motor rotation angle is scaled as:

[0059] ;

[0060] in is the angle after scaling, is the angle after motor calibration.

[0061] For the motor rotation speed, the maximum value is determined by combining the exoskeleton device motor and leg rod structure, and the human joint limit. , the minimum value is , the scaled motor rotation speed is obtained by the following formula :

[0062] ;

[0063] in is the scaled angular velocity, is the motor angular velocity before scaling.

[0064] During data acquisition, a fixed 2-byte frame header and 2-byte frame trailer are added, which are padded with a 4-byte timestamp, 36-byte IMU time-synchronized data, 8-byte scaled motor data, and a 1-byte corresponding motion tag to form a single frame of data. During the initialization phase, the wearer, wearing the exoskeleton, remains standing and stationary, waiting for the IMU and motor initialization to complete. After sensor initialization, the wearer begins movement. The wearer completes a series of different motion patterns, including climbing and descending stairs, walking, standing still, and ascending and descending slopes. During this time, multiple frames of heterogeneous sensor data are continuously recorded and stored in the exoskeleton's onboard 2Gbit Nand storage device.

[0065] Steps 1-5 specifically involve synchronously recording heterogeneous sensor data, importing motion tag data externally, and synchronously saving it to each frame of heterogeneous sensor data. In this embodiment, the motion tag generation process is as follows: the wearer or assistant connects to the exoskeleton's Bluetooth via a mobile app to issue motion tags. As the participant performs different modes of exercise, the current motion state is annotated as a motion tag on the mobile app. Different motion modes correspond to different motion tags. The motion tags are sent to the exoskeleton device and populated into the corresponding frame of heterogeneous sensor data. All data frames are continuously saved to build a complete motion pattern dataset.

[0066] After collecting data while wearing the exoskeleton, the exoskeleton is controlled through a mobile phone app to upload data from the onboard 2Gbit storage Nand device. The data uploaded by the exoskeleton is received on the computer, and the IMU and joint motor measurements in real physical units are restored according to the scaling formula. The motion pattern recognition dataset constructed in combination with the motion labels is used for subsequent model construction and testing.

[0067] The step 2 specifically includes the following steps:

[0068] Step 2-1, dividing the motion pattern recognition dataset into a training set and a test set;

[0069] Two partitioning methods were used to verify the network's effectiveness on different data sets. Method 1 completely isolated the data from different participants, splitting them in an 8:2 ratio. This validated the transferability of the motion pattern recognition model across different individuals. Method 2 isolated the dataset by recording time, verifying the generalization of the motion pattern recognition model to data from different time periods. The training and test sets were used for model architecture design and training.

[0070] In this embodiment, the training set contains over 1.2 million frames of motion data, and the test set contains 300,000 frames of data. The dataset includes 6 sensors and 7 types of gait patterns.

[0071] Step 2-2: preprocess the training set. Figure 3 , Figure 4 The specific layout of the exoskeleton sensors and the visualization images of different gait motion data in this embodiment are respectively shown.

[0072] Define the data format of the inertial measurement unit :

[0073] };

[0074] in, is the acceleration information, is the angular velocity information, i is the three coordinate axes; that is, the single-frame waist inertial measurement unit information contains six dimensions;

[0075] Define the data format of joint motors :

[0076] ;

[0077] in, and Respectively represent the lifting angles of the left and right legs. The initialization angle for static standing is 180. The forward lifting angle is the positive direction, and the reverse is the negative direction. are the angular velocities of the left and right legs respectively, the forward lifting angle is the positive direction, and the reverse is the negative direction;

[0078] The two types of sensor data mentioned above are inconsistent in dimension and cannot be stacked in another dimension. Therefore, in order to use data from different modalities for pattern recognition at the same time, the joint motor and IMU data are preprocessed and spliced ​​as the input data of the motion pattern recognition network model;

[0079] Data on the inertial measurement unit The data of the joint motor To splice:

[0080] ;

[0081] in, After splicing, it is used as the input data of the motion pattern recognition network model. 、 and These are the data from the inertial measurement units installed on the waist, left thigh, and right thigh respectively.

[0082] In the motion pattern recognition network model constructed in this embodiment, the multimodal data is flattened before input into the network, resulting in IMU data of length 6 and motor data of length 4. To ensure a lightweight network design, the IMU and motor data are concatenated on the same dimension before input into the network, meaning that a single frame of data is represented as a one-dimensional data structure of length 10.

[0083] Step 2-3: Use the preprocessed data as input, combine deep learning technology and MobileNet network structure to build an exoskeleton motion pattern recognition network model;

[0084] To ensure the exoskeleton motion pattern recognition network model is lightweight, the network only uses a one-dimensional convolutional structure to model the input data. Regarding the input, considering that a gait takes approximately 0.5-0.8 seconds during normal activity, and considering the sensor sampling frequency, the time window size is designed to be 60.

[0085] To ensure the exoskeleton motion pattern recognition network model is as lightweight as possible while retaining sufficient data fitting capabilities, the design principles of MobileNet were adopted. Data dimensionality is increased at the front end of the exoskeleton motion pattern recognition network model to amplify the differences in data characteristics across motion patterns in high-dimensional space, making it easier for the model to learn the differences in sensor motion data distribution across different motion patterns. Subsequently, dimensionality reduction is performed on the sensor data preprocessed and concatenated in step 2-2. This reduces the parameters of the exoskeleton motion pattern recognition network model in the subsequent classification layer, ensuring that the model can run on devices with lower computing power.

[0086] In this embodiment, the input data is first normalized by a batch normalization layer at the front end of the network, and its mathematical expression is:

[0087] ;

[0088] in and are the mean and standard deviation of each channel, respectively. The feature extraction module employs a strategy of first increasing the dimensionality and then reducing it: the first convolutional layer expands the number of channels from 8 to 32, extracting temporal features using a 3×1 convolution kernel, leveraging the high-dimensional space to amplify differences in motion patterns. The second convolutional layer compresses the number of channels to 16, preserving key features while significantly reducing the number of parameters. After flattening, the feature map is compressed to a 298-dimensional feature vector using a max pooling layer with a kernel size of 3, and finally mapped to the classification space by a fully connected layer.

[0089] like Figure 5 The figure shows the specific network structure. The data dimension changes during the entire forward propagation process are: input tensor [batch_size, 10, 60] → batch normalization [10, 60] → first convolution [32, 58] → second convolution [16, 56] → flattening

[896] → pooling

[298] → classification output [7].

[0090] The data is then subjected to dimensionality reduction processing to reduce the network parameters of the subsequent classification layers, ensuring that the model can subsequently run on the terminal side with low computing power.

[0091] Step 2-4: Utilize a stochastic gradient descent optimizer on the motion pattern recognition network model, and set an early stopping condition. When the condition is met, terminate the training and save the best model as the final motion pattern recognition model.

[0092] Steps 2-4 are as follows: Initialize model parameters according to the preset network structure and calculate the cross-entropy loss based on the predicted output and the true label. Combined with momentum and weight decay, use the Stochastic Gradient Descent (SGD) optimizer to update model parameters based on the calculated gradients. Continuously monitor the performance of the test set, set an early stopping condition, and terminate training when it is met. Save the best model as the motion pattern recognition model.

[0093] The step 3 specifically includes the following steps:

[0094] Step 3-1: Based on the motion pattern recognition model, the number of sensors used in exoskeleton pattern recognition is reduced to lightweight the network model structure and data input size. Details are as follows. While ensuring recognition accuracy, the number and types of required sensors are gradually reduced to reduce system complexity, power consumption, and cost. In this embodiment, a high-precision baseline model is trained using the most complete dataset of waist IMU, left thigh IMU, right thigh IMU, and left and right joint motor sensors. Its performance on the test set is recorded as a benchmark. A progressive ablation study and retraining strategy is employed. Based on the sensor configuration in the motion pattern recognition dataset, ablation experiments are designed using a subset of these sensor combinations. Sensors are gradually removed, excluding the left and right leg IMU data. Using the same network structure and training process as the baseline model, multiple network models are retrained. Their performance on the test set is recorded for comparison, focusing on changes in accuracy on unseen individuals. Based on the performance of the retrained models, further simplification is performed by removing the Euler angle pose data from the IMU data, retaining only the raw six-axis IMU data. The corresponding models are then retrained and evaluated. Weighing the complexity of the model structure and recognition accuracy, the model with the highest recognition rate within a complexity threshold is selected as the motion pattern recognition model with a simplified sensor configuration. In this example, by comparing the performance, robustness, and hardware benefits of different configuration models on the test set, the simplest sensor configuration (waist IMU + two hip joint motors) that meets the minimum accuracy requirements is selected as the final motion pattern recognition model with a simplified sensor configuration.

[0095] Step 3-2: Prune the network based on the motion recognition model with a simplified sensor configuration. The specific process is as follows: Using the channel pruning method from structured pruning, the model is slimmed down by removing unimportant output channels from the convolutional layers, ultimately obtaining a pruned motion pattern recognition model.

[0096] In this embodiment, an unpruned model of a pre-trained motion recognition model with a simplified sensor configuration is loaded, and the L1 norm is used to evaluate channel importance. Channels to be removed from each layer are selected based on a preset layer-wise pruning rate (in this embodiment, a pruning rate as low as 10% for shallow layers and as high as 30% for deep layers) and the L1 norm. The filters corresponding to these channels are then physically removed. Finally, the pruned model is trained on the original training set for a small number of epochs (10 in this embodiment) using a low learning rate to allow the network to adapt to the new narrow structure and restore accuracy.

[0097] Step 3-3: quantize the pruned motion pattern recognition model. In this embodiment, Figure 6This paper demonstrates a comparison of model parameter counts before and after quantization. The specific process is as follows. A post-training quantization method is used. A representative subset of unlabeled data from the training set is selected as a calibration dataset. In this example, 100-500 samples are used. A pruned model trained in PyTorch (an open source deep learning framework for machine learning and deep learning) is converted and quantized. The pruned model is exported to the ONNX (Open Neural Network Exchange) format, an open format for representing deep neural network models. The pruned model is then converted and quantized using the NNoM (a lightweight deep learning library) toolchain. During the conversion process, an INT8 quantization strategy is specified, quantizing both weights and activations to 8-bit integer format. The calibration dataset is then fed into the tool, which then outputs a quantized model file, including a lightweight motion pattern recognition model. After the quantized model is generated, it is rigorously evaluated on a PC on a complete test set, comparing its accuracy to the original floating-point model.

[0098] After completing the steps of this method, the lightweight motion pattern recognition model is deployed on an embedded device. Using the NNoM toolchain, each layer of the network structure of the lightweight motion pattern recognition model is reconstructed into a C language structure. Based on the input and output data size of each layer of the network structure, a C language array is designed to store the input and output of each layer of the network structure. Finally, the converted C language file is integrated into the hip exoskeleton control project. In this embodiment, the platform is an embedded STM32F407 microcontroller with a system frequency of 168MHz, a Flash memory size of 1MB, an SRAM1 size of 128KB, and an SRAM2 size of 64KB. During the initial model loading phase, a 32KB space is allocated for loading the model. The BSP driver for the IMU and the electric joint machine angle sensor is integrated to achieve real-time data access. In this embodiment, for motor data, the raw data is simply dimensioned; for IMU measurements, the raw data needs to be dimensioned, calibrated, and then time-synchronized with the motor sensor data. In addition, considering that the motor data and IMU data come from heterogeneous sensors, there are large differences in numerical scales. Large differences in numerical distribution can easily affect the accuracy of the model algorithm. Therefore, we normalize the motor angle distributed in [120, 300], the motor rotation speed distributed in [-180, 180], and the IMU data distributed around zero to [-1, 1]. In actual operation, the maximum measurement values ​​of different sensors on different axes in the data set used for model training are counted. and ; During inference, the original data Normalize:

[0099] ;

[0100] The normalized data Temporarily save the feed model.

[0101] An efficient circular buffer is constructed to manage data input. In this embodiment, a sliding window of fixed length 60 is maintained to update sensor data. Specifically, an array of length 60 is defined to store normalized IMU measurements and motor data in chronological order. When an index reaches the end of the array, it is automatically reset to zero, and the loop continues. This is similar to a C++ double-ended queue (DQ), but manually maintaining a circular array reduces computational overhead and storage consumption. During inference, considering that model inference takes a long time, while exoskeleton sensor data reading and underlying control require strict and accurate control cycles, model inference is placed in the main thread loop, while other sensor reading and control logic are placed in timed interrupts. The underlying control logic has the highest priority, while model inference has the lowest. During actual processing, the model automatically reads the latest index in the array, loads the sensor data sequence in chronological order, infers the motion pattern, and ultimately identifies the hip exoskeleton wearer's motion pattern in real time for subsequent control strategy switching.

[0102] Finally, in actual prototype testing, this embodiment achieved gait recognition accuracy exceeding 98%, with a delay of less than 0.5s between gaits and a short response time. After quantization, the inference speed increased by 4 times, with a single inference time of less than 13ms.

[0103] The present invention provides a system for hip exoskeleton motion pattern recognition based on deep learning, comprising the following modules:

[0104] The hip exoskeleton data acquisition module is used to collect the wearer's motion data and combine it with external input motion labels to construct a motion pattern recognition dataset;

[0105] A motion pattern recognition module, which develops and trains a hip exoskeleton motion pattern recognition network model based on the motion pattern recognition dataset, extracts motion features, and recognizes terrain and wearer motion patterns;

[0106] The quantized deployment module is used to lightweight the motion pattern recognition network model and deploy it to the embedded single-chip microcomputer on the hip exoskeleton device, so that the hip exoskeleton can recognize the wearer's motion pattern in real time.

[0107] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0108] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A hip exoskeleton motion pattern recognition method based on deep learning, characterized in that: The following steps are involved: Step 1: Use the sensors on the hip exoskeleton to capture the wearer's real-time motion data, collect and save the motion data to obtain a data set, generate motion label true values ​​for the data set through the corresponding host computer, and establish a motion pattern recognition data set; Step 2: constructing a network model structure based on the motion pattern recognition dataset, realizing feature fusion of heterogeneous sensor multimodal data, outputting terrain and motion pattern recognition results, and training the network model on the motion pattern recognition dataset to achieve convergence, thereby obtaining a motion pattern recognition network model; Step 3: Based on the motion pattern recognition network model, after optimizing the sensor configuration through ablation experiments, network pruning is used to compress the model parameters, and at the same time, INT8 quantization technology is combined to build a lightweight motion pattern recognition model.

2. A hip exoskeleton motion pattern recognition method based on deep learning according to claim 1, characterized in that: The step 1 specifically includes the following steps: Step 1-1, collecting sensor data from an inertial measurement unit and a joint motor, wherein the sensor data from the inertial measurement unit includes acceleration information, angular velocity information, and posture information, and the sensor data from the joint motor includes angle information and angular velocity information of the wearer's thigh swing; Step 1-2: calibrate the above heterogeneous sensor data; Steps 1-3: Time synchronization of two heterogeneous sensors based on the calibrated sensor data; Steps 1-4: Record and save the data collected from the heterogeneous sensors after time synchronization is completed; Steps 1-5: The simultaneously recorded heterogeneous sensor data are imported and synchronously saved into each frame of heterogeneous sensor data to obtain a motion pattern recognition dataset.

3. The hip exoskeleton motion pattern recognition method based on deep learning according to claim 2, characterized in that: The steps 1-2 specifically include: First, perform offline internal calibration of the inertial measurement unit: place the inertial measurement unit horizontally for a period of time, collect accelerometer and gyroscope measurements, and calculate the average measurement value to calibrate the zero bias of the accelerometer and gyroscope: ； ； Where N is the total number of frames collected by the inertial measurement unit, For the k-th frame measurement, and are the raw measurements of the accelerometer and gyroscope of the inertial measurement unit, g is the acceleration of gravity, and It is the zero bias of offline calibration; Then perform real-time calibration on the collected raw data of the inertial measurement unit: ； ； in, and This is the data after calibration of the inertial measurement unit; Calibrate the sensor data of the joint motor: ； ； in, and is the original angle of the joint motor, N is the number of turns, clockwise is the positive turn, and counterclockwise is the negative turn; and The angle measurement of the joint motor after calibration; It is the joint motor zero position corresponding to the preset standing state.

4. The method for hip exoskeleton motion pattern recognition based on deep learning according to claim 3, characterized in that: The steps 1-3 are specifically as follows: The sensor data of the inertial measurement unit is linearly interpolated to obtain the joint motor data time. The sensor data of the inertial measurement unit is used to complete the time synchronization of the collected data of the two heterogeneous sensors: ； ； in, 、 Joint motor data time Left and right adjacent inertial measurement unit data frames; and are the times corresponding to the data frames respectively; and This is the measurement after the inertial measurement unit and joint motor data acquisition time are synchronized.

5. The method for hip exoskeleton motion pattern recognition based on deep learning according to claim 4, characterized in that: The steps 1-4 are specifically as follows: Save the accelerometer data in the xyz three axes after the inertial measurement unit is synchronized with the joint motor time And the gyroscope's three-axis data in xyz ; For the angle and angular velocity information of the joint motor, scale it and save it; The joint motor rotation angle is scaled as: ； in, is the angle after scaling, is the angle after motor calibration, and is the minimum and maximum value of the motor angle during the movement. is the angle scaling factor; The angular velocity of the joint motor is scaled as: ； in, is the scaled angular velocity, is the motor rotation angular velocity, and is the minimum and maximum value of the motor's angular velocity during the movement. is the angular velocity scaling factor.

6. The method for hip exoskeleton motion pattern recognition based on deep learning according to claim 5, characterized in that: The step 2 specifically includes the following steps: Step 2-1, dividing the motion pattern recognition dataset into a training set and a test set; Step 2-2, performing data preprocessing on the training set; Step 2-3: Use the preprocessed data as input, combine the deep learning network and MobileNet, and build a motion pattern recognition network model; Step 2-4: Utilize a stochastic gradient descent optimizer on the motion pattern recognition network model, and set an early stopping condition. After the condition is met, terminate the training and save the best model as the final motion pattern recognition network model.

7. The method for hip exoskeleton motion pattern recognition based on deep learning according to claim 6, characterized in that: The step 2-2 is specifically as follows: Defining the data of the inertial measurement unit Format: }； in, is the acceleration information, is the angular velocity information, i is the three coordinate axes; Define the data of the joint motor Format: ； in, and Respectively represent the lifting angles of the left and right legs. The initialization angle for static standing is 180. The forward lifting angle is the positive direction, and the reverse is the negative direction. are the angular velocities of the left and right legs respectively, the forward lifting angle is the positive direction, and the reverse is the negative direction; Data on the inertial measurement unit The data of the joint motor To splice: ； in, After splicing, it is used as input data for the motion pattern recognition network model.

8. The method for hip exoskeleton motion pattern recognition based on deep learning according to claim 7, characterized in that: The step 3 specifically includes the following steps: Step 3-1, simplifying the sensor configuration of the inertial measurement unit and the joint motor, inputting the simplified sensor configuration into the final motion pattern recognition network model and training the final motion pattern recognition network model, recording the motion pattern recognition accuracy of the trained final motion pattern recognition network model on the test set, and selecting the model with the highest motion pattern recognition accuracy within a complexity less than a threshold as the motion pattern recognition model with the simplified sensor configuration; Step 3-2: The motion pattern recognition model based on the simplified sensor configuration adopts the channel pruning method in structured pruning to finally obtain a pruned motion pattern recognition model; Step 3-3: Convert the pruned motion pattern recognition model to the universal ONNX format and quantize it into 8-bit integer weights and activations. Use the NNoM toolchain to convert it into a C language motion pattern recognition model file and deploy it to the MCU on the exoskeleton. At the same time, after the inertial measurement unit and joint motor data are calibrated and time-synchronized on the microcontroller, normalization is added to unify the inertial measurement unit data and joint motor data distributed at different data scales; and an efficient circular buffer is built on the microcontroller to manage the normalized inertial measurement unit data and joint motor data; the motion recognition model reads the inertial measurement unit data and joint motor data in the buffer in chronological order to infer the motion pattern, thereby enabling the exoskeleton to recognize the wearer's motion pattern.

9. The system for hip exoskeleton motion pattern recognition method based on deep learning according to claims 1-8, characterized in that: Includes the following modules: The hip exoskeleton data acquisition module is used to collect the wearer's motion data and combine it with external input motion labels to construct a motion pattern recognition dataset; A motion pattern recognition module, which develops and trains a hip exoskeleton motion pattern recognition network model based on the motion pattern recognition dataset, extracts motion features, and recognizes terrain and wearer motion patterns; The quantized deployment module is used to lightweight the motion pattern recognition network model and deploy it to the embedded single-chip microcomputer on the hip exoskeleton device, so that the hip exoskeleton can recognize the wearer's motion pattern in real time.

Images