A deep learning-based hip exoskeleton motion pattern recognition method and system

Abstract

The application discloses a kind of based on deep learning's hip exoskeleton movement mode identification method and system.The method includes: data acquisition and processing: through the inertial measurement unit and joint motor of hip exoskeleton, heterogeneous sensor data are acquired;Offline / real-time calibration and time synchronization are carried out to data, and data set is constructed in combination with motion label;Model construction and training: splice multimodal sensor data as input, construct deep learning network, fuse heterogeneous features and identify terrain and movement mode, train model and save optimal model;Lightweight deployment: through ablation experiment optimization sensor configuration, adopt channel pruning and compress model parameter quantity, and generate light weight model in combination with INT8 quantization technology.The system includes three major modules of data acquisition, mode identification, quantization deployment.The application solves the problem of heterogeneous sensor data fusion and real-time deployment, significantly improves the precision and efficiency of exoskeleton movement recognition.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of exoskeleton intelligent control, and particularly relates to a hip exoskeleton motion mode recognition method and system based on deep learning. BACKGROUND

[0002] For people with limited or weak lower limb function, the current traditional auxiliary means has limited function, and relying on professional rehabilitation physicians faces the dilemma of shortage of manpower, uneven effect and low efficiency. Therefore, innovative technologies are needed to solve the rehabilitation needs. Exoskeleton technology provides a very potential solution to the above problems. The current traditional exoskeleton has poor intelligence level, and provides assistance through preset rules, which restricts its application in complex motion scenarios. Based on artificial intelligence technology, the human-machine state can be perceived and active assistance can be provided to effectively assist daily activities and promote rehabilitation training. Among them, the precise gait and terrain recognition ability significantly improves the core performance and safety of the exoskeleton.

[0003] The sensor system of the prior art is too complex. Chinese patent application file CN120093566A discloses an ankle joint hybrid exoskeleton system fused with functional electrical stimulation, which needs to precisely attach electrode sheets to the tibialis anterior muscle and the soleus muscle, making it difficult for patients to operate independently, and limiting the universality of the system. Chinese patent application file CN119839839A relies on 7 types of sensing devices including CMOS, radar, GPS-RTK, double-stage inertial sensor, foot pressure sensor, etc., which has high difficulty in hardware integration and greatly increases the manufacturing cost and maintenance burden. At the same time, a large amount of sensor data requires high computing power, and real-time performance cannot be guaranteed on a platform with scarce computing power. SUMMARY

[0004] The application aims to provide a hip exoskeleton motion mode recognition method based on deep learning to solve the technical problems of heterogeneous sensor data fusion and real-time deployment, and significantly improve the accuracy and efficiency of exoskeleton motion recognition.

[0005] To achieve the purpose of the application, on the one hand, the application provides a hip exoskeleton motion mode recognition method based on deep learning, which includes the following steps:

[0006] Step 1, using sensors on the hip exoskeleton to capture real-time motion data of the wearer, collecting and saving the motion data to obtain a data set, generating motion label true values through a corresponding host computer, and establishing a motion mode recognition data set;

[0007] Step 2, based on the motion mode recognition data set, a network model structure is constructed to realize feature fusion of heterogeneous sensor multi-modal data, output terrain and motion mode recognition results, and train the network model on the motion mode recognition data set to make it converge, and obtain a motion mode recognition network model.

[0008] Step 3, based on the motion pattern recognition network model, after optimizing the sensor configuration through ablation experiment, the network pruning compression model parameter amount is adopted, and the INT8 quantization technology is combined to construct a lightweight motion pattern recognition model.

[0009] In another aspect, the application also provides a system of a hip exoskeleton motion pattern recognition method based on deep learning, comprising the following modules:

[0010] A hip exoskeleton data acquisition module is used to acquire wearer motion data and combine external input motion labels to construct a motion pattern recognition data set;

[0011] A motion pattern recognition module is used to develop and train a hip exoskeleton motion pattern recognition network model based on the motion pattern recognition data set, extract motion features, and recognize terrain and wearer motion patterns;

[0012] A quantization deployment module is used to deploy the lightweight motion pattern recognition network model to the embedded single-chip microcomputer carried by the hip exoskeleton device, so that the hip exoskeleton can recognize the wearer's motion pattern in real time.

[0013] Compared with the prior art, the significant progress of the application is that: (1) the inertial sensor installed in the waist of the exoskeleton and the motion data information of the motor encoder on both sides of the hip are adopted, which greatly simplifies the complexity and cost of the sensor system; (2) the application efficiently extracts motion data features by combining artificial intelligence technology, and realizes accurate gait and terrain recognition ability; (3) the application constructs a hip exoskeleton system through self-developed hip exoskeleton devices carrying various sensors and a deep learning-based exoskeleton motion pattern recognition method, which achieves more than 98% classification accuracy in five types of gait recognition tasks such as walking, uphill and downhill, and sitting. After quantization, the parameter amount is only 20KB. The actual measurement reasoning time on the low-power embedded single-chip microcomputer side is only 13ms.

[0014] To make the function and structure of the application clearer, the following further describes the application in combination with the drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the application, and form a part of the application. The illustrative embodiments of the application and their descriptions serve to explain the application, and do not constitute an improper limitation on the application. In the drawings:

[0016] Figure 1 is a deep learning-based exoskeleton motion pattern recognition method block diagram of the application;

[0017] Figure 2It is a hip exoskeleton data acquisition module diagram of the application;

[0018] Figure 3 It is a specific layout diagram of the exoskeleton sensor of the application;

[0019] Figure 4 It is a visualization image diagram of different gait motion data of the exoskeleton sensor of the application;

[0020] Figure 5 It is a hip exoskeleton motion pattern recognition network model structure diagram of the application;

[0021] Figure 6 It is a parameter quantity comparison diagram before and after the quantization of the hip exoskeleton motion pattern recognition model of the application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0023] The hip exoskeleton motion pattern recognition method based on deep learning of the application combines Figure 1 , and comprises the following steps:

[0024] Step 1, the real-time motion data of a wearer is captured by using the sensors on the hip exoskeleton, the motion data is collected and saved to obtain a data set, the data set is used to generate motion label true values by a corresponding host computer, and a motion pattern recognition data set is established;

[0025] Step 2, a network structure is constructed based on the motion pattern recognition data set, feature fusion of multi-modal data of heterogeneous sensors is realized, a terrain and motion pattern recognition result is output, and the network model is trained on the motion pattern recognition data set to make it converge, so as to obtain a motion pattern recognition network model;

[0026] Step 3, based on the motion pattern recognition network model, after optimizing the sensor configuration through an ablation experiment, the number of sensors used is reduced to reduce the model complexity and optimize the calculation overhead under the premise of ensuring the recognition accuracy, the network pruning is used to compress the model parameter quantity, the model weight is quantized by combining the INT8 quantization technology, the calculation complexity is reduced, the inference efficiency is improved, a lightweight motion pattern recognition model is constructed, and the lightweight motion pattern recognition model is deployed to the MCU installed on the exoskeleton, so as to realize the motion pattern recognition of the exoskeleton on the wearer.

[0027] The step 1 specifically comprises the following steps:

[0028] Step 1-1, collecting sensor data of a waist inertial measurement unit, a left thigh inertial measurement unit, a right thigh inertial measurement unit and a joint motor, the sensor data of the waist inertial measurement unit comprising acceleration information, angular velocity information of a wearer waist root node, the sensor data of the left thigh inertial measurement unit comprising acceleration information, angular velocity information of a wearer left thigh close to a knee joint position, the sensor data of the right thigh inertial measurement unit comprising acceleration information, angular velocity information and attitude information of a wearer right thigh close to a knee joint position, and the joint motor sensor data comprising angle information and angular velocity information of a wearer thigh swing;

[0029] The hip exoskeleton data acquisition module of the embodiment is shown in Figure 2 As shown in the figure, an STM32F407IGT6 is selected as a master control unit MCU (Microcontroller Unit); a joint motor installed on the left and right sides of the hip of the hip exoskeleton device contains a built-in motor magnet encoder, which is connected to the MCU communication through a CAN (Controller Area Network) line, and is used to record the angle information and angular velocity information of a wearer thigh swing; an inertial measurement unit IMU (Inertial Measurement Unit) is installed on the exoskeleton fitting waist position and the left and right thigh close to the knee joint position, and the IMU communicates with the MCU through an IIC (Inter-Integrated Circuit), and is used to record the acceleration information, angular velocity information and attitude information of a wearer waist root node and left and right thighs;

[0030] The motor receives information through inquiry, and in the exoskeleton system of the embodiment, we set the control period to 10 ms, and periodically inquire the motor joint angle and angular velocity in the 10 ms timing interrupt, so the motor message frequency is 100 Hz. We set the ODR (output data register) update frequency of the IMU to 500 Hz, and in the MCU, set the IIC clock frequency to 100 kHz, and set 2 ms timing interrupt to read the IMU raw data, so the IMU message frequency is 500 Hz;

[0031] Step 1-2, calibrating the above heterogeneous sensor data;

[0032] Step 1-3, time synchronizing two heterogeneous sensors based on the calibrated sensor data;

[0033] Step 1-4, recording and saving the data of the heterogeneous sensors collected after time synchronization.

[0034] Step 1-5, the isomerous sensor data recorded at the same time is saved to each frame of isomerous sensor data through the imported motion label data and is synchronized to obtain a motion pattern recognition data set.

[0035] The motion label making process is as follows: the wearer or the helper connects the exoskeleton Bluetooth through an electronic information device (mobile phone, tablet, notebook computer, etc.) to receive sensor messages and issue motion labels. Taking the mobile phone APP label making as an example, when the participant performs different modes of motion, after clicking the different motion action buttons on the APP, the motion label is sent to the exoskeleton device to label the motion state at the current time. The motion label occupies 1 byte, the start collection label is 0, standing is 1, walking is 2, going upstairs is 3, going downstairs is 4, motion switching state is 5, uphill is 6, downhill is 7, and running is 8. The motion label is sent to the exoskeleton device to fill into the corresponding frame of isomerous sensor data, and all data frames are continuously saved to build a complete motion pattern data set. The constructed data set is used for subsequent model construction and testing.

[0036] The step 1-2 specifically includes:

[0037] First, the IMU is calibrated offline: the IMU is horizontally placed and horizontally placed for about 15 minutes. In theory, the accelerometer measurement is zero during the IMU is stationary period; but the actual IMU has a zero offset, and the and are calculated, and the average measurement value is used to calibrate the zero offset of the accelerometer and the gyroscope:

[0038] ;

[0039] ;

[0040] wherein N is the total number of frames collected by the inertial measurement unit, represents the kth frame measurement, and are the original measurements of the accelerometer and the gyroscope of the IMU, g is the gravitational acceleration, and are the offline calibrated zero offsets;

[0041] Then, the collected inertial measurement unit original data is calibrated in real time:

[0042] ;

[0043] ​​;

[0044] wherein, and are calibrated data of the inertial measurement unit;

[0045] In the process of installing the joint motor to the exoskeleton, there is a difference in the installation rotation angle, which causes the mechanical zero of the motor of different devices to be inconsistent. Therefore, it is stipulated to wear the exoskeleton in a standing state and start, and the sensor data of the joint motor is calibrated at the time of starting to realize the consistency of the zero of the motor of different devices. At the same time, in order to avoid the jump of the motor angle from 180° to -180° in normal motion, the actual human joint can be lifted forward and backward from the standing state respectively, and the angle of the standing state is set to in the initialization stage through software setting. At the same time, by adding a circle detection, it is stipulated that the leg lifting of the left and right legs is a positive direction. Through the above operation, the motor angle of the left and right legs has an opposite change trend in normal motion, and the data distribution difference is large, which is beneficial to the convergence of the model.

[0046] The sensor data of the joint motor is calibrated:

[0047] ;

[0048] ;

[0049] wherein, and are the original angles of the joint motor, N is the number of over-coil, clockwise is a positive over-coil, and counterclockwise is a reverse over-coil; and are the calibrated joint motor angle measurements.

[0050] In actual use, for multiple exoskeletons, it is not necessary to additionally set the mechanical zero and the positive direction of rotation of the motor, which greatly reduces the workload.

[0051] The steps 1-3 are specifically: since the IMU and the joint motor are not designed with synchronous trigger collection at the hardware bottom layer, the IMU data and the joint motor data collected are not consistent in the time stamp. The IMU and the motor hardware are connected to the same MCU, and the two sensors use the same clock source time; the IMU data is linearly interpolated to the motor measurement time stamp, and the data time synchronization of the heterogeneous sensors is realized. In the embodiment, a 2ms high-frequency timing interrupt is set at the MCU bottom layer, the IMU register is read in the timing interrupt, and the real physical measurement value is obtained, and the single-chip microcomputer time is recorded; the motor information is queried and read, and a 10ms timing is set to send a motor query instruction, and the motor data time is recorded in the receiving interrupt ;

[0052] The calibrated IMU sensor data and Linear interpolation is performed to obtain the sensor data of the inertial measurement unit at the joint motor data acquisition time The acquisition data time synchronization of the two heterogeneous sensors is completed:

[0053] ;

[0054] ;

[0055] wherein, 、 is the joint motor data time left and right adjacent inertial measurement unit data frames; and are the corresponding times of the data frames, respectively; and are the measurements of the IMU and the joint motor data acquisition time synchronization. Since the sampling frequency of the inertial measurement unit is very high, the time synchronization error of the IMU and the motor two heterogeneous sensors is small and can be ignored by the interpolation approximation method.

[0056] The steps 1-4 are specifically: in the embodiment, each measurement value is int16 type and occupies 2 bytes, so the length of a frame of IMU data is 12 bytes, and the measurements of the waist IMU, the left thigh IMU and the right thigh IMU are integrated into the data length of one frame, which is 36 bytes;

[0057] Save the accelerometer xyz three-axis data of the three inertial measurement units and the left and right two joint motors after time synchronization and the xyz three-axis data of the gyroscope ; for the angle and angular velocity information of the joint motor, the angle and angular velocity information of the joint motor are scaled and saved;

[0058] In the embodiment, the angle θ and the rotation speed are float type and occupy four bytes, which are scaled to uint16 occupying 2 bytes to reduce the data size, and the scaling coefficient , is determined, and the left and right motor angle and angular velocity data length is 8 bytes after scaling. Specifically, the maximum and minimum values of the motor angle are determined in combination with the motor and leg structure of the exoskeleton device and the joint limit of the human body, and scaling is performed. In actual upright standing, the motor angle is 180°; the motor angle decreases when the leg is lifted backward, and the minimum value is ; when the leg is lifted upward, the motor angle increases, and the maximum value is The joint motor rotation angle is scaled as:

[0059] ;

[0060] wherein is the scaled angle, is the calibrated angle of the motor.

[0061] For the motor rotation speed, in combination with the exoskeleton device motor and leg pole structure, human joint limit, the maximum value is determined as , the minimum value is , and the scaled motor rotation speed is obtained by the following formula

[0062] ;

[0063] wherein is the scaled angular velocity, is the motor rotation angular velocity before scaling.

[0064] In the data collection, a fixed 2-byte frame header and a 2-byte frame tail are added, wherein 4-byte timestamps, 36-byte IMU time-synchronized data, 8-byte scaled motor data, and 1-byte corresponding motion labels are filled to synthesize a frame of data. At the same time, the wearer wears the exoskeleton in the initialization stage to keep standing and static, and waits for the IMU and motor initialization to be completed. After completing the sensor initialization, the wearer starts to move. A series of different motion mode activities such as going upstairs, going downstairs, walking, static, uphill and downhill are completed, and during the period, multiple frames of heterogeneous sensor data of the wearer during the motion are continuously recorded and saved to the 2Gbit storage Nand device on the exoskeleton board.

[0065] The steps 1-5 are specifically: synchronously recording the heterogeneous sensor data, importing the motion label data through the outside and synchronously saving to each frame of heterogeneous sensor data. In the embodiment, the motion label making process is as follows: the wearer or the helper connects the exoskeleton Bluetooth through the mobile phone APP for issuing the motion label. When the participant performs different modes of motion, the motion state at the current time is labeled on the mobile phone AP as the motion label, different motion modes correspond to different motion labels, the motion label is sent to the exoskeleton device and filled into the corresponding frame of heterogeneous sensor data, and all data frames are continuously saved to build a complete motion mode data set.

[0066] After collecting the data by wearing the exoskeleton, the data in the 2Gbit storage Nand device on the board is uploaded by the mobile phone APP control exoskeleton, the data uploaded by the exoskeleton is received on the computer, and the real physical unit IMU and joint motor measurement are restored according to the scaling formula, and the motion mode recognition data set constructed in combination with the motion label is used for subsequent model construction and test.

[0067] The step 2 specifically includes the following steps:

[0068] ​Step 2-1, divide the motion pattern recognition data set into a training set and a test set;

[0069] Two division methods are used to verify the effectiveness of the network on different data. Method one is to divide according to the recording personnel, and completely isolate the data of different participants, and divide according to the ratio of 8:2. The migration of the motion pattern recognition model on different individuals is verified. Method two is to divide according to time, and isolate the data set according to the data recording time, and verify the generalization of the motion pattern recognition model on data in different periods. The training set and the test set are output for model structure design and training.

[0070] In this embodiment, the training set contains more than 1.2 million frames of motion data, and the test set contains 300,000 frames of data. The data set contains 6 sensors, and 7 types of gait patterns.

[0071] Step 2-2, data preprocessing is performed on the training set; in this embodiment, Figure 3 , Figure 4 The specific layout of the exoskeleton sensor and the visualization image of different gait motion data in this embodiment are shown respectively,

[0072] Definition of data format of inertial measurement unit :

[0073] };

[0074] Wherein, is acceleration information, is angular velocity information, and i is the three coordinate axes; that is, a single frame of waist inertial measurement unit information contains six dimensions;

[0075] Definition of data format of joint motor :

[0076] ;

[0077] Wherein, and represent the lifting angles of the left and right legs respectively, and the initial angle of static standing is 180, the forward lifting angle is in the positive direction, and vice versa is in the negative direction, are the lifting angular velocities of the left and right legs respectively, and the forward lifting angle is in the positive direction, and vice versa is in the negative direction;

[0078] The above two types of sensor data are inconsistent in dimension, and cannot be stacked in another dimension. Therefore, in order to simultaneously use data of different modalities for pattern recognition, the joint motor and IMU data are preprocessed and spliced as input data of the motion pattern recognition network model;

[0079] The data of the inertial measurement unit Data of the joint motor Concatenation:

[0080] ;

[0081] wherein, is the input data of the motion pattern recognition network model after concatenation, , and are the inertial measurement unit data installed on the waist, left thigh and right thigh, respectively.

[0082] In the motion pattern recognition network model constructed in this embodiment, the multi-modal data is flattened before being input into the network, and IMU data with a length of 6 and motor data with a length of 4 are obtained. In order to ensure that the network design is light enough, the IMU and motor data are concatenated in the same dimension before being input into the network, that is, a single frame of data is expressed as a one-dimensional data with a length of 10.

[0083] Step 2-3, taking the preprocessed data as input, combining deep learning technology and MobileNet network structure, constructing an exoskeleton motion pattern recognition network model;

[0084] In order to ensure the light weight of the parameters of the exoskeleton motion pattern recognition network model, the network only uses one-dimensional convolution structure to model the input data. In terms of input, considering that one gait in normal activities lasts about 0.5-0.8 seconds, combined with the sampling frequency of the sensor, the size of the time window is designed to be 60.

[0085] In order to ensure that the exoskeleton motion pattern recognition network model is as light as possible while retaining sufficient data fitting capability, the design idea of MobileNet is borrowed. The data is upgraded at the front end of the exoskeleton motion pattern recognition network model, aiming to amplify the differences in data characteristics of different motion patterns in high-dimensional space, so that the model can more easily learn the differences in the distribution of sensor motion data between different motion patterns. Then, the preprocessed and concatenated sensor data in step 2-2 is processed by dimension reduction, reducing the exoskeleton motion pattern recognition network model parameters of the subsequent classification layer, so that the model can run on the low-power end side.

[0086] In this embodiment, the input data is first standardized by a batch normalization layer at the front end of the network, and the mathematical expression is:

[0087] ;

[0088] wherein and The mean and standard deviation of each channel, respectively. The feature extraction module adopts the strategy of dimensionality reduction after dimensionality reduction: the first convolutional layer expands the channel number from 8 to 32, extracts the time feature through a 3x1 convolutional kernel, and uses the high-dimensional space to amplify the difference of motion patterns; the second convolutional layer compresses the channel number to 16, retains the key features while significantly reducing the parameter size. After flattening the feature map, it is compressed to a 298-dimensional feature vector through a maximum pooling layer with a kernel size of 3, and finally mapped to the classification space by the fully connected layer.

[0089] As shown in the specific network structure. Figure 5 The data dimension changes during the entire forward propagation process: input tensor [batch_size, 10, 60] → batch normalization [10, 60] → first convolution [32, 58] → second convolution [16, 56] → flatten

[896] → pooling

[298] → classification output [7].

[0090] Then the data is reduced in dimension to reduce the network parameters of the subsequent classification layer and ensure that the model can run on the low-performance end side.

[0091] Step 2-4, the motion pattern recognition network model is optimized using a stochastic gradient descent optimizer, and an early stopping condition is set. When the condition is reached, the training is terminated and the best model is saved as the final motion pattern recognition model.

[0092] Step 2-4 is as follows: initialize the model parameters according to the preset network structure, calculate the cross-entropy loss according to the predicted output and the true label. Combined with momentum and weight decay, use the stochastic gradient descent (SGD) optimizer to update the model parameters according to the calculated gradient. Continuously monitor the performance of the test set, set an early stopping condition, and terminate training when the condition is reached. The best model is saved as the motion pattern recognition model.

[0093] The step 3 specifically includes the following steps:

[0094] Step 3-1, based on the motion pattern recognition model, reduce the number of sensors used by the exoskeleton pattern recognition to lightweight the network model structure and data input size, the specific details are as follows. Reduce the number and type of sensors required step by step to reduce system complexity, power consumption and cost while ensuring recognition accuracy. In this embodiment, a high-precision baseline model is trained on the most complete waist IMU, left thigh IMU, right thigh IMU and left and right joint motor sensor configuration dataset, and its performance on the test set is recorded as a benchmark. Using a progressive ablation study and retraining strategy, based on the sensor configuration in the motion pattern recognition dataset, an ablation experiment is designed using part of the sensor combination, starting to remove sensors, removing left and right leg IMU data, using the same network structure and training process as the baseline model, retraining multiple network models, recording their performance on the test set for comparison, and focusing on the accuracy change on unseen individuals. According to the performance of the retrained multiple models, further simplify and remove the Euler angle attitude data in the IMU data, and only retain the original data of the six-axis IMU. Retrain and evaluate the corresponding model. Trade off model structure complexity and recognition accuracy, and select the model with the highest recognition rate within the threshold complexity as the simplified sensor configuration motion pattern recognition model. In this embodiment, by comparing the performance, robustness and hardware benefits of different configuration models on the test set, the simplest sensor configuration (waist IMU + hip 2 joint motors) is selected as the final simplified sensor configuration motion recognition model under the premise of meeting the minimum accuracy requirement.

[0095] Step 3-2, based on the simplified sensor configuration motion recognition model, perform network pruning, the specific process is as follows. Adopt the channel pruning method in structured pruning to achieve model slimming by removing unimportant output channels in convolutional layers, and finally obtain a pruned motion pattern recognition model.

[0096] In this embodiment, the unpruned model of the trained simplified sensor configuration motion recognition model is loaded, and the L1 norm criterion is used to evaluate the importance of the channel. According to the pre-set hierarchical pruning rate (in the specific embodiment, the shallow layer pruning rate is low, such as 10%, and the deep layer pruning rate is high, such as 30%) and the L1 criterion, the channels to be removed in each layer are selected, and then the filters corresponding to these channels are physically removed. Finally, the pruned model is trained for a small number of epochs (10 in this embodiment) on the original training set with a small learning rate, so that the network adapts to the new narrow structure and restores the accuracy.

[0097] Step 3-3, quantize the pruned motion pattern recognition model, in this embodiment, Figure 6The model parameter quantity before and after model quantization is shown, and the specific process is as follows. The post-training quantization method is adopted. A representative unlabeled data subset is selected from the training set as a calibration data set, 100-500 samples are used in this embodiment, the pruned model trained by PyTorch (an open source deep learning framework for machine learning and deep learning) is converted and quantized, the pruned model is exported to ONNX (Open Neural Network Exchange, an open format for representing deep neural network models) format, and then converted and quantized using NNoM (a lightweight deep learning library) tool chain. In the conversion process, specify the INT8 quantization strategy, quantize the weights and activations to 8-bit integer format, and input the calibration data set into the tool. The NNoM tool outputs a quantized model file, and outputs the lightweight motion pattern recognition model. After the quantized model is generated, the quantized model is strictly evaluated on the complete test set on the PC side, and the precision difference between the original floating point model and the quantized model is compared.

[0098] After the steps of the method, the lightweight motion pattern recognition model is deployed to an embedded device, the NNoM tool chain is used to reconstruct each layer of the lightweight motion pattern recognition model network structure into a structure in C language, and an array in C language is designed according to the input and output data size of each layer of the network structure to store the input and output of each layer of the network structure. Finally, the converted C language file is integrated into the hip exoskeleton control project. In this embodiment, the platform is an embedded single-chip microcomputer STM32F407, the system main frequency is 168MHz, the Flash size is 1MB, the SRAM1 size is 128KB, and the SRAM2 size is 64KB. In the initialization model loading stage, a space of 32KB is allocated for loading the model. The BSP driver of the IMU and the motor joint angle sensor is integrated to realize real-time data access. In this embodiment, for motor data, the original data is restored to the dimension; for IMU measurement, the original data is restored to the dimension, and then data calibration is performed, and then the motor sensor data is time-synchronized. In addition, considering that the motor data and the IMU data are from heterogeneous sensors, there is a large difference in numerical scale, and the numerical distribution difference is large, which easily affects the model algorithm accuracy. Therefore, we normalize the motor angle distributed in [120, 300], the motor rotation speed distributed in [-180, 180], and the IMU data distributed around zero to [-1, 1]. In actual operation, the maximum value of different sensors in different axes in the data set used for model training is and ; during inference, the original data is normalized:

[0099] ;

[0100] The normalized data The temporary feeding model is temporarily fed.

[0101] An efficient circular buffer is constructed to manage data input. In this embodiment, a sliding window of fixed length 60 is maintained to update sensor data. That is, an array of length 60 is defined to store normalized IMU measurements and motor data in chronological order. When the index increases to the end of the array, the index is automatically set to zero, and the cycle is repeated. The effect is similar to a double-ended queue in C++, but a circular array is manually maintained, with lower computational overhead and storage consumption. During inference, considering that model inference takes a long time, and exoskeleton sensor data reading and underlying control require strict and accurate control period. Therefore, we place the model inference in the main thread loop, and other sensor reading and control logic in the timing interrupt. The underlying control logic has the highest priority, and the model inference has the lowest priority. In actual processing, the model automatically reads the latest index of the array, loads the sensor data sequence in the array in chronological order, infers the motion pattern, and finally identifies the hip exoskeleton wearer's motion pattern in real time, which is used for subsequent control strategy switching.

[0102] Finally, in the actual prototype test of this embodiment, the gait recognition accuracy is higher than 98%, the different gait switching delay is lower than 0.5s, and the response time is short. After quantization, the inference speed is improved by 4 times, and the single inference time is less than 13ms.

[0103] The system of the hip exoskeleton motion pattern recognition method based on deep learning of the application comprises the following modules:

[0104] The hip exoskeleton data acquisition module is used for acquiring wearer motion data and combining external input motion labels to construct a motion pattern recognition data set.

[0105] The motion pattern recognition module is used for developing and training a hip exoskeleton motion pattern recognition network model based on the motion pattern recognition data set, extracting motion features, and recognizing terrain and wearer motion patterns.

[0106] The quantization deployment module is used for deploying the lightened motion pattern recognition network model to an embedded single-chip microcomputer carried by the hip exoskeleton device, so that the hip exoskeleton can recognize the wearer's motion pattern in real time.

[0107] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the same general inventive concepts embodied by the described embodiments. That is, although the present application is described in terms of particular embodiments and illustrative figures, it should be apparent that the scope of the present application is not limited to these specific embodiments.

[0108] While the embodiments of the application have been shown and described herein, it is to be understood that the scope of the application, jointly pointed out in the appended claims, is not to be limited to the above-described embodiments but can be otherwise variously changed, modified, replaced, and altered within the principles and spirit of the present application.

Claims

1. A deep learning-based hip exoskeleton motion pattern recognition method, characterized in that, It comprises the following steps: Step 1, capturing real-time motion data of the wearer by using sensors on the hip exoskeleton, collecting and saving the motion data to obtain a data set, generating motion label true value by corresponding host computer, and establishing motion pattern recognition data set; Step 2, constructing network model structure based on the motion pattern recognition data set, realizing feature fusion of multi-modal data of heterogeneous sensors, outputting terrain and motion pattern recognition result, and training the network model on the motion pattern recognition data set to make it converge, obtaining the motion pattern recognition network model; the motion pattern recognition network model is constructed by combining deep learning network and MobileNet; Step 3, based on the motion pattern recognition network model, after optimizing the sensor configuration through ablation experiment, adopting network pruning to compress model parameter quantity, and combining INT8 quantization technology, constructing light weight motion pattern recognition model.

2. The deep learning-based hip exoskeleton motion pattern recognition method according to claim 1, characterized in that, The step 1 specifically comprises the following steps: Step 1-1, collecting sensor data of inertial measurement unit and joint motor, the sensor data of the inertial measurement unit including acceleration information, angular velocity information and attitude information, the joint motor sensor data including angle information and angular velocity information of wearer's thigh swing; Step 1-2, calibrating the above sensor data; Step 1-3, time synchronization of two heterogeneous sensors based on calibrated sensor data; Step 1-4, recording and saving the data of heterogeneous sensors collected after time synchronization; Step 1-5, synchronously saving the simultaneously recorded heterogeneous sensor data into each frame of heterogeneous sensor data through imported motion label data, and obtaining motion pattern recognition data set.

3. The deep learning-based hip exoskeleton motion pattern recognition method according to claim 2, characterized in that, The step 1-2 specifically comprises: First, offline internal parameter calibration of inertial measurement unit: place the inertial measurement unit horizontally for a period of time, collect accelerometer and gyroscope measurement values, and calculate average measurement values for calibrating zero offset of accelerometer and gyroscope: ； ； where N is the total number of frames collected by the inertial measurement unit, is the measurement of the kth frame, and are the raw measurements of the accelerometer and gyroscope of the inertial measurement unit, respectively, and g is the acceleration of gravity, and are the biases of the accelerometer and gyroscope, respectively, Then, real-time calibration of collected inertial measurement unit raw data: ； ； wherein, and are calibrated data from an inertial measurement unit. Calibration of sensor data of joint motor: ； ； wherein, and is the original angle of the joint motor, N is the number of over-turns, clockwise is positive over-turn, and counterclockwise is reverse over-turn; and is the calibrated joint motor angle measurement; is the preset joint motor zero position corresponding to the standing state.

4. The deep learning-based hip exoskeleton motion pattern recognition method according to claim 3, characterized in that, The step 1-3 specifically comprises: The acquisition data time synchronization of the two heterogeneous sensors is completed by linearly interpolating the sensor data of the inertial measurement unit through the joint motor data time of the inertial measurement unit. ； ； wherein, , is the time of the joint motor data is the left and right adjacent inertial measurement unit data frames; and is the time corresponding to the data frames, respectively; and is the measurement after the inertial measurement unit and joint motor data collection time synchronization.

5. The deep learning-based hip exoskeleton motion pattern recognition method according to claim 4, characterized in that, The step 1-4 specifically comprises: Save the accelerometer in xyz three-axis data after the inertial measurement unit and joint motor time synchronization and the gyroscope in xyz three-axis data ; for the angle and angular velocity information of joint motor, save after scaling Joint motor rotation angle scaling is: ； wherein, is the scaled angle, is the motor calibrated angle, and are the minimum and maximum values of the motor angle during the motion, is the angle scaling factor; Joint motor rotation angular velocity scaling is: ； wherein, is the scaled angular velocity, is the motor rotation angular velocity, and is the minimum and maximum value of the motor rotation angular velocity during the motion, is the angular velocity scaling factor.

6. The deep learning-based hip exoskeleton motion pattern recognition method according to claim 5, characterized in that, The step 2 specifically comprises the following steps: Step 2-1, dividing the motion pattern recognition data set into training set and test set; Step 2-2, data preprocessing of the training set; Step 2-3, training the motion pattern recognition network model by taking the preprocessed data as input; Step 2-4, using stochastic gradient descent optimizer for the motion pattern recognition network model, setting early stopping condition, terminating training and saving the best model as the final motion pattern recognition network model when the condition is reached.

7. The deep learning-based hip exoskeleton motion pattern recognition method according to claim 6, characterized in that, The step 2-2 specifically comprises: Data defining an inertial measurement unit Format: }； wherein is acceleration information, is angular velocity information, i is three coordinate axes; Data defining the joint motor Format: ； wherein, and θL, θR represent the lifting angle of the left and right legs, respectively, and the initial angle of the stationary standing is 180, the forward lifting angle is the positive direction, and the reverse is the negative direction, ωL, ωR represent the lifting angular velocity of the left and right legs, respectively, and the forward lifting angle is the positive direction, and the reverse is the negative direction; data of the inertial measurement unit data of the joint motors stitching: ； wherein, is the input data after stitching as the input data of the motion pattern recognition network model.

8. The deep learning-based hip exoskeleton motion pattern recognition method according to claim 7, characterized in that, The step 3 specifically comprises the following steps: Step 3-1, simplify the sensor configuration of the inertial measurement unit and joint motor, input the simplified sensor configuration into the final motion pattern recognition network model and train the final motion pattern recognition network model, record the motion pattern recognition accuracy of the trained final motion pattern recognition network model on the test set, and select the model with the highest motion pattern recognition accuracy within the threshold complexity as the motion pattern recognition model of the simplified sensor configuration; Step 3-2, based on the motion pattern recognition model of the simplified sensor configuration, adopt the channel pruning method in structured pruning, and finally obtain a pruned motion pattern recognition model; Step 3-3, convert the pruned motion pattern recognition model into a general ONNX format, and quantize it to 8-bit integer weights and activations, convert it to a C language motion pattern recognition model file using the NNoM tool chain, and deploy it on the single-chip microcomputer carried by the exoskeleton; At the same time, after calibrating and time synchronizing the inertial measurement unit and joint motor data on the single-chip microcomputer, adding normalization to normalize the inertial measurement unit data and joint motor data with different data scale distributions; and constructing an efficient circular buffer on the single-chip microcomputer to manage the normalized inertial measurement unit data and joint motor data; the motion recognition model reads the inertial measurement unit data and joint motor data in the buffer in time sequence to infer the motion pattern, realizing the motion pattern recognition of the exoskeleton on the wearer.

9. The system of a deep learning-based hip exoskeleton motion pattern recognition method according to any one of claims 1-8, characterized in that, It includes the following modules: A hip exoskeleton data acquisition module for acquiring wearer motion data and combining external input motion labels to construct a motion pattern recognition dataset; A motion pattern recognition module that develops and trains a hip exoskeleton motion pattern recognition network model based on the motion pattern recognition dataset, extracts motion features, and recognizes terrain and wearer motion patterns; A quantization and deployment module for lightweighting the motion pattern recognition network model and deploying it to the embedded single-chip microcomputer carried by the hip exoskeleton device, enabling the hip exoskeleton to recognize the wearer's motion pattern in real time.

Images