Wheelchair control method, wheelchair and equipment based on sEMG and IMU fusion

Abstract

The present invention provides a wheelchair control method, a wheelchair and a device based on the fusion of sEMG and IMU, the method comprising the following steps: obtaining synchronously collected array electromyographic signals of a human forearm and inertial signals of a hand; preprocessing the inertial signals and the array electromyographic signals; extracting features of the preprocessed array electromyographic signals and inertial signals respectively; fusing the extracted array electromyographic signal features with the inertial signal features as inputs of a gesture recognition model to obtain gesture recognition results; and controlling the movement of the wheelchair through the gesture recognition results. The present invention fuses surface electromyographic signals and IMU signals to perform control gesture recognition for an intelligent wheelchair, combining the stable characteristics of the IMU signals with the anticipatory characteristics of the movement of the surface electromyographic signals, making the gesture recognition results more accurate and able to adapt to the characteristics of the diversity of disability levels and complexity of disability types among the existing wheelchair users.

Description

Technical Field

[0001] The present invention relates to the field of intelligent wheelchair control technology, and in particular to a wheelchair control method, a wheelchair and equipment based on the fusion of sEMG and IMU. Background Art

[0002] In daily life, relatively inexpensive wheelchairs are commonly used as assistive devices for daily walking. Traditional wheels + chairs, electric wheels + chairs + control handles are the mainstream of wheelchairs. At present, the complexity of disability types and the increasing demand for travel are the basic contradictions in the development of wheelchair assistance. Traditional electric wheelchairs require users to use joysticks to control the movement of the wheelchair. Maintaining a posture for a long time will cause user fatigue and muscle soreness, which is not conducive to users with upper limb muscle damage. In addition, inadvertent posture changes in related parts will cause sudden changes in the wheelchair's driving path, causing unnecessary safety hazards, and gesture recognition requires less hand strength for patients. However, existing gesture recognition only recognizes gesture information through a single channel of electromyographic signals or video signals, which cannot adapt to the characteristics of the diversity of disability levels and complexity of disability types among the existing wheelchair users, resulting in low accuracy of gesture recognition. Summary of the invention

[0003] In order to achieve the above-mentioned purpose and other advantages of the present invention, the first purpose of the present invention is to provide a wheelchair control method based on sEMG and IMU fusion, comprising the following steps:

[0004] Acquire synchronously collected array electromyographic signals of the human forearm and inertial signals of the hand;

[0005] Preprocessing the inertial signal and the array electromyographic signal;

[0006] Perform feature extraction on the preprocessed array electromyographic signals and inertial signals respectively;

[0007] The extracted array electromyographic signal features and inertial signal features are fused as the input of the gesture recognition model to obtain the gesture recognition results;

[0008] The movement of the wheelchair is controlled according to the gesture recognition result.

[0009] Furthermore, the step of preprocessing the inertial signal and the array electromyographic signal comprises:

[0010] Performing filtering processing on the inertial signal and the array electromyographic signal;

[0011] Normalizing the array electromyographic signal by using the collected maximum voluntary contraction force signal;

[0012] A double threshold method is used to identify the start and end points of the inertial signal;

[0013] Determining the action start and end points of the array-type electromyographic signal based on the action start and end points of the inertial signal;

[0014] The array-type electromyographic signals and inertial signals within the start and end points of the action are segmented and processed respectively through a sliding window.

[0015] Furthermore, the steps of extracting features from the preprocessed array electromyographic signals and inertial signals include:

[0016] For the array electromyographic signal, extract the muscle synergy feature of each sliding window;

[0017] For the inertial signal, an inertial signal feature of each sliding window is extracted.

[0018] Furthermore, the step of extracting muscle synergy features of each sliding window includes:

[0019] For the segmented array electromyographic signal within a given time window , calculate a The Pearson correlation coefficient matrix is ​​used as the muscle synergy matrix :

[0020] ;

[0021] Where n is the number of channels of array electromyographic signals, represents the segmented electromyographic signal of the ith channel at time t, represents the segmented electromyographic signal of the jth channel at time t, , is the synergy strength between the ith and jth EMG signal channels in the time window, and is the mean of channel i and channel j, T is the total number of signal frames in a time window after data segmentation;

[0022] The muscle synergy matrix is ​​expanded to become a one-dimensional vector. The dimension of the expanded vector is (1, );

[0023] Extract the mean, standard deviation, maximum value, minimum value, and two channel positions corresponding to the maximum value of the expanded vector to obtain a feature vector as the muscle synergy feature .

[0024] Furthermore, the step of extracting muscle synergy features of each sliding window also includes:

[0025] For the muscle synergy feature, the Lasso regression model is used to perform sparse coding feature optimization.

[0026] Furthermore, the step of optimizing sparse coding features using Lasso regression includes:

[0027] A Lasso regression model is constructed for each feature dimension, and the Lasso optimization problem is:

[0028] ;

[0029] in, is the regression model, express and The error between , is the feature vector of the u-th sample, , is the total number of samples, The other feature matrices after removing the target feature from the regression model built for each feature dimension, is the regression coefficient vector, is the regularization coefficient, is the L2 norm, express and The square error between is the L1 norm, express The sum of the absolute values ​​of

[0030] Get the regularization coefficient of the input , after Lasso optimization, the converged regression coefficient vector is obtained , After iterative updates, the coefficients of some feature dimensions are reduced to 0, and finally the sparsely optimized features are obtained. .

[0031] Furthermore, the step of fusing the extracted array electromyographic signal features with the inertial signal features as input of the gesture recognition model includes:

[0032] The inertial signal features extracted from each time window are combined with the muscle synergy features after sparse coding feature optimization into a feature vector as the input of the gesture recognition model.

[0033] Furthermore, the gesture recognition model adopts a pseudo-label semi-supervised learning network model.

[0034] Furthermore, the training of the pseudo-label semi-supervised learning network model includes an initialization training process and a training process using unlabeled data; wherein,

[0035] The initialization training process includes:

[0036] Initialize and train the student model and teacher model using labeled data until the model converges to obtain the pre-trained student model and teacher model.

[0037] The training process using unlabeled data includes:

[0038] The feature vector formed by combining the inertial signal features extracted from each time window with the muscle synergy features after sparse coding feature optimization is input into the pre-trained teacher model as unlabeled data, and the recognition result is output;

[0039] The recognition results are screened, and the results with a confidence level greater than a threshold are selected as pseudo labels to obtain pseudo label data;

[0040] Using the pseudo-labeled data and the labeled data to train the pre-trained student model, using the total loss to back-propagate and update the student model, and using the exponential moving average method to update the teacher model;

[0041] The training process using unlabeled data is repeatedly performed until the student model converges, and finally a trained student model is obtained.

[0042] Furthermore, the step of controlling the movement of the wheelchair by using the gesture recognition result includes:

[0043] According to the gesture recognition results, the values ​​of various parameters in the topic published by the gesture recognition node are set to control the wheelchair to perform corresponding actions.

[0044] The second object of the present invention is to provide an intelligent wheelchair, which applies the above method, including a gesture recognition node, a wheelchair chassis motion control node, and a wheelchair body. The gesture recognition node is used to perform gesture recognition based on the synchronously collected array electromyographic signals of the human forearm and the inertial signals of the hand and publish topics according to the gesture recognition results. The wheelchair chassis motion control node subscribes to the topic and controls the movement of the wheelchair body based on the received topic data.

[0045] The third object of the present invention is to provide an electronic device, comprising: a memory on which a program code is stored; a processor connected to the memory and implementing the above method when the program code is executed by the processor.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention provides a wheelchair control method, a wheelchair and a device based on the fusion of sEMG and IMU, the method comprising the following steps: obtaining synchronously collected array electromyographic signals of a human forearm and inertial signals of a hand; preprocessing the inertial signals and the array electromyographic signals; extracting features of the preprocessed array electromyographic signals and inertial signals respectively; fusing the extracted array electromyographic signal features with the inertial signal features as inputs of a gesture recognition model to obtain gesture recognition results; and controlling the movement of the wheelchair through the gesture recognition results. The present invention fuses surface electromyographic signals and IMU signals to perform control gesture recognition for an intelligent wheelchair, combining the stable characteristics of the IMU signals with the anticipatory characteristics of the movement of the surface electromyographic signals, making the gesture recognition results more accurate and able to adapt to the characteristics of the diversity of disability levels and complexity of disability types among the existing wheelchair users.

[0048] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0050] Figure 1 This is a flow chart of the wheelchair control method based on sEMG and IMU fusion in Example 1;

[0051] Figure 2 This is a schematic diagram of the overall framework of the wheelchair control method based on sEMG and IMU fusion in Example 1;

[0052] Figure 3 Schematic diagram of array surface electromyography signal and IMU signal acquisition in Example 1;

[0053] Figure 4 This is a flow chart of array surface electromyography signal and IMU signal preprocessing in Example 1;

[0054] Figure 5 This is a flow chart of array-type myoelectric signal and inertial signal feature extraction in Example 1;

[0055] Figure 6 The muscle synergy feature extraction and sparse coding feature optimization flow chart of Example 1;

[0056] Figure 7 A schematic diagram of a semi-supervised learning network for generating dual-branch pseudo-labels in Example 1;

[0057] Figure 8 The training flow chart of the pseudo-label semi-supervised learning network model of Example 1;

[0058] Fig. 9 This is the control schematic diagram of the smart wheelchair;

[0059] Fig.10 This is the electrical schematic diagram of the smart wheelchair;

[0060] Fig.11 This is the flowchart of the electromyography recognition part;

[0061] Fig.12 This is the program logic block diagram of the wheelchair body;

[0062] Fig.13 is a schematic diagram of an electronic device of Example 3;

[0063] Fig.14 This is a schematic diagram of the storage medium of Example 4. DETAILED DESCRIPTION

[0064] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. It should be noted that, under the premise of no conflict, the embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0065] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0066] The figure numbers in this application are only used to distinguish the various steps in the scheme, and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Example 1

[0068] In order to adapt to the characteristics of the diversity of disability levels and complexity of disability types among the existing wheelchair users, this embodiment provides a wheelchair control method based on the fusion of sEMG and IMU, such as Figure 1 , Figure 2 As shown, the following steps are included:

[0069] S1, obtaining synchronously collected array electromyographic signals (sEMG signals) of the human forearm and inertial signals (IMU signals) of the hand;

[0070] In some embodiments, an array-type surface electromyography sensor is used to collect an array-type electromyogram of a human forearm, and an IMU sensor is used to collect an IMU signal of a hand. The position of the array-type surface electromyography sensor and the IMU sensor is as follows: Figure 3 shown.

[0071] Since the synchronous collection is of array electromyographic signals and inertial signals over a period of time, the patient may make multiple movements during this period, and there is usually a rest time between each movement. In order to perform gesture recognition, it is necessary to obtain data during the user's movements, and the data during the hand rest period is not needed. Therefore, it is necessary to obtain the time period of each complete movement of the user. The beginning and end of this time period are the starting and ending points.

[0072] S2, preprocessing the inertial signal and the array electromyographic signal;

[0073] In some embodiments, Figure 4 As shown, the step of preprocessing the inertial signal and the array electromyographic signal includes:

[0074] S21, filtering the inertial signal and the array electromyographic signal;

[0075] Among them, the wavelet filtering algorithm is used to filter the original array electromyographic signal to remove noise and obtain a cleaner array electromyographic signal;

[0076] The Butterworth filter is used to filter the original IMU signal to remove noise and obtain a cleaner IMU signal.

[0077] S22, normalizing the array electromyographic signal by using the collected maximum voluntary contraction force (MVC) signal;

[0078] For example, using the peak value of the maximum voluntary contraction (MVC) as the normalization standard, the amplitude of the EMG signal at each moment is expressed as a percentage of the MVC.

[0079] S23, using a double threshold method to identify the start and end points of the action of the inertial signal;

[0080] Optionally, based on the dual threshold method, the nonlinear energy operator (NEO) is combined to identify the start and end points of the action. The role of NEO is to amplify the non-stationary signal in the signal and improve the algorithm's processing ability for non-stationary signals.

[0081] S24, determining the action start and end points of the array electromyographic signal based on the action start and end points of the inertial signal.

[0082] Since the sEMG signal and the IMU signal are collected synchronously, the start and end points of the action of the array-type electromyographic signal can be determined based on the start and end points of the action of the inertial signal;

[0083] S25, segmenting the array-type electromyographic signals and inertial signals at the start and end points of the action respectively through a sliding window;

[0084] The signal length processed once is divided into two parts through a sliding window.

[0085] S3, extracting features from the preprocessed array electromyographic signals and inertial signals respectively;

[0086] In some embodiments, Figure 5 As shown, the steps of extracting features from the preprocessed array electromyographic signal and the inertial signal include:

[0087] S31, extracting muscle synergy features of each sliding window for the array electromyographic signal;

[0088] The muscle synergy feature is a specific feature extracted from the array electromyographic signal that can represent the muscle synergy pattern.

[0089] In some embodiments, Figure 6 As shown, the step of extracting muscle synergy features of each sliding window includes:

[0090] S311, for the segmented array electromyographic signal within a given time window , calculate a The Pearson correlation coefficient matrix is ​​used as the muscle synergy matrix :

[0091] ;

[0092] Where n is the number of channels of array electromyographic signals, represents the segmented electromyographic signal of the ith channel at time t, represents the segmented electromyographic signal of the jth channel at time t, , is the correlation between the ith and jth EMG signal channels, and is the mean of channel i and channel j, T is the total number of signal frames in a time window after data segmentation;

[0093] S312, in order to extract the features of the muscle synergy matrix R, firstly, the muscle synergy matrix needs to be expanded to become a one-dimensional vector. The dimension of the expanded vector is (1, );

[0094] S313, then extract the mean, standard deviation, maximum value, minimum value, and two channel positions corresponding to the maximum value of the expanded vector (a 2D vector) to obtain a feature vector as the muscle synergy feature .

[0095] In order to enhance the recognition accuracy of weak electromyographic signals, it is necessary to extract a more separable signal for the problem of weak electromyographic signals when the hand function is insufficient. In some embodiments, the muscle synergy feature is combined with the sparse coding feature optimization technology. Specifically, the step of extracting the muscle synergy feature of each sliding window also includes:

[0096] S314: For the muscle synergy feature, a Lasso (least absolute shrinkage and selection operator) regression model is used to perform sparse coding feature optimization.

[0097] Specifically, the step of optimizing sparse coding features using Lasso regression includes:

[0098] In order to sparsely encode and optimize the muscle synergy features, a Lasso regression model is constructed for each feature dimension. The Lasso optimization problem is:

[0099] ;

[0100] in, Regression model, express and The error between , is the feature vector of the u-th sample, , is the total number of samples, The other feature matrices after removing the target feature from the regression model built for each feature dimension, is the regression coefficient vector, is the regularization coefficient, is the L2 norm, express and The square error between is the L1 norm, express The sum of the absolute values ​​of

[0101] In this embodiment, the extracted muscle synergy feature is m×k dimensional, where m refers to the number of samples and k refers to the dimension of the feature, that is, According to the features of these k dimensions, k Lasso regression models are constructed respectively, and each Lasso regression model processes It refers to the features after removing the features of the i-th (i=1~k) column, and the dimension is .

[0102] Get the regularization coefficient of the input , after Lasso optimization, the converged regression coefficient vector is obtained , After iterative updates, the coefficients of some feature dimensions are reduced to 0, thus playing the role of sparse features, and finally obtaining sparse optimized features .

[0103] By combining muscle synergy features with sparse coding feature optimization technology, the recognition accuracy of weak electromyographic signals is enhanced. Weak electromyographic signals are often interfered by noise. This embodiment can extract a relatively stable synergy pattern by analyzing the synergy features of multiple muscles, thereby enhancing the overall performance of electromyographic signals. Even if the electromyographic signals of some single channels are weak, the synergy of multiple channels can compensate for this deficiency. Compared with traditional electromyographic feature extraction, traditional electromyographic features mainly reflect the amplitude change of electromyographic signals of a single channel. When the electromyographic signals of patients are weak, the changes in electromyographic features are not obvious, making it difficult for the model to distinguish and causing gesture recognition errors. Muscle synergy features mainly reflect the synergy between muscles. Even if the electromyographic signals of some single channels of patients are weak, the synergy of multiple muscles can make the changes in muscle synergy features more obvious, thereby improving the accuracy of gesture recognition. At the same time, due to the stability of the IMU signal, the motion signal can be stably captured, and the recognition accuracy can be further improved after fusion with the sEMG signal.

[0104] S32: For the inertial signal, extract inertial signal features of each sliding window.

[0105] Specifically, the average value of the inertial signal in each sliding window is used as the inertial signal feature of each sliding window.

[0106] S4, fusing the extracted array electromyographic signal features with the inertial signal features as the input of the gesture recognition model to obtain the gesture recognition result;

[0107] Optionally, the gestures for controlling the smart wheelchair can be set to four gestures: palm flexion, dorsiflexion, hand opening, and fist clenching, which correspond to the four driving instructions of the smart wheelchair: turn left, turn right, back, and forward. The smart wheelchair is controlled to make corresponding movements according to the gesture recognition results.

[0108] In combination with the above embodiment, the step of fusing the extracted array electromyographic signal features with the inertial signal features as the input of the gesture recognition model includes:

[0109] The inertial signal features extracted from each time window are combined with the muscle synergy features after sparse coding feature optimization into a feature vector as the input of the gesture recognition model.

[0110] In order to realize the adaptive function of the gesture recognition model, so that the gesture recognition model can be automatically iterated and updated during user use, thereby gradually improving the recognition accuracy, a pseudo-label semi-supervised learning network can be used to realize the adaptive function of the model. Preferably, the gesture recognition model adopts a pseudo-label semi-supervised learning network model.

[0111] In some embodiments, a dual-branch pseudo-label generation semi-supervised learning network (DBPL-Net) is constructed, and the network structure is as follows: Figure 7 As shown, the pseudo labels generated by the teacher network are used to guide the student network learning, thereby realizing the pseudo-label semi-supervised learning network model using unlabeled data for model training.

[0112] Among them, the pseudo-label semi-supervised learning network consists of two parts: the teacher network and the student network. Both the teacher network and the student network architecture adopt the LSTM model with the same structure.

[0113] Loss of labeled data :

[0114] ;

[0115] in, is the true label, Input for the model predictions, , is the total number of samples with labeled data.

[0116] Unlabeled data loss :

[0117] ;

[0118] in, For unlabeled data, For Sample data after data enhancement, , is the total number of samples of unlabeled data, and They represent the parameters of the teacher network and the student network respectively.

[0119] Total loss:

[0120] ;

[0121] A hyperparameter that balances the loss between labeled and unlabeled.

[0122] Teacher Network Prediction: Unlabeled Data As the input of the teacher network model, the output prediction result is used as the pseudo label of this sample .

[0123] Filter high confidence samples: Only retain samples whose predicted probability exceeds a threshold τ (for example, τ can be set to 0.9).

[0124] Student network learning: The screened samples and their pseudo labels are used to train the student network.

[0125] Teacher network parameter update: Update using exponential moving average (EMA) method

[0126] ;

[0127] in, are the current parameters of the teacher network, are the current parameters of the student network, is a hyperparameter between 0 and 1 that controls the smoothness of the updates.

[0128] Student network parameter update: back-propagation update using the total loss L.

[0129] Finally, the probability distribution of gesture categories is predicted by the student network, and the category with the highest probability is output as the final gesture recognition result.

[0130] In some embodiments, Figure 8 As shown, the training of the pseudo-label semi-supervised learning network model includes an initialization training process and a training process using unlabeled data; wherein,

[0131] The initialization training process includes:

[0132] S41. Through labeled data Initialize the student model and the teacher model until the model converges to obtain a pre-trained student model With the teacher model ;

[0133] The use of unlabeled data The training process includes:

[0134] S42: The feature vector formed by combining the inertial signal features extracted from each time window with the muscle synergy features after sparse coding feature optimization is input as unlabeled data into the pre-trained teacher model. The recognition result is output ;

[0135] S43, screening the recognition results, selecting the recognition result confidence greater than the threshold τ (for example, setting τ=0.9) as the pseudo label, and obtaining the pseudo label data ;

[0136] S44, using the pseudo-label data With the labeled data For the pre-trained student model For training, use the total loss Model for students Perform back propagation updates and use the exponential moving average (EMA) method to update the teacher model Make updates;

[0137] S45, repeat the training process using unlabeled data (i.e., repeat steps S42 to S44) until the student model Converge, and finally get a trained student model .

[0138] Optionally, before the user uses the smart wheelchair for the first time, the smart wheelchair uses pre-provided labeled data Train to obtain an initial network model, which is the pre-trained student model mentioned above With the teacher model .

[0139] Each time the user uses the smart wheelchair, the smart wheelchair will automatically collect and record sEMG data and IMU data synchronously. When the user stops using the smart wheelchair, the smart wheelchair will automatically preprocess the collected and recorded unlabeled sEMG data and IMU data (that is, the preprocessing process described in the above embodiment) to obtain the preprocessed unlabeled data. , using labeled data With unlabeled data The pseudo-label semi-supervised learning network model is trained and updated to obtain an updated pseudo-label semi-supervised learning network model.

[0140] Since the degree of disability of each patient is different, the muscle coordination pattern of each person's hand is different. For this, the use of pseudo-label driven semi-supervised learning network can enable the gesture recognition algorithm to synchronously collect the patient's unlabeled data during the patient's daily use, and automatically use these unlabeled data for offline model training, thereby improving the generalization ability of the model, so that the model gradually automatically updates and optimizes iteratively during use, continuously improves the recognition accuracy, and finally realizes the adaptive function of the gesture recognition algorithm; compared with the fully supervised learning network, if the fully supervised learning network model is to be updated and optimized, manual data collection and labeling processing are required to finally obtain a labeled training set. On the one hand, the update work requires a lot of manpower; on the other hand, the manual processing of data can only be achieved by technical personnel, and patients cannot update the model independently; on the other hand, the model finally obtained by updating this method cannot achieve model adaptation because the data used for training comes from multiple patients.

[0141] S5. Controlling the movement of the wheelchair according to the gesture recognition result.

[0142] In some embodiments, the step of controlling the movement of the wheelchair using the gesture recognition result includes:

[0143] According to the gesture recognition results, the values ​​of various parameters in the topic published by the gesture recognition node are set to control the wheelchair to perform corresponding actions.

[0144] Since the user gesture recognition process is a continuous recognition, when a preset gesture is recognized, the smart wheelchair is controlled to perform a corresponding action. When any of the preset gestures is not recognized, the smart wheelchair does not take any action, that is, stops.

[0145] like Fig. 9 , Fig.11 As shown, the intelligent wheelchair includes a gesture recognition node, a wheelchair chassis motion control node, and a wheelchair body. The gesture recognition node is used to perform gesture recognition based on the synchronously collected array electromyographic signals of the human forearm and the inertial signals of the hand and publish topics according to the gesture recognition results. The wheelchair chassis motion control node subscribes to the topic and controls the movement of the wheelchair body based on the received topic data.

[0146] The gesture recognition node implements gesture recognition using the method described in the above embodiment. Optionally, the gesture recognition node and the wheelchair chassis motion control node can use a publisher-subscriber communication method based on ROS2, the gesture recognition node publishes the / cmd_vel topic, and the wheelchair chassis motion control node subscribes to the / cmd_vel topic.

[0147] Optionally, five different results can be obtained through the above gesture recognition model, namely, turn left, turn right, forward, backward, and stop. By setting the values ​​of the parameters in the corresponding / cmd_vel topic, the wheelchair can be controlled to make different actions, as shown in Table 1. The / cmd_vel topic is a predefined geometry_msgs / msg / Twist type data format in the ROS system, and its data content includes six values ​​of linear velocity and angular velocity along the XYZ direction. Since the smart wheelchair usually moves in a plane, and according to the analysis of the actual movement trajectory of the wheelchair, in this data format, only the linear velocity along the y-axis and the angular velocity around the z-axis are not 0, and the rest are all 0, that is, Linear.y=Linear.z=Angular.x= Angular.y=0. The wheelchair chassis motion control node parses the data received in the / cmd_vel topic to drive the smart wheelchair to move arbitrarily in the plane.

[0148] Table 1 Specific values ​​of various parameters in the / cmd_vel topic corresponding to the gesture recognition results

[0149]

[0150] like Fig.10 , Fig.12 As shown in the figure, the wheelchair chassis motion control node can be set up with a host computer and a slave computer. The host computer Jetson Xavier Nx host runs the ROS2 system, subscribes to the / cmd_vel topic, and transmits the topic data to the slave computer STM32 through SPI communication. The slave computer performs inverse kinematics analysis on the differential drive wheel to obtain the rotation speed of each wheel, and then sends speed control instructions to the driver through the CAN bus to control the movement of the hub motor.

[0151] This embodiment provides a wheelchair control method based on the fusion of sEMG and IMU, combines muscle synergy features with sparse coding feature optimization technology and fuses IMU signals to solve the problem of low gesture recognition accuracy caused by the inability of existing gesture recognition methods to adapt to the diversity of disability levels and complexity of disability types among existing wheelchair users (i.e., patients' insufficient hand motor function), and uses a pseudo-label semi-supervised learning network to implement the adaptive function of the model, so that the model is automatically iterated and updated during the patient's use, thereby gradually improving the recognition accuracy and realizing the adaptive function of the algorithm. Example 2

[0152] An intelligent wheelchair uses the above-mentioned wheelchair control method based on sEMG and IMU fusion. For a detailed description of the method, reference may be made to the corresponding description in the above-mentioned method embodiment, which will not be repeated here.

[0153] This embodiment provides a smart wheelchair, such as Fig. 9 As shown, it includes a gesture recognition node, a wheelchair chassis motion control node, and a wheelchair body. The gesture recognition node is used to perform gesture recognition based on the synchronously collected array electromyographic signals of the human forearm and the inertial signals of the hand and publish topics according to the gesture recognition results. The wheelchair chassis motion control node subscribes to the topic and controls the movement of the wheelchair body based on the received topic data.

[0154] The gesture recognition node implements gesture recognition using the method described in the above method embodiment. Please refer to the corresponding description in the above method embodiment, which will not be repeated here. Fig. 9 , Fig.11 As shown, the gesture recognition node and the wheelchair chassis motion control node can adopt a publisher-subscriber communication method based on ROS2, the gesture recognition node publishes the / cmd_vel topic, and the wheelchair chassis motion control node subscribes to the / cmd_vel topic.

[0155] Optionally, the gesture recognition model of the above method embodiment can obtain five different results, namely, turn left, turn right, forward, backward, and stop. By setting the values ​​of the parameters in the corresponding / cmd_vel topic, the wheelchair can be controlled to make different actions, as shown in Table 1. The / cmd_vel topic is a predefined geometry_msgs / msg / Twist type data format in the ROS system, and its data content includes six values ​​of linear velocity and angular velocity along the XYZ direction. Since smart wheelchairs usually move in a plane, and according to the analysis of the actual movement trajectory of the wheelchair, in this data format, only the linear velocity along the y-axis and the angular velocity around the z-axis are not 0, and the rest are all 0, that is, Linear.y=Linear.z=

[0156] Angular.x = Angular.y = 0. The wheelchair chassis motion control node parses the data received in the / cmd_vel topic to drive the smart wheelchair to move arbitrarily within the plane.

[0157] like Fig.10 , Fig.12 As shown in the figure, the wheelchair chassis motion control node can be set up with a host computer and a slave computer. The host computer Jetson Xavier Nx host runs the ROS2 system, subscribes to the / cmd_vel topic, and transmits the topic data to the slave computer STM32 through SPI communication. The slave computer performs inverse kinematics analysis on the differential drive wheel to obtain the rotation speed of each wheel, and then sends speed control instructions to the driver through the CAN bus to control the movement of the hub motor.

[0158] The smart wheelchair of this embodiment may include the following process:

[0159] A signal acquisition module, used to acquire synchronously collected array electromyographic signals of the human forearm and inertial signals of the hand;

[0160] A signal preprocessing module, used for preprocessing the inertial signal and the array electromyographic signal;

[0161] A feature extraction module is used to extract features from the preprocessed array electromyographic signal and inertial signal respectively;

[0162] A gesture recognition module is used to fuse the extracted array electromyographic signal features with the inertial signal features as the input of the gesture recognition model to obtain the gesture recognition result;

[0163] The wheelchair motion control module is used to control the wheelchair motion according to the gesture recognition result.

[0164] Based on the technical solution of the above embodiment, optionally, the step of preprocessing the inertial signal and the array electromyographic signal includes:

[0165] Performing filtering processing on the inertial signal and the array electromyographic signal;

[0166] Normalizing the array electromyographic signal by using the collected maximum voluntary contraction force signal;

[0167] A double threshold method is used to identify the start and end points of the inertial signal;

[0168] Determining the action start and end points of the array-type electromyographic signal based on the action start and end points of the inertial signal;

[0169] The array-type electromyographic signals and inertial signals within the start and end points of the action are segmented and processed respectively through a sliding window.

[0170] Based on the technical solution of the above embodiment, optionally, the step of extracting features from the preprocessed array electromyographic signal and the inertial signal respectively includes:

[0171] For the array electromyographic signal, extract the muscle synergy feature of each sliding window;

[0172] For the inertial signal, an inertial signal feature of each sliding window is extracted.

[0173] Based on the technical solution of the above embodiment, optionally, the step of extracting the muscle synergy feature of each sliding window includes:

[0174] For the segmented array electromyographic signal within a given time window , calculate a The Pearson correlation coefficient matrix is ​​used as the muscle synergy matrix :

[0175] ;

[0176] Where n is the number of channels of array electromyographic signals, represents the segmented electromyographic signal of the ith channel at time t, represents the segmented electromyographic signal of the jth channel at time t, , is the synergy strength between the ith and jth EMG signal channels in the time window, and is the mean of channel i and channel j, T is the total number of signal frames in a time window after data segmentation;

[0177] The muscle synergy matrix is ​​expanded to become a one-dimensional vector. The dimension of the expanded vector is (1, );

[0178] Extract the mean, standard deviation, maximum value, minimum value, and two channel positions corresponding to the maximum value of the expanded vector to obtain a feature vector as the muscle synergy feature .

[0179] Based on the technical solution of the above embodiment, optionally, the step of extracting the muscle synergy feature of each sliding window further includes:

[0180] For the muscle synergy feature, the Lasso regression model is used to perform sparse coding feature optimization.

[0181] Based on the technical solution of the above embodiment, optionally, the step of optimizing sparse coding features by using Lasso regression includes:

[0182] A Lasso regression model is constructed for each feature dimension, and the Lasso optimization problem is:

[0183] ;

[0184] in, is the regression model, express and The error between , is the feature vector of the u-th sample, , is the total number of samples, The other feature matrices after removing the target feature from the regression model built for each feature dimension, is the regression coefficient vector, is the regularization coefficient, is the L2 norm, express and The square error between is the L1 norm, express The sum of the absolute values ​​of

[0185] Get the regularization coefficient of the input , after Lasso optimization, the converged regression coefficient vector is obtained , After iterative updates, the coefficients of some feature dimensions are reduced to 0, and finally the sparsely optimized features are obtained. .

[0186] Based on the technical solution of the above embodiment, optionally, the step of fusing the extracted array electromyographic signal features with the inertial signal features as the input of the gesture recognition model includes:

[0187] The inertial signal features extracted from each time window are combined with the muscle synergy features after sparse coding feature optimization into a feature vector as the input of the gesture recognition model.

[0188] Based on the technical solution of the above embodiment, optionally, the gesture recognition model adopts a pseudo-label semi-supervised learning network model.

[0189] Based on the technical solution of the above embodiment, optionally, the training of the pseudo-label semi-supervised learning network model includes an initialization training process and a training process using unlabeled data; wherein,

[0190] The initialization training process includes:

[0191] Initialize and train the student model and teacher model using labeled data until the model converges to obtain the pre-trained student model and teacher model.

[0192] The training process using unlabeled data includes:

[0193] The feature vector formed by combining the inertial signal features extracted from each time window with the muscle synergy features after sparse coding feature optimization is input into the pre-trained teacher model as unlabeled data, and the recognition result is output;

[0194] The recognition results are screened, and the results with a confidence level greater than a threshold are selected as pseudo labels to obtain pseudo label data;

[0195] Using the pseudo-labeled data and the labeled data to train the pre-trained student model, using the total loss to back-propagate and update the student model, and using the exponential moving average method to update the teacher model;

[0196] The training process using unlabeled data is repeatedly performed until the student model converges, and finally a trained student model is obtained.

[0197] Based on the technical solution of the above embodiment, optionally, the step of controlling the movement of the wheelchair by using the gesture recognition result includes:

[0198] According to the gesture recognition results, the values ​​of various parameters in the topic published by the gesture recognition node are set to control the wheelchair to perform corresponding actions.

[0199] This embodiment provides an intelligent wheelchair, which adopts the above-mentioned wheelchair control method based on sEMG and IMU fusion. This embodiment fuses surface electromyography signals and IMU signals to perform control gesture recognition of the intelligent wheelchair, combining the stable characteristics of IMU signals with the anticipatory characteristics of surface electromyography signals, so that the gesture recognition results are more accurate and can adapt to the characteristics of the diversity of disability levels and complexity of disability types among the existing wheelchair users. Example 3

[0200] An electronic device, such as Fig.13 As shown, it includes: a memory on which program codes are stored; a processor connected to the memory, and when the program codes are executed by the processor, a wheelchair control method based on sEMG and IMU fusion is implemented. For a detailed description of the method, reference can be made to the corresponding description in the above method embodiment, which will not be repeated here. Example 4

[0201] A computer readable storage medium such as Fig.14 As shown, program instructions are stored thereon, and when the program instructions are executed by the processor, a wheelchair control method based on sEMG and IMU fusion is implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, and no further description is given here.

[0202] The number of devices and processing scales described here are used to simplify the description of the present invention. Applications, modifications and variations of the present invention will be obvious to those skilled in the art.

[0203] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and implementation modes. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

[0204] The apparatus, electronic device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, electronic device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device and non-volatile computer storage medium will not be repeated here.

[0205] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software units for implementing the method and structures within the hardware component.

[0206] The systems, devices or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions in various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or more software and / or hardware.

[0207] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0208] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0209] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0210] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0211] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0212] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program units. Generally, program units include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program units may be located in local and remote computer storage media, including storage devices.

[0213] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0214] The above description is only an embodiment of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of one or more embodiments of this specification.

Claims

1. A wheelchair control method based on sEMG and IMU fusion, characterized in that: The following steps are involved: Acquire synchronously collected array electromyographic signals of the human forearm and inertial signals of the hand; Preprocessing the inertial signal and the array electromyographic signal; Perform feature extraction on the preprocessed array electromyographic signals and inertial signals respectively; The extracted array electromyographic signal features and inertial signal features are fused as the input of the gesture recognition model to obtain the gesture recognition results; Controlling the movement of the wheelchair through the gesture recognition result; The steps of respectively extracting features from the pre-processed array electromyographic signal and the inertial signal include: For the array electromyographic signal, extract the muscle synergy feature of each sliding window; For the inertial signal, extracting inertial signal features of each sliding window; The step of extracting muscle synergy features of each sliding window comprises: For the segmented array electromyographic signal within a given time window , calculate a The Pearson correlation coefficient matrix is ​​used as the muscle synergy matrix : ； Where n is the number of channels of array electromyographic signals, represents the segmented electromyographic signal of the ith channel at time t, represents the segmented electromyographic signal of the jth channel at time t, , , , is the synergy strength between the ith and jth EMG signal channels in the time window, and is the mean of channel i and channel j, T is the total number of signal frames in a time window after data segmentation; The muscle synergy matrix is ​​expanded to become a one-dimensional vector. The dimension of the expanded vector is (1, ); Extract the mean, standard deviation, maximum value, minimum value, and two channel positions corresponding to the maximum value of the expanded vector to obtain a feature vector as the muscle synergy feature .

2. A wheelchair control method based on sEMG and IMU fusion as claimed in claim 1, characterized in that: The step of preprocessing the inertial signal and the array electromyographic signal comprises: Performing filtering processing on the inertial signal and the array electromyographic signal; Normalizing the array electromyographic signal by using the collected maximum voluntary contraction force signal; A double threshold method is used to identify the start and end points of the inertial signal; Determining the action start and end points of the array-type electromyographic signal based on the action start and end points of the inertial signal; The array-type electromyographic signals and inertial signals within the start and end points of the action are segmented and processed respectively through a sliding window.

3. A wheelchair control method based on sEMG and IMU fusion as claimed in claim 1, characterized in that: The step of extracting muscle synergy features of each sliding window also includes: For the muscle synergy feature, the Lasso regression model is used to perform sparse coding feature optimization.

4. A wheelchair control method based on sEMG and IMU fusion as claimed in claim 3, characterized in that: The step of fusing the extracted array electromyographic signal features with the inertial signal features as the input of the gesture recognition model includes: The inertial signal features extracted from each time window are combined with the muscle synergy features after sparse coding feature optimization into a feature vector as the input of the gesture recognition model.

5. A wheelchair control method based on sEMG and IMU fusion as claimed in claim 4, characterized in that: The gesture recognition model adopts a pseudo-label semi-supervised learning network model.

6. A wheelchair control method based on sEMG and IMU fusion as claimed in claim 5, characterized in that: The training of the pseudo-label semi-supervised learning network model includes an initialization training process and a training process using unlabeled data; wherein, The initialization training process includes: Initialize and train the student model and teacher model using labeled data until the model converges to obtain the pre-trained student model and teacher model. The training process using unlabeled data includes: The feature vector formed by combining the inertial signal features extracted from each time window with the muscle synergy features after sparse coding feature optimization is input into the pre-trained teacher model as unlabeled data, and the recognition result is output; The recognition results are screened, and the results with a confidence level greater than a threshold are selected as pseudo labels to obtain pseudo label data; Using the pseudo-labeled data and the labeled data to train the pre-trained student model, using the total loss to back-propagate and update the student model, and using the exponential moving average method to update the teacher model; The training process using unlabeled data is repeatedly performed until the student model converges, and finally a trained student model is obtained.

7. A wheelchair control method based on sEMG and IMU fusion as claimed in claim 1, characterized in that: The step of controlling the wheelchair movement by using the gesture recognition result comprises: According to the gesture recognition results, the values ​​of various parameters in the topic published by the gesture recognition node are set to control the wheelchair to perform corresponding actions.

8. An intelligent wheelchair, using the method according to any one of claims 1 to 7, characterized in that: It includes a gesture recognition node, a wheelchair chassis motion control node, and a wheelchair body. The gesture recognition node is used to perform gesture recognition based on the synchronously collected array electromyographic signals of the human forearm and the inertial signals of the hand and publish topics according to the gesture recognition results. The wheelchair chassis motion control node subscribes to the topic and controls the movement of the wheelchair body based on the received topic data.

9. An electronic device, characterized in that: include: A memory having program code stored therein; A processor connected to the memory, and when the program code is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images