A method and system for predicting robotic arm rehabilitation data based on deep learning
Through deep learning technology, the spatiotemporal characteristics in rehabilitation data are extracted and combined with long and short-term memory neural networks and attention mechanisms are solved, and the problem that traditional rehabilitation robots cannot automatically adjust training parameters is realized, achieving targeted and effective robotic arms in upper limb rehabilitation to adapt to the individual differences in patients’ rehabilitation needs.
Patent Information
- Application Number
- CN202411074409.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-07
AI Technical Summary
Traditional rehabilitation robots cannot automatically adjust training parameters, resulting in insufficient targeted and effective rehabilitation training and cannot fully adapt to patients' individual differences and rehabilitation needs.
Using deep learning-based robotic arm rehabilitation data prediction method, the patient's rehabilitation data is collected and processed, and the spatiotemporal features are extracted using residual convolution modules of different sizes of convolution kernels, and combined with long and short-term memory neural networks and attention mechanisms to predict. , automatically adjust training parameters to achieve targeted rehabilitation.
Automatic parameter adjustment of the robotic arm during the rehabilitation process of upper limbs is realized, the targeted and effective rehabilitation training is improved, and it can better adapt to patients' individual differences and rehabilitation needs.
Smart Images

Figure CN118588289B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robotic arm application technology, and in particular to a robotic arm rehabilitation data prediction method and system based on deep learning. Background Art
[0002] Upper limb dysfunction has become one of the common health problems in society. Therefore, it is of great clinical significance to develop an effective upper limb rehabilitation robot system to assist patients in rehabilitation training and improve their quality of life. Traditional rehabilitation training mainly relies on the guidance of manual rehabilitation therapists and the patient's self-training. There are problems such as difficulty in quantifying the training effect and low individualization. Traditional rehabilitation robots often complete training tasks through preset programs and force control, which cannot fully adapt to the individual differences and rehabilitation needs of patients.
[0003] In terms of technology, the continuous advancement of sensor technology enables rehabilitation robots to accurately sense the patient's movement status and force output, enabling detailed assessment and monitoring of the patient's motor ability.
[0004] In the prior art, during the process of upper limb rehabilitation for patients, it is impossible to automatically adjust the training parameters, so that it is impossible to provide targeted training and rehabilitation for patients, which makes the training less effective. Summary of the invention
[0005] Based on this, the purpose of the present invention is to provide a deep learning-based robotic arm rehabilitation data prediction method and system to at least solve the above-mentioned deficiencies in the prior art.
[0006] In a first aspect, the present invention provides a method for predicting robotic arm rehabilitation data based on deep learning, the method comprising:
[0007] Collecting a number of rehabilitation data of the patient each time using the robotic arm, processing the plurality of rehabilitation data to obtain a rehabilitation data set, and performing standardization processing on the rehabilitation data set to obtain a standardized rehabilitation data set;
[0008] Extracting the spatiotemporal features in the standardized rehabilitation data set based on a plurality of residual convolution modules with convolution kernels of different sizes to obtain a plurality of spatiotemporal features, and fusing the plurality of the spatiotemporal features to obtain multi-scale features;
[0009] Predicting the multi-scale features based on a long short-term memory neural network to capture the dependency of the multi-scale features on time;
[0010] The multi-scale features and the dependency of the multi-scale features with time are fused based on the attention mechanism to obtain the final output data of the robotic arm.
[0011] Compared with the prior art, the beneficial effects of the present invention are: a number of spatiotemporal features are extracted and fused through several residual convolution modules with convolution kernels of different sizes, and then predicted based on the long short-term memory neural network, as well as through the attention mechanism of the long short-term memory neural network, so as to achieve automatic adjustment of the training parameters of the robotic arm, and to obtain the robotic arm data of the patient when using the robotic arm for rehabilitation in a targeted manner, thereby making the robotic arm more effective in the process of upper limb rehabilitation of patients.
[0012] Furthermore, some of the rehabilitation data include simple information of the patient, game information using the robotic arm, the patient's arm extension and rotation angle, arm movement speed, recording time, grip strength, and pushing force.
[0013] Furthermore, the step of processing a plurality of the rehabilitation data to obtain a rehabilitation data set, and performing standardization processing on the rehabilitation data set to obtain a standardized rehabilitation data set comprises:
[0014] performing missing value processing, outlier processing, duplicate value processing, and data type conversion on the plurality of rehabilitation data in sequence to obtain a rehabilitation data set, and saving the rehabilitation data set;
[0015] Calculating a mean value and a standard deviation of each feature in the rehabilitation data set, and performing standardization processing on the rehabilitation data set based on the mean value and the standard deviation;
[0016] The standardized rehabilitation data set is divided into a training set, a validation set, and a test set.
[0017] Furthermore, the expression of the standardization process is:
[0018] ;
[0019] In the formula, represents the rehabilitation data set after normalization, represents the original rehabilitation dataset, represents the mean, represents the standard deviation.
[0020] Furthermore, the steps of extracting the spatiotemporal features in the standardized rehabilitation data set by using a plurality of residual convolution modules based on convolution kernels of different sizes to obtain a plurality of spatiotemporal features, and fusing the plurality of spatiotemporal features to obtain multi-scale features include:
[0021] Inputting the standardized rehabilitation data set into a plurality of residual convolution modules with convolution kernels of different sizes, wherein each of the residual convolution modules is composed of a five-layer convolutional neural network combined with a residual network;
[0022] Capturing local features in the standardized rehabilitation dataset based on the five-layer convolutional neural network, and performing feature enhancement by the residual network;
[0023] The local features are fused based on several residual convolution modules with convolution kernels of different sizes to obtain multi-scale features.
[0024] Furthermore, the extraction expression of the residual convolution module is:
[0025] ;
[0026] In the formula, represents the spatiotemporal features extracted by the residual convolution module with a convolution kernel of 3, , represents two different learnable parameters, Represents a multi-layer convolutional network with a convolution kernel of 3. represents the activation function, represents the rehabilitation data set after standardization;
[0027] The expression of multi-scale features obtained by feature fusion is:
[0028] ;
[0029] represents the multi-scale features, , Respectively represent the spatiotemporal features extracted by the other two residual convolution modules, , , They represent the learnable parameters of the corresponding scales respectively.
[0030] Furthermore, the multi-scale features and the dependency of the multi-scale features with time are fused based on the attention mechanism to obtain the final output data of the robotic arm:
[0031] ;
[0032] ;
[0033] In the formula, represents the weight of the t-th attention score, represents the learnable parameters of the attention mechanism, represents the bias of the attention mechanism, represents the number of time series, represents the tth vector in the value vector, represents the result of weighted summation, and represents the final output of the value vector weighted according to the attention weight.
[0034] In a second aspect, the present invention further provides a robot arm rehabilitation data prediction system based on deep learning, the system comprising:
[0035] A collection and processing module, used for collecting a number of rehabilitation data of the patient each time using the robotic arm, processing the plurality of rehabilitation data to obtain a rehabilitation data set, and performing standardization processing on the rehabilitation data set to obtain a standardized rehabilitation data set;
[0036] An extraction and fusion module, used for extracting the spatiotemporal features in the standardized rehabilitation data set based on a plurality of residual convolution modules with convolution kernels of different sizes to obtain a plurality of spatiotemporal features, and fusing the plurality of the spatiotemporal features to obtain multi-scale features;
[0037] A prediction and capture module, used to predict the multi-scale features based on a long short-term memory neural network to capture the dependency of the multi-scale features on time;
[0038] A fusion output module is used to fuse the multi-scale features and the dependency of the multi-scale features with time based on an attention mechanism to obtain final output data of the robotic arm.
[0039] Furthermore, the collection and processing module includes:
[0040] a processing and saving unit, used for sequentially performing missing value processing, abnormal value processing, duplicate value processing and data type conversion on a plurality of the rehabilitation data to obtain a rehabilitation data set, and saving the rehabilitation data set;
[0041] a calculation processing unit, configured to calculate a mean value and a standard deviation of each feature in the rehabilitation data set, and perform standardization processing on the rehabilitation data set based on the mean value and the standard deviation;
[0042] The division unit is used to divide the rehabilitation data set after the standardization process to obtain a training set, a validation set and a test set.
[0043] Furthermore, the extraction and fusion module includes:
[0044] An input unit, used to input the standardized rehabilitation data set into a plurality of residual convolution modules with convolution kernels of different sizes, wherein each of the residual convolution modules is composed of a five-layer convolutional neural network combined with a residual network;
[0045] A capture unit, used for capturing local features in the standardized rehabilitation data set based on the five-layer convolutional neural network, and converging by the residual network;
[0046] A fusion unit is used to fuse the local features based on a plurality of residual convolution modules with convolution kernels of different sizes to obtain multi-scale features. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 is a flowchart of a method for predicting robotic arm rehabilitation data based on deep learning in a first embodiment of the present invention;
[0048] Figure 2 Schematic diagram of the structure of a model composed of a residual convolution module, a long short-term memory neural network module and an attention mechanism module in the first embodiment of the present invention;
[0049] Figure 3 This is a structural block diagram of a deep learning-based robotic arm rehabilitation data prediction system in the second embodiment of the present invention.
[0050] Description of main component symbols:
[0051] 10. Collection and processing module; 20. Extraction and fusion module; 30. Prediction and capture module; 40. Fusion and output module.
[0052] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION
[0053] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.
[0054] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.
[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.
[0056] Embodiment 1
[0057] See also Figure 1 , which shows a method for predicting robot arm rehabilitation data based on deep learning in a first embodiment of the present invention, the method comprises steps S1 to S4:
[0058] S1, collecting a number of rehabilitation data of the patient each time using the robotic arm, processing the plurality of rehabilitation data to obtain a rehabilitation data set, and standardizing the rehabilitation data set to obtain a standardized rehabilitation data set;
[0059] It should be explained that each time a patient uses the robotic arm rehabilitation device, the record is saved in the database system to collect the patient's rehabilitation data. In this embodiment, the rehabilitation data includes simple information about the patient, game information using the robotic arm, the patient's arm extension rotation angle, arm movement speed, recording time, gripping force and pushing force, and also includes device path information. For specific rehabilitation data information, please refer to Table 1:
[0060] Table 1
[0061]
[0062] Specifically, the step S1 includes steps S11 to S13:
[0063] S11, sequentially performing missing value processing, outlier processing, duplicate value processing, and data type conversion on the plurality of rehabilitation data to obtain a rehabilitation data set, and saving the rehabilitation data set;
[0064] It is understandable that the collected rehabilitation data needs to be cleaned. A large amount of data will have errors in the process of recording, storing and transmitting, so the rehabilitation data needs to be cleaned. In this embodiment, the cleaning includes missing value processing, abnormal value processing, duplicate value processing, data type conversion and data preservation in sequence;
[0065] Missing value processing: Check whether there are anomalies in the rehabilitation data, delete the records with missing time data or user ID, and fill the records with zero values for the missing other attributes;
[0066] Outlier processing: Use visualization tools such as box plots and scatter plots to identify outliers in the data, delete outliers in time data, correct outliers at various angles of the arm (replace values outside the specified range with boundary values), replace outliers in other attributes, and use the mean to replace outliers;
[0067] Duplicate value processing: Check whether there are completely duplicate samples in the rehabilitation data, and delete the duplicate and redundant samples;
[0068] Data type conversion: Ensure that the data types in the rehabilitation data are consistent with the data types required by the model, including numeric and date types. For example, the start time and end time types are date types, and the angles of various arms are numeric types.
[0069] Data preservation: Save the cleaned data in a standard format. In this embodiment, the saved data format is CSV for subsequent model training and analysis;
[0070] S12, calculating the mean and standard deviation of each feature in the rehabilitation data set, and standardizing the rehabilitation data set based on the mean and the standard deviation;
[0071] It is understandable that when processing continuous rehabilitation data, the dimensional difference problem between different features is usually encountered. The dimensional difference refers to the large difference in the numerical ranges of different features, which may affect the subsequent model training and performance. Therefore, in this embodiment, it is necessary to standardize the rehabilitation data set and adjust the numerical ranges of different features to a similar scale so that the model can better learn the relationship between the features.
[0072] It should be explained that the mean and standard deviation of each feature in the rehabilitation data set are first calculated, and then standardized. The expression of the standardized processing is:
[0073] ;
[0074] In the formula, represents the rehabilitation data set after normalization, represents the original rehabilitation dataset, represents the mean, represents the standard deviation;
[0075] S13, dividing the standardized rehabilitation data set to obtain a training set, a validation set, and a test set;
[0076] It should be explained that the standardized rehabilitation data set is divided into a training set, a validation set and a test set. In the present embodiment, the training set occupies 70%, and the validation set and the test set each occupy 15%.
[0077] S2, extracting the spatiotemporal features in the standardized rehabilitation data set based on a plurality of residual convolution modules with convolution kernels of different sizes to obtain a plurality of spatiotemporal features, and fusing the plurality of spatiotemporal features to obtain multi-scale features;
[0078] It can be understood that the model in this embodiment includes three groups of residual convolution modules, long short-term memory neural network modules and attention mechanism modules. The specific structure diagram of the model is as follows: Figure 2As shown;
[0079] It should be explained that the extraction expression of the residual convolution module is:
[0080] ;
[0081] In the formula, represents the spatiotemporal features extracted by one of the residual convolution modules, , represents two different learnable parameters, Represents a multi-layer convolutional network with a convolution kernel of 3. represents the activation function, represents the rehabilitation data set after standardization;
[0082] Specifically, the step S2 includes steps S21 to S23:
[0083] S21, inputting the standardized rehabilitation data set into a plurality of residual convolution modules with convolution kernels of different sizes, wherein each of the residual convolution modules is composed of a five-layer convolutional neural network combined with a residual network;
[0084] S22, capturing local features in the standardized rehabilitation dataset based on the five-layer convolutional neural network, and performing feature enhancement by the residual network;
[0085] S23, fusing the local features based on a plurality of residual convolution modules with convolution kernels of different sizes to obtain multi-scale features;
[0086] The expression of multi-scale features obtained by feature fusion is:
[0087] ;
[0088] represents the multi-scale features, , Respectively represent the spatiotemporal features extracted by the other two residual convolution modules, , , They represent the learnable parameters of the corresponding scales respectively.
[0089] It is understandable that in order to extract more comprehensive spatiotemporal features from the complex standardized rehabilitation data set, the standardized rehabilitation data set is input into three residual convolution modules with different sizes of convolution kernels. Each residual convolution module consists of a five-layer convolutional neural network combined with a weighted residual network. The convolutional neural network can effectively capture the local features of the input data, and the weighted residual structure can more effectively learn complex feature representations, enhance feature learning, promote network convergence, and reduce the risk of overfitting. Convolutions with different convolution kernels can also process data of different granularities at the same time when processing data, making feature extraction more complete and improving the accuracy of subsequent planning.
[0090] S3, predicting the multi-scale features based on a long short-term memory neural network to capture the dependency between the multi-scale features and time;
[0091] It can be understood that the dependency of multi-scale features and time is captured by the long short-term memory neural network. The long short-term memory neural network is a variant of the recurrent neural network (RNN) commonly used to process sequence data. Compared with the traditional RNN, the long short-term memory neural network introduces a structure called a gate unit to more effectively capture and remember the long-term dependencies in the sequence. The core of the long short-term memory neural network is its internal memory unit, which consists of three gate units: forget gate, input gate and output gate. These gates control the flow of information, allowing the long short-term memory neural network to selectively remember or ignore different parts of the input data. Among them, the forget gate is specifically defined as:
[0092] ;
[0093] The specific definition of the input gate is:
[0094] ;
[0095] ;
[0096] Detailed definition of the output gate:
[0097] ;
[0098] ;
[0099] in, , , They represent the activation value of the input gate, the activation value of the forget gate, and the activation value of the output gate, respectively. represents the sigmoid function, which is used to compress values between 0 and 1, and tanh is the hyperbolic tangent function, which is used to compress values between -1 and 1. , , , Respectively represent the input gate weight, forget gate weight, cell state weight, and output gate weight, , , , They represent the bias of the input gate, the bias of the forget gate, the bias of the cell state, and the bias of the output gate, respectively. Represents the hidden state at the previous moment, Represents the input data at the current moment, represents the hidden state at time step t, represents the current candidate cell state, represents the cell state at time step t.
[0100] S4, predicting the multi-scale features and the dependency between the multi-scale features and time based on the attention mechanism to obtain final output data of the robotic arm;
[0101] It should be noted that the multi-scale features output by the long short-term memory network and the dependency between the multi-scale features and time are input into the attention module;
[0102] The expression for fusing the multi-scale features and the dependency of the multi-scale features with time based on the attention mechanism to obtain the final output data of the robotic arm is:
[0103] ;
[0104] ;
[0105] In the formula, represents the weight of the t-th attention score, represents the learnable parameters of the attention mechanism, represents the bias of the attention mechanism, represents the number of time series, represents the tth vector in the value vector, represents the result of weighted summation, and represents the final output of the value vector weighted according to the attention weight.
[0106] It is worth noting that during the training process, the model calculates the output prediction value through forward propagation. Then, the root mean square error (RMSE) is used as the loss function to calculate the difference between the prediction result and the actual label. The definition of RMSE, that is, the expression of the root mean square error, is:
[0107] ;
[0108] In the formula, represents the predicted value of the model, is the actual value of the label, n is the number of samples, and RMSE can effectively measure the difference between the predicted value and the actual value. The smaller the value, the better the model performance.
[0109] In addition, after the loss is calculated, the gradient of the loss function relative to each weight is calculated through the Backward Propagation method. The weights are updated using the optimization algorithm (Adam optimizer) so that the loss function value gradually decreases. The entire training process will be iterated multiple times until the model performs stably on the validation set, indicating that the model parameters have been fully optimized.
[0110] After the model training is completed, an independent test set is used to evaluate the model. The data in the test set has never been used in the training process to ensure the objectivity and reliability of the evaluation results. During the test process, the root mean square error (RMSE) is also used as a performance indicator to measure the prediction accuracy of the model. By calculating the RMSE on the test set, we can intuitively understand the generalization ability of the model, that is, the performance of the model on unseen data.
[0111] In addition, a large amount of patient rehabilitation data, such as movement data, physiological parameters, and treatment feedback, is used to build and optimize deep learning models. These models can learn from complex rehabilitation data and predict the rehabilitation progress and needs of each patient. For example, the model can infer the most effective rehabilitation strategy, including exercise plan, rehabilitation schedule, and treatment frequency, based on the patient's initial condition and rehabilitation feedback.
[0112] In summary, the deep learning-based robotic arm rehabilitation data prediction method in the above-mentioned embodiment of the present invention extracts and fuses several spatiotemporal features through several residual convolution modules with convolution kernels of different sizes, and then predicts based on the long short-term memory neural network, as well as through the attention mechanism of the long short-term memory neural network. It can automatically adjust the training parameters of the robotic arm, and can also obtain the robotic arm data of the patient when using the robotic arm for rehabilitation, thereby making the robotic arm more effective in the process of upper limb rehabilitation of patients.
[0113] Embodiment 2
[0114] See also Figure 3 , which shows a robot arm rehabilitation data prediction system based on deep learning in a second embodiment of the present invention, the system includes:
[0115] The collection and processing module 10 is used to collect a number of rehabilitation data of the patient each time using the robotic arm, process the rehabilitation data to obtain a rehabilitation data set, and perform standardization processing on the rehabilitation data set to obtain a standardized rehabilitation data set;
[0116] An extraction and fusion module 20, an extraction and fusion module, is used to extract the spatiotemporal features in the standardized rehabilitation data set based on a plurality of residual convolution modules with convolution kernels of different sizes to obtain a plurality of spatiotemporal features, and to perform feature fusion on the plurality of the spatiotemporal features to obtain a multi-scale feature;
[0117] The extraction expression of the residual convolution module is:
[0118] ;
[0119] In the formula, represents the spatiotemporal features extracted by one of the residual convolution modules, , represents two different learnable parameters, Represents a multi-layer convolutional network with a convolution kernel of 3. represents the activation function, represents the rehabilitation data set after standardization;
[0120] The expression of multi-scale features obtained by feature fusion is:
[0121] ;
[0122] represents the multi-scale features, , Respectively represent the spatiotemporal features extracted by the other two residual convolution modules, , , They represent the learnable parameters of the corresponding scales respectively;
[0123] A prediction and capture module 30, configured to predict the multi-scale features based on a long short-term memory neural network to capture the dependency of the multi-scale features on time;
[0124] A fusion output module 40, used to fuse the multi-scale features and the dependency between the multi-scale features and time based on an attention mechanism to obtain final output data of the robotic arm;
[0125] The expression for fusing the multi-scale features and the dependency of the multi-scale features with time based on the attention mechanism to obtain the final output data of the robotic arm is:
[0126] ;
[0127] ;
[0128] In the formula, represents the weight of the t-th attention score, represents the learnable parameters of the attention mechanism, represents the bias of the attention mechanism, represents the number of time series, represents the tth vector in the value vector, represents the result of weighted summation, and represents the final output of the value vector weighted according to the attention weight.
[0129] In some optional embodiments, the collection and processing module 10 includes:
[0130] a processing and saving unit, used for sequentially performing missing value processing, abnormal value processing, duplicate value processing and data type conversion on a plurality of the rehabilitation data to obtain a rehabilitation data set, and saving the rehabilitation data set;
[0131] a calculation processing unit, configured to calculate a mean value and a standard deviation of each feature in the rehabilitation data set, and perform standardization processing on the rehabilitation data set based on the mean value and the standard deviation;
[0132] The expression of the standardization process is:
[0133] ;
[0134] In the formula, represents the rehabilitation data set after normalization, represents the original rehabilitation dataset, represents the mean, represents the standard deviation;
[0135] The division unit is used to divide the rehabilitation data set after the standardization process to obtain a training set, a validation set and a test set.
[0136] In some optional embodiments, the extraction and fusion module 20 includes:
[0137] An input unit, used to input the standardized rehabilitation data set into a plurality of residual convolution modules with convolution kernels of different sizes, wherein each of the residual convolution modules is composed of a five-layer convolutional neural network combined with a residual network;
[0138] A capture unit, used for capturing local features in the standardized rehabilitation data set based on the five-layer convolutional neural network, and performing feature enhancement by the residual network;
[0139] A fusion unit is used to fuse the local features based on a plurality of residual convolution modules with convolution kernels of different sizes to obtain multi-scale features.
[0140] The functions or operation steps implemented when the above modules and units are executed are generally the same as those in the above method embodiments, and will not be repeated here.
[0141] The implementation principle and technical effects of the robotic arm rehabilitation data prediction system provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0142] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0143] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. A deep learning-based robotic arm rehabilitation data prediction method, It is characterized in that The method comprises: Collecting a number of rehabilitation data of the patient each time using the robotic arm, processing the rehabilitation data to obtain a rehabilitation data set, and standardizing the rehabilitation data set to obtain a standardized rehabilitation data set, wherein the rehabilitation data include simple information of the patient, game information using the robotic arm, the patient's arm extension rotation angle, arm movement speed, recording time, gripping force, and pushing force; Extracting the spatiotemporal features in the standardized rehabilitation data set based on a plurality of residual convolution modules with convolution kernels of different sizes to obtain a plurality of spatiotemporal features, and fusing the plurality of spatiotemporal features to obtain multi-scale features; The steps of extracting the spatiotemporal features in the standardized rehabilitation data set by using a plurality of residual convolution modules based on convolution kernels of different sizes to obtain a plurality of spatiotemporal features, and fusing the plurality of spatiotemporal features to obtain multi-scale features include: Inputting the standardized rehabilitation data set into a plurality of residual convolution modules with convolution kernels of different sizes, wherein each of the residual convolution modules is composed of a five-layer convolutional neural network combined with a residual network; Capturing local features in the standardized rehabilitation dataset based on the five-layer convolutional neural network, and performing feature enhancement by the residual network; A plurality of residual convolution modules based on convolution kernels of different sizes fuse the local features to obtain multi-scale features; The extraction expression of the residual convolution module is: ; In the formula, represents the spatiotemporal features extracted by the residual convolution module with a convolution kernel of 3, , represents two different learnable parameters, Represents a multi-layer convolutional network with a convolution kernel of 3. represents the activation function, represents the rehabilitation data set after standardization; The expression of multi-scale features obtained by feature fusion is: ; represents the multi-scale features, , Respectively represent the spatiotemporal features extracted by the other two residual convolution modules, , , They represent the learnable parameters of the corresponding scales respectively; Predicting the multi-scale features based on a long short-term memory neural network to capture the dependency of the multi-scale features on time, wherein the long short-term memory neural network introduces a structure called a gating unit to more effectively capture and memorize long-term dependencies in a sequence to predict the multi-scale features; Based on the attention mechanism, the multi-scale features and the dependency between the multi-scale features and time are integrated to obtain the final output data of the robotic arm; The expression for fusing the multi-scale features and the dependency of the multi-scale features with time based on the attention mechanism to obtain the final output data of the robotic arm is: ; ; In the formula, represents the weight of the t-th attention score, represents the learnable parameters of the attention mechanism, represents the bias of the attention mechanism, represents the number of time series, represents the tth vector in the value vector, represents the result of weighted summation, and represents the final output of the value vector weighted according to the attention weight.
2. The method for predicting robotic arm rehabilitation data based on deep learning according to claim 1, It is characterized in that The step of processing the plurality of rehabilitation data to obtain a rehabilitation data set, and performing standardization processing on the rehabilitation data set to obtain a standardized rehabilitation data set comprises: performing missing value processing, outlier processing, duplicate value processing, and data type conversion on the plurality of rehabilitation data in sequence to obtain a rehabilitation data set, and saving the rehabilitation data set; Calculating a mean value and a standard deviation of each feature in the rehabilitation data set, and performing standardization processing on the rehabilitation data set based on the mean value and the standard deviation; The standardized rehabilitation data set is divided into a training set, a validation set, and a test set.
3. The method for predicting robotic arm rehabilitation data based on deep learning according to claim 2, It is characterized in that The expression of the standardization process is: ; In the formula, represents the rehabilitation data set after normalization, represents the original rehabilitation dataset, represents the mean, represents the standard deviation.
4. A deep learning-based robotic arm rehabilitation data prediction system, which is used to implement the deep learning-based robotic arm rehabilitation data prediction method according to any one of claims 1 to 3, It is characterized in that The system comprises: A collection and processing module, used for collecting a number of rehabilitation data of the patient each time using the robotic arm, processing the plurality of rehabilitation data to obtain a rehabilitation data set, and performing standardization processing on the rehabilitation data set to obtain a standardized rehabilitation data set; An extraction and fusion module, used for extracting the spatiotemporal features in the standardized rehabilitation data set based on a plurality of residual convolution modules with convolution kernels of different sizes to obtain a plurality of spatiotemporal features, and fusing the plurality of the spatiotemporal features to obtain multi-scale features; The extraction and fusion module comprises: An input unit, used to input the standardized rehabilitation data set into a plurality of residual convolution modules with convolution kernels of different sizes, wherein each of the residual convolution modules is composed of a five-layer convolutional neural network combined with a residual network; A capture unit, used for capturing local features in the standardized rehabilitation data set based on the five-layer convolutional neural network, and performing feature enhancement by the residual network; A fusion unit, used for fusing the local features based on a plurality of residual convolution modules with convolution kernels of different sizes to obtain multi-scale features; A prediction and capture module, used to predict the multi-scale features based on a long short-term memory neural network to capture the dependency of the multi-scale features on time; A fusion output module is used to fuse the multi-scale features and the dependency of the multi-scale features with time based on an attention mechanism to obtain final output data of the robotic arm.
5. The deep learning-based robotic arm rehabilitation data prediction system according to claim 4, It is characterized in that The collection and processing module comprises: a processing and saving unit, used for sequentially performing missing value processing, abnormal value processing, duplicate value processing and data type conversion on a plurality of the rehabilitation data to obtain a rehabilitation data set, and saving the rehabilitation data set; a calculation processing unit, configured to calculate a mean value and a standard deviation of each feature in the rehabilitation data set, and perform standardization processing on the rehabilitation data set based on the mean value and the standard deviation; The division unit is used to divide the rehabilitation data set after the standardization process to obtain a training set, a validation set and a test set.
Citation Information
Patent Citations
Training of upper limb stroke rehabilitation robotic system using kinetic pose estimation algorithm and deep learning
AU2020102303A4
Upper limb rehabilitation robot motion intention recognition model training and recognition method and device
CN114159080A
Lower limb rehabilitation robot follow-up control method based on deep reinforcement learning
CN114344093A
Robot capable of achieving weight reduction and gravity center transfer
CN114344094A
Natural image co-reference target positioning system and method based on eye movement and voice
CN115237255A
