Mirror image control hand rehabilitation method, system and terminal
By receiving the electromyography signals and images of healthy hands, using multimodal fusion strategies and deep learning models, accurate mirror control of the affected hands is achieved, which solves the problem of low accuracy of rehabilitation training in the existing technology and improves the patient's rehabilitation training effect.
Patent Information
- Application Number
- CN202410167778.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-06
- Publication Date
- 2024-12-13
AI Technical Summary
The prior art In the upper limb rehabilitation training of stroke patients, the accuracy of mirror control is low, it is difficult for patients to persist in training for a long time, and the lack of effective rehabilitation treatment leads to a decrease in muscle strength.
By receiving the electromyography signal and hand images of the patient's healthy hands, a multimodal fusion strategy is adopted, combined with the Transformer model and SSD model, the state of the healthy hands is recognized, and after the fusion process, the rehabilitation training auxiliary equipment is controlled to make the affected hands perform. Corresponding actions to achieve more accurate rehabilitation training.
It improves the accuracy of hand status recognition on the healthy side, ensures the accuracy and reliability of hand movements on the affected side, and enhances the efficiency of rehabilitation training and the patient's treatment experience.
Smart Images
Figure CN119112590A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of hand rehabilitation training, and in particular to a mirror-controlled hand rehabilitation method, system and terminal. Background Art
[0002] Stroke is an acute cerebrovascular disease, a group of diseases that cause brain tissue damage due to sudden rupture of brain blood vessels or blood vessel blockage that prevents blood from flowing into the brain. After a stroke occurs, muscle strength declines, leading to hemiplegia in a large number of patients, especially residual upper limb motor dysfunction, which is a common sequela symptom of stroke survivors and seriously affects the normal life of patients. In addition, the number of unilateral limb injuries caused by various accidents is also increasing. Due to the lack of effective rehabilitation treatment, after a long period of accumulation, these patients will experience varying degrees of muscle atrophy, resulting in a serious decline in muscle strength, and then unilateral limb motor dysfunction. Therefore, it is very necessary to provide effective upper limb rehabilitation training services. -
[0003] At present, rehabilitation training services are mainly provided by rehabilitation trainers and simple rehabilitation equipment. Rehabilitation trainers help patients with treatment through massage, which requires a lot of labor and is short of staff, which often prevents hospitals from arranging treatment for patients in time and misses the best treatment period; some ordinary rehabilitation training, although there is a mirror training function, the rehabilitation process is through the patient's own imagination of the affected hand performing the same action training, in fact, the affected hand does not move, this kind of training is difficult for patients to persist in for a long time. On this basis, the existing technology further proposes to detect the surface electromyography signal of the healthy hand and then identify the hand movement state of the healthy hand, and mirror control the affected hand according to the hand state of the healthy hand, so as to achieve rehabilitation training for the affected hand. In the above method, the movement state of the healthy hand is determined only by the recognition of a single electromyography signal, and the recognition accuracy still needs to be further improved. Summary of the invention
[0004] The purpose of the present invention is to overcome the problems of the prior art and to provide a mirror-controlled hand rehabilitation method, system and terminal.
[0005] The object of the present invention is to achieve the following technical solution: a mirror-controlled hand rehabilitation method, the method comprising the following steps:
[0006] Receive the electromyographic signals and hand images of the patient's healthy hand during rehabilitation training;
[0007] Classify and process the electromyographic signal to obtain a first hand state recognition result;
[0008] Performing target detection processing on the hand image to obtain a second hand state recognition result;
[0009] The first hand state recognition result and the second hand state recognition result are fused to obtain a final hand state recognition result of the healthy hand;
[0010] According to the final hand state recognition result of the healthy hand, or the final hand state recognition result and the size of the electromyographic signal, the rehabilitation training auxiliary equipment is controlled to drive the patient's affected hand to make corresponding movements, thereby realizing rehabilitation training for the affected hand.
[0011] In one example, before classifying the electromyographic signal, the electromyographic signal is also preprocessed, including:
[0012] Perform wavelet decomposition on the electromyographic signal to obtain high-frequency signals and low-frequency signals at multiple decomposition layers;
[0013] The high-frequency signal of each decomposition layer is denoised, and the calculation expression of denoising is:
[0014]
[0015] Among them, d j,k represents the kth wavelet coefficient of the jth layer; Δ represents the threshold; ρ is a coefficient greater than 0;
[0016] The denoised high-frequency signal and the low-frequency signal of the corresponding level are reconstructed to obtain the denoised electromyographic signal.
[0017] In one example, the electromyographic signal is classified and processed, including:
[0018] Train the Transformer model based on the self-attention mechanism;
[0019] Convert the electromyographic signal into a vector representation matrix;
[0020] Input the vector representation matrix into the encoder of the Transformer model for feature extraction to obtain the encoding information matrix;
[0021] The encoded information matrix is input into the decoder of the Transformer model for decoding processing to obtain the first hand state recognition result.
[0022] In one example, the Transformer model is trained using the cross entropy loss function. The cross entropy loss function L CE_smoothing The calculation expression is:
[0023]
[0024] Where m is the number of categories; λ is the smoothing parameter; y irepresents the true label of the i-th category; the label k range is [1,m]; z is the unstandardized probability.
[0025] In one example, object detection processing is performed on a hand image, including:
[0026] Add multiple prediction convolution layers to the prediction convolution layer of the traditional SSD model;
[0027] Train the SSD model;
[0028] The hand image is input into the SSD model, and the basic network and auxiliary convolutional layer are used to extract features. Then, the prediction convolutional layer is used to perform target detection processing to obtain the second hand state recognition result.
[0029] In one example, when training the SSD model, it includes:
[0030] Introducing generalized intersection-over-union and center point regression loss into the total loss function, the total loss function L total The expression is:
[0031]
[0032] L GIoU =1-GIoU(B w ,B v )
[0033]
[0034] Where N is the number of matching samples L conf Classification loss function; L loc is the regression loss function; α is the weight coefficient of the loss function; L GIoU is the generalized intersection-over-union loss function; L center is the center point regression loss function; β is the weight coefficient of the generalized intersection-over-union loss, γ is the weight coefficient of the center point regression loss; B w is the prediction box; B v is the real frame; The center point coordinates of the box predicted by the model; p i is the center point coordinate of the real frame.
[0035] It should be further explained that the technical features corresponding to the various examples of the above method can be combined or replaced with each other to form a new technical solution.
[0036] An example of the present invention further provides a mirror-controlled hand rehabilitation system, the system comprising a signal processing module, the signal processing module comprising a signal receiving unit, a classification unit, a target detection unit, an information fusion unit and a control unit;
[0037] The signal receiving unit is used to receive the electromyographic signal and hand image of the patient's healthy hand during the rehabilitation training;
[0038] The classification unit is used to classify the electromyographic signal to obtain a first hand state recognition result;
[0039] The target detection unit is used to perform target detection processing on the hand image to obtain a second hand state recognition result;
[0040] The information fusion unit is used to fuse the first hand state recognition result and the second hand state recognition result to obtain a final hand state recognition result of the healthy hand;
[0041] The control unit controls the final hand state recognition result of the healthy hand to control the rehabilitation training auxiliary equipment to drive the patient's affected hand to make corresponding movements, thereby realizing rehabilitation training for the affected hand.
[0042] In one example, the system further includes an information acquisition module, including an electromyographic signal acquisition unit and an image acquisition unit;
[0043] The electromyographic signal acquisition unit is used to collect the electromyographic signals of the patient's healthy hand during the rehabilitation training;
[0044] The image acquisition unit is used to acquire hand images of the patient's healthy hand during the movement of the patient's healthy hand during rehabilitation training.
[0045] In one example, the electromyographic signal acquisition unit is an arm ring structure, including a plurality of box bodies, a plurality of cavities are formed in the box bodies at intervals, and electromyographic signal sensors are arranged in the cavities, and the box bodies are connected by elastic plastic.
[0046] It should be further explained that the technical features corresponding to the above system examples can be combined or replaced with each other to form a new technical solution.
[0047] The present invention also includes a terminal, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and when the processor runs the computer instructions, the steps of the mirror-controlled hand rehabilitation method formed by any one or more of the above examples are executed.
[0048] Compared with the prior art, the present invention has the following beneficial effects:
[0049] 1. In one example, surface electromyography signal classification and hand image recognition are used to identify the healthy hand status respectively, and finally the two recognition results are fused. The multimodal fusion strategy is adopted to improve the reliability of the final hand status recognition result, thereby ensuring the recognition accuracy of the healthy hand status.
[0050] 2. In one example, through threshold denoising processing, the advantages of hard threshold method and soft threshold method can be taken into account. The denoised signal has good edge properties, continuity, smooth denoising effect, and the signal will not have large fluctuations after processing.
[0051] 3. In one example, by introducing label smoothing into the cross entropy loss function of the Transformer model, it is possible to prevent model overfitting and improve generalization performance.
[0052] 4. In one example, adding generalized intersection-over-union to the loss function of the SSD model can optimize the model parameters and help the model better learn the position and shape information of the gesture, thereby improving the accuracy of the gesture recognition algorithm; adding center point regression loss to the loss function of the SSD model can help the model better predict the center point position of the gesture bounding box, and then more accurately identify the position and shape of the gesture. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings. The accompanying drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to represent the same or similar parts. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.
[0054] Figure 1 A method flow chart provided for an example of the present invention;
[0055] Figure 2 This is a plan view of the arm ring structure provided as an example of the present invention.
[0056] In the figure: 1-box, 2-electromyographic signal sensor, 3-elastic material, 4-circuit board. DETAILED DESCRIPTION
[0057] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0058] In the description of the present invention, it should be noted that the directions or positional relationships indicated by "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are directions or positional relationships based on the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the use of ordinal numbers (e.g., "first and second", "first to fourth", etc.) is to distinguish objects, and is not limited to this order, and cannot be understood as indicating or implying relative importance.
[0059] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0060] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0061] In one example, if Figure 1 As shown, a mirror-controlled hand rehabilitation method comprises the following steps:
[0062] S1: Receive the electromyographic signals and hand images of the patient's healthy hand during rehabilitation training;
[0063] S2: classify and process the electromyographic signal to obtain a first hand state recognition result;
[0064] S3: performing target detection processing on the hand image to obtain a second hand state recognition result;
[0065] S4: fusing the first hand state recognition result and the second hand state recognition result to obtain a final hand state recognition result of the healthy hand;
[0066] S5: According to the final hand state recognition result of the healthy hand, or the final hand state recognition result and the size of the electromyographic signal, the rehabilitation training auxiliary equipment is controlled to drive the patient's affected hand to make corresponding movements, thereby realizing rehabilitation training for the affected hand.
[0067] Optionally, steps S2 and S3 may be executed simultaneously or in any order, that is, step S3 may be executed first and then step S2.
[0068] The patient wears a signal acquisition device on the arm of the healthy hand, which is provided with an electromyographic signal sensor. When the healthy hand undergoes rehabilitation training, different hand states of the healthy hand will lead to different changes in electromyographic signals, including fisting, single or multi-finger extension, hand extension, etc. In step S1, it is preferred to use multiple electromyographic signal sensors to collect multi-channel electromyographic signals in the rehabilitation exercise of the healthy hand in real time. For example, the signal is collected by electromyographic signal sensors evenly distributed on the healthy arm of the patient, avoiding the defect of inaccurate data caused by interference, noise, etc. on a single channel or a few channels, and ensuring the stability and accuracy of signal acquisition. In step S1, the hand image is collected by a camera, and the camera can be fixed by a supporting device to collect the hand image of the healthy hand.
[0069] In step S2, the electromyographic signal is classified and processed, which can be implemented by existing CNN, LSTM, Transformer and other models, and the Transformer model is preferably used in this example. In step S3, the hand image is subjected to target detection, which can be implemented by existing Yolo, SSD and other models, and the SSD model is preferably used in this example.
[0070] In step S4, the first hand state recognition result and the second hand state recognition result are fused by assigning different weights to the first hand state recognition result and the second hand state recognition result.
[0071] In step S5, the mirror control is to control the patient's affected hand to make similar movements based on the final hand state recognition result. At the same time, the size of the surface electromyography signal of the healthy hand at this time can be used as a given force to apply the same force to the affected hand. The size of the electromyography signal is determined based on the collected electromyography amplitude. When the surface electromyography signal becomes larger, the force applied to the affected hand is increased; when the surface electromyography signal becomes smaller, the force applied to the affected hand is reduced. By obtaining the size of the surface electromyography signal value of the patient's healthy hand, it is helpful to control the strength of the affected hand movement, allowing the patient to better adjust more conveniently according to their own situation, thereby achieving more effective rehabilitation treatment. Optionally, the auxiliary equipment is an existing equipment, such as a mechanical glove in the invention patent application with publication number CN111696645A and name "A hand exoskeleton rehabilitation training device and method based on surface electromyography signal". -
[0072] More specifically, mirror control is training for patients based on the actual conditions of their affected hand. If the affected hand of the patient loses the function of free extension, single finger or multi-finger movement, then the healthy hand can stretch, single finger or multi-finger movement, and the glove worn on the affected hand in the control auxiliary device can also produce corresponding movements by detecting the state of the healthy hand's extension, single finger or multi-finger movement, that is, the glove drives the affected hand to stretch, single finger or multi-finger movement, and exerts the same force to control the movement of the affected hand according to the actual output force of the healthy hand. Through such reciprocating rehabilitation training, the patient's affected hand muscle strength can be strengthened, and the affected hand can move without the control of the healthy hand, achieving the ultimate rehabilitation effect. -
[0073] Furthermore, force is applied to the patient's affected hand, mainly through the controller connected to two pairs of wires, one end of each pair of wires is connected to the controller, and the other end is connected to two electrodes, which are placed on the arm of the affected hand. Finally, the controller emits a current corresponding to the surface electromyography signal value of the healthy hand. When the current flows through the wires and electrodes, it can stimulate the affected hand to make movements of the same force, thereby achieving mirror control, and can control the affected hand to make similar movements according to the surface electromyography signal of the patient's healthy hand, which is easy to meet the self-control of hemiplegic patients and improve the efficiency of rehabilitation training.
[0074] The method of the present invention aims to replace the rehabilitation training treatment of a rehabilitation trainer. During mirror therapy, the patient's healthy hand can actually control the affected hand to make corresponding movements, and can follow in real time according to the detected surface electromyography signal value of the healthy hand, thereby changing the force applied to the affected hand; in addition, in addition to collecting surface electromyography signals for hand state recognition, a visual sensor is also used to classify and recognize the gestures of the healthy hand, and the two recognition methods of surface electromyography signals and sensors are combined, and a multi-modal fusion strategy is adopted to improve the recognition accuracy of the healthy hand state, so that patients can feel a better treatment experience, stimulate patients to take active treatment, and optimize the patient's usage experience.
[0075] In one example, before classifying the electromyographic signal, the original electromyographic signal collected is a non-stationary signal and is susceptible to interference from the environment and other bioelectric signals (such as electrocardiogram signals). Therefore, the original electromyographic signal needs to be preprocessed, including:
[0076] S01: Perform wavelet decomposition on the electromyographic signal to obtain high-frequency signals and low-frequency signals of multiple decomposition layers;
[0077] S02: De-noise the high-frequency signal of each decomposition layer. The calculation expression of the de-noising process is:
[0078]
[0079] Among them, dj,k represents the kth wavelet coefficient of the jth layer; Δ represents the threshold value, which can be obtained based on historical experience or can be a variable threshold value; ρ is a coefficient greater than 0;
[0080] S03: reconstructing the denoised high-frequency signal and the low-frequency signal of the corresponding level to obtain a denoised electromyographic signal.
[0081] In step S01, the signal is decomposed by wavelet, and each layer of decomposition will obtain high-frequency signals and low-frequency signals. Then the low-frequency signal is decomposed again to obtain new high-frequency signals and low-frequency signals. And so on, multiple layers of high-frequency signals and low-frequency signals are obtained.
[0082] In step S02, based on the above denoising processing expression, it can be seen that the threshold can automatically follow the change of the decomposition layer number, so as to take into account the advantages of the hard threshold method and the soft threshold method. The denoised signal has good edge properties, continuity, and smooth denoising effect, and the processed signal will not have large fluctuations.
[0083] Optionally, signal preprocessing can also be achieved through filtering, including two key steps: bandpass filtering and time window processing. Specifically, a 5Hz-200Hz bandpass fourth-order Butterworth filter is used to remove low-frequency noise, high-frequency interference and motion artifacts in the electromyographic signal, thereby significantly improving the signal-to-noise ratio. In addition, in the stage of time window processing, the electromyographic signal corresponding to the task time period is accurately extracted according to the experimental paradigm, so as to make full use of the corresponding data of the time window in each trial, better meet the needs of subsequent analysis, and improve the efficiency of data processing. -
[0084] Through the above preprocessing means, the weak electromyographic signal with noise interference can be denoised to obtain a more stable electromyographic signal, which is convenient for signal feature extraction.
[0085] In one example, a Transformer model based on a self-attention mechanism is used to classify electromyographic signals. The Transformer model includes an encoder and a decoder. The classification process includes:
[0086] S21: Train the Transformer model to obtain a Transformer model that meets the classification performance requirements;
[0087] S22: Convert the electromyographic signal into a vector representation matrix of the input sequence (one-dimensional sequence representation, channel*number of samples*trials);
[0088] S23: inputting the vector representation matrix into an encoder, and performing encoding processing through multiple encoding layers to obtain an encoding information matrix;
[0089] S24: Input the encoded information matrix into the decoder, and perform decoding processing through multiple decoding layers, linear layers and Softmax logistic regression layers to obtain a first hand state recognition result.
[0090] This example introduces the self-attention mechanism into the Transformer model, which enables the model to establish connections between different positions in the input sequence, making it easier to capture the global information in the sequence. The calculation process of the self-attention mechanism is expressed as:
[0091]
[0092] Among them, Q h is the query vector; H h is the key vector; V h is the value vector; n is the number of parallel self-attention layers; d k is the vector dimension of input and output. In order to further improve the calculation speed of the attention mechanism, the Multi-head attention mechanism is used to enable multiple heads to participate in the attention calculation in parallel. The specific calculation of Multi-head attention is as follows:
[0093] MHSA(Q n ,H n ,V n )=[head 1 ;head 2 ,…,head n ]W
[0094] Where W is the projection matrix.
[0095] In one example, the Transformer model is trained using the cross entropy loss function. The cross entropy loss function L CE_smoothing The calculation expression is:
[0096]
[0097] Where m is the number of categories; λ is the smoothing parameter; y i represents the true label of the i-th category; the label k range is [1, m]; z is the unstandardized probability. This example introduces label smoothing in the cross entropy loss function of the Transformer model to prevent model overfitting and improve generalization performance.
[0098] Furthermore, in addition to classifying the gesture state of the healthy hand, the magnitude of the surface electromyographic signal value is also calculated. The gesture state classification is to regard each gesture state as a classification category, use the cross entropy loss function to measure the difference between the predicted result and the true label, match the electromyographic signal data with its corresponding gesture state label, and gradually optimize and improve the accuracy of gesture state recognition by minimizing the cross entropy loss function.
[0099] In one example, an SSD model is used to perform object detection processing on a hand image. The SSD model includes a base network, an auxiliary convolution layer, and a prediction convolution layer connected in sequence. The base network can be a VGG-16 network. In this example, the object detection processing includes:
[0100] S31: Add multiple prediction convolution layers to the prediction convolution layer of the traditional SSD model;
[0101] S32: training the SSD model to obtain an SSD model that meets the target detection performance;
[0102] S33: The hand image is input into the SSD model, and the basic network and auxiliary convolutional layer are used to extract features. The prediction convolutional layer is used to perform target detection processing, and the coordinates, confidence and category information of the prediction box are output to obtain the second hand state recognition result.
[0103] In step S31, when collecting the hand image, the size of the target hand in the image is uncertain. In order to achieve more accurate detection, multiple prediction convolution layers are added to the last few layers of the network to achieve multi-scale target detection. Optionally, each detection frame can be screened by introducing a filtering strategy, that is, evaluating the position, size and other information of the detection frame, removing the interfering detection frame, and then outputting the final detection result.
[0104] In step S32, when training the SSD model, the object detection and positioning tasks in the image are combined into an end-to-end classification and positioning model, and a loss function is introduced, which is expressed as:
[0105]
[0106] Among them, L conf Classification loss function; L loc is the regression loss function; N is the number of matching samples, and α is the weight coefficient of the classification loss function and the regression loss function.
[0107] In one example, when training the SSD model, the following steps are also included:
[0108] The generalized intersection-over-union ratio and center point regression loss are introduced into the total loss function, and the SSD model is trained through the total loss function. The total loss function L total The expression is:
[0109]
[0110] L GIoU =1-GIoU(B w ,B v )
[0111]
[0112] Among them, L GIoU is the generalized intersection-over-union loss function; L center is the center point regression loss function; β is the weight coefficient of the generalized intersection-over-union loss, γ is the weight coefficient of the center point regression loss; B w is the prediction box; B v is the real frame; The center point coordinates of the box predicted by the model; p i is the center point coordinate of the real frame.
[0113] In this example, adding generalized intersection-over-union to the loss function of the SSD model can optimize the model parameters and help the model better learn the position and shape information of the gesture, thereby improving the accuracy of the gesture recognition algorithm; adding center point regression loss to the loss function of the SSD model can help the model better predict the center point position of the gesture bounding box, and thus more accurately identify the position and shape of the gesture.
[0114] Combining the above examples, a preferred method of the present invention is obtained, comprising the following steps:
[0115] S100: receiving electromyographic signals and hand images of the patient's healthy hand during rehabilitation training;
[0116] S200: Preprocessing of electromyographic signals:
[0117] Perform wavelet decomposition on the electromyographic signal to obtain high-frequency signals and low-frequency signals at multiple decomposition layers;
[0118] Perform denoising on the high-frequency signal of each decomposition layer;
[0119] The denoised high-frequency signal and the corresponding low-frequency signal are reconstructed to obtain the denoised electromyographic signal;
[0120] S300: Classify and process the pre-processed electromyographic signal:
[0121] Train the Transformer model to obtain a Transformer model that meets the classification performance requirements;
[0122] Convert the electromyographic signal into a vector representation matrix of the input sequence;
[0123] Input the vector representation matrix into the encoder of the Transformer model for feature extraction to obtain the encoding information matrix;
[0124] Inputting the encoded information matrix into the decoder of the Transformer model for decoding processing to obtain a first hand state recognition result;
[0125] S400: Target detection processing of hand image:
[0126] Add multiple prediction convolution layers to the prediction convolution layer of the traditional SSD model;
[0127] Train the SSD model to obtain an SSD model that meets the target detection performance;
[0128] The hand image is input into the SSD model, and the basic network and auxiliary convolutional layer are used to extract features. Then, the prediction convolutional layer is used to perform target detection processing to obtain the second hand state recognition result.
[0129] S500: assigning different weights to the first hand state recognition result and the second hand state recognition result and then performing fusion processing to obtain a final hand state recognition result of the healthy hand;
[0130] S600: According to the final hand state recognition result of the healthy hand and the size of the electromyographic signal, the rehabilitation training auxiliary equipment is controlled to drive the patient's affected hand to make corresponding movements, thereby achieving rehabilitation training for the affected hand.
[0131] The present invention also includes a mirror-controlled hand rehabilitation system, which has the same inventive concept as the hand rehabilitation method formed by any one of the above examples or a combination of multiple examples, and the system includes a signal processing module, including a signal receiving unit, a classification unit, a target detection unit, an information fusion unit and a control unit;
[0132] Among them, the signal receiving unit is used to receive the electromyographic signal and hand image of the patient's healthy hand during the movement of rehabilitation training; the classification unit is used to classify the electromyographic signal to obtain a first hand state recognition result; the target detection unit is used to perform target detection processing on the hand image to obtain a second hand state recognition result; the information fusion unit is used to fuse the first hand state recognition result and the second hand state recognition result to obtain a final hand state recognition result of the healthy hand; the control unit controls the rehabilitation training auxiliary equipment to drive the patient's affected hand to make corresponding movements according to the final hand state recognition result of the healthy hand, or the final hand state recognition result and the size of the electromyographic signal, thereby realizing rehabilitation training for the affected hand.
[0133] In one example, the system also includes an information acquisition module, including an electromyographic signal acquisition unit and an image acquisition unit; wherein the electromyographic signal acquisition unit is used to collect electromyographic signals during the movement of the patient's healthy hand during rehabilitation training, and may be an electromyographic signal sensor; the image acquisition unit is used to collect hand images during the movement of the patient's healthy hand during rehabilitation training, and may be a camera.
[0134] In one example, the electromyographic signal acquisition unit is an arm ring structure, including several boxes, wherein multiple cavities are arranged in intervals in the boxes, and electromyographic signal sensors are arranged in the cavities, and the boxes are connected by elastic plastic. Optionally, the non-cavity part of the box can be used to lay out the circuit board. Figure 2 As shown, the armband structure is arranged at an angle of about 30 degrees with the inner side tilted downward. There are 8 plastic boxes 1 made of insulating materials, each box is used to place 3 electromyographic signal sensors 2. The 8 insulating plastic boxes 1 are connected in series to form a circle to form a ring-shaped structure, which is convenient for the hand to wear. There are 24 electromyographic signal sensors 2, divided into 8 groups, 3 in each group, and every 3 electromyographic signal sensors 2 are fixed to a circuit board 4, and then placed on the inner side of the armband and fixed in the box 1 without falling off. Through contact with the surface skin, the function of detecting and collecting the surface electromyographic signal of the arm is achieved. There are 8 circuit boards 4, and 3 electromyographic signal sensors 2 are fixed on each circuit board 4, and then they are installed in the plastic box 1 respectively, and the output end of the circuit board is connected to the signal transmission line. The boxes 1 are connected to each other through an elastic material 3 with a certain degree of soft elasticity to meet the arbitrary scaling when worn on arms of different sizes, and a wire is passed through the inside of the soft elastic material 3 to connect all the electromyographic signal sensors 2, so that their signals are collected in one of the plastic boxes 1 for unified management. Since the armband is made of soft and elastic material, it can stably form good contact with the skin surface after being worn, and can better detect surface electromyographic signals.
[0135] An example of the present invention further provides a terminal, which has the same inventive concept as any example or a combination of multiple examples corresponding to the above-mentioned mirror-controlled hand rehabilitation method, including a memory and a processor, wherein the memory stores computer instructions that can be run on the processor, and the processor executes the steps of the above-mentioned mirror-controlled hand rehabilitation method when running the computer instructions. The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.
[0136] In one example, the terminal, i.e., the electronic device, is presented in the form of a general-purpose computing device, and the components of the electronic device may include but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).
[0137] The storage unit stores a program code, and the program code can be executed by the processing unit, so that the processing unit performs the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the processing unit can perform the above-mentioned mirror-controlled hand rehabilitation method.
[0138] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache memory unit, and may further include a read-only memory unit (ROM).
[0139] The storage unit may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0140] The bus may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0141] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0142] Through the above description, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method of the exemplary embodiment of the present application.
[0143] The above specific implementation methods are detailed descriptions of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.
Claims
1. A mirror-controlled hand rehabilitation method, characterized in that: The following steps are involved: Receive the electromyographic signals and hand images of the patient's healthy hand during rehabilitation training; Classify and process the electromyographic signal to obtain a first hand state recognition result; Performing target detection processing on the hand image to obtain a second hand state recognition result; The first hand state recognition result and the second hand state recognition result are fused to obtain a final hand state recognition result of the healthy hand; According to the final hand state recognition result of the healthy hand, or the final hand state recognition result and the size of the electromyographic signal, the rehabilitation training auxiliary equipment is controlled to drive the patient's affected hand to make corresponding movements, thereby realizing rehabilitation training for the affected hand.
2. The mirror-controlled hand rehabilitation method according to claim 1, characterized in that: Before the electromyographic signal is classified and processed, the electromyographic signal is preprocessed, including: Perform wavelet decomposition on the electromyographic signal to obtain high-frequency signals and low-frequency signals at multiple decomposition layers; The high-frequency signal of each decomposition layer is denoised, and the calculation expression of denoising is: Among them, d j,k represents the kth wavelet coefficient of the jth layer; Δ represents the threshold; ρ is a coefficient greater than 0; The denoised high-frequency signal and the low-frequency signal of the corresponding level are reconstructed to obtain the denoised electromyographic signal.
3. The mirror-controlled hand rehabilitation method according to claim 1, characterized in that: Classify and process the electromyographic signals, including: Train the Transformer model based on the self-attention mechanism; Convert the electromyographic signal into a vector representation matrix; Input the vector representation matrix into the encoder of the Transformer model for feature extraction to obtain the encoding information matrix; The encoded information matrix is input into the decoder of the Transformer model for decoding processing to obtain the first hand state recognition result.
4. The mirror-controlled hand rehabilitation method according to claim 3, characterized in that: The Transformer model is trained using the cross entropy loss function. The cross entropy loss function L CE_smoothing The calculation expression is: Where m is the number of categories; λ is the smoothing parameter; y i represents the true label of the i-th category; the label k range is [1,m]; z is the unstandardized probability.
5. The mirror-controlled hand rehabilitation method according to claim 1, characterized in that: Perform object detection processing on hand images, including: Add multiple prediction convolution layers to the prediction convolution layer of the traditional SSD model; Train the SSD model; The hand image is input into the SSD model, and the basic network and auxiliary convolutional layer are used to extract features. Then, the prediction convolutional layer is used to perform target detection processing to obtain the second hand state recognition result.
6. The mirror-controlled hand rehabilitation method according to claim 5, characterized in that: When training the SSD model, it includes: Introducing generalized intersection-over-union and center point regression loss into the total loss function, the total loss function L total The expression is: L GIoU =1-GIoU(B w ,B v ) Where N is the number of matching samples L conf Classification loss function; L loc is the regression loss function; α is the weight coefficient of the loss function; L GIoU is the generalized intersection-over-union loss function; L center is the center point regression loss function; β is the weight coefficient of the generalized intersection-over-union loss, γ is the weight coefficient of the center point regression loss; B w is the prediction box; B v is the real frame; The center point coordinates of the box predicted by the model; p i is the center point coordinate of the real frame.
7. A mirror-controlled hand rehabilitation system, characterized in that: It includes a signal processing module, which includes a signal receiving unit, a classification unit, a target detection unit, an information fusion unit and a control unit; The signal receiving unit is used to receive the electromyographic signal and hand image of the patient's healthy hand during the rehabilitation training; The classification unit is used to classify the electromyographic signal to obtain a first hand state recognition result; The target detection unit is used to perform target detection processing on the hand image to obtain a second hand state recognition result; The information fusion unit is used to fuse the first hand state recognition result and the second hand state recognition result to obtain the final hand state recognition result of the healthy hand; The control unit controls the final hand state recognition result of the healthy hand to control the rehabilitation training auxiliary equipment to drive the patient's affected hand to make corresponding movements, thereby realizing rehabilitation training for the affected hand.
8. The mirror-controlled hand rehabilitation system according to claim 7, characterized in that: The system also includes an information acquisition module, including an electromyographic signal acquisition unit and an image acquisition unit; The electromyographic signal acquisition unit is used to collect the electromyographic signals of the patient's healthy hand during the rehabilitation training; The image acquisition unit is used to acquire hand images of the patient's healthy hand during the movement of the patient's healthy hand during rehabilitation training.
9. The mirror-controlled hand rehabilitation system according to claim 8, characterized in that: The electromyographic signal acquisition unit is an arm ring structure, comprising a plurality of box bodies, wherein a plurality of cavities are formed at intervals in the box bodies, and electromyographic signal sensors are arranged in the cavities, and the box bodies are connected by elastic plastic.
10. A terminal comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, characterized in that: When the processor runs the computer instructions, the processor performs the steps of a mirror-controlled hand rehabilitation method as described in any one of claims 1-6.