Auxiliary rehabilitation system for affected limb, computer program product and storage medium

Abstract

The invention provides an auxiliary rehabilitation system for an affected limb, a computer program product and a storage medium. The auxiliary rehabilitation system for the affected limb comprises a brain-computer interface used for acquiring a motor imagery signal of a user; the exoskeleton equipment is used for assisting the affected limb of the user to do corresponding movement according to the motor imagery signal; the image acquisition equipment is used for acquiring a first motion image of the affected limb; and the mixed reality display equipment is used for processing the first motion image according to the first motion data of the affected limb to generate a second motion image of the healthy limb, and displaying the second motion image to the user so as to carry out bidirectional closed-loop re-learning from the central nervous system to the peripheral nerve on the user.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical technology, and in particular to an auxiliary rehabilitation system for an affected limb, a computer program product, and a computer-readable storage medium. Background Art

[0002] Brain-computer interface (BCI) is a technology that directly converts brain signals into control of external devices or applications. In the process of stroke rehabilitation, motor imagery (MI) is combined with BCI as a very important rehabilitation method, which allows patients to simulate the process of movement in their brains through motor imagery, that is, imagine themselves doing a certain movement without actually performing the movement. This imaginative activity can activate the area of ​​the brain responsible for controlling movement and promote the release of neurotransmitters, thereby helping to restore impaired motor function. Combined with BCI technology, the patient's motor imagery signals can be captured and identified in real time and converted into instructions to control external devices.

[0003] In stroke rehabilitation, traditional motor imagery combined with the application of brain-computer interfaces can capture brain signals and convert them into control commands, but lacks the support of mixed reality technology, making it difficult to adjust training programs in a timely manner based on real-time monitoring data and provide personalized feedback and guidance. Therefore, traditional motor imagery combined with the application of brain-computer interfaces often lacks an immersive experience, and patients may lack enthusiasm and motivation for training, reducing participation and training effectiveness. At the same time, the rehabilitation needs and abilities of each stroke patient are different, so the lack of a customized training program may limit the effectiveness of rehabilitation.

[0004] In order to overcome the above-mentioned defects of the prior art, there is an urgent need in the art for an auxiliary rehabilitation technology for an affected limb, which can provide the user with real-time feedback of a second motion image of the healthy limb while assisting the user in exercising, thereby improving the rehabilitation effect of the affected limb. Summary of the invention

[0005] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or decisive elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides an auxiliary rehabilitation technology for an affected limb, a computer program product, and a computer-readable storage medium, which are used to provide the user with real-time feedback of a second motion image of the healthy limb while assisting the user in exercising, thereby improving the rehabilitation effect of the affected limb.

[0007] Specifically, the auxiliary rehabilitation system for the affected limb provided according to the first aspect of the present invention includes: a brain-computer interface for acquiring a user's motor imagination signal; an exoskeleton device for assisting the user's affected limb to perform corresponding movements according to the motor imagination signal; an image acquisition device for acquiring a first motion image of the affected limb; and a mixed reality display device for processing the first motion image according to the first motion data of the affected limb to generate a second motion image of the healthy limb, and displaying the second motion image to the user, so as to perform a two-way closed-loop re-learning of the user from the central nervous system to the peripheral nerves.

[0008] Furthermore, in some embodiments of the present invention, the assisted rehabilitation system also includes: a motion capture device for collecting the first motion data of the affected limb, wherein the motion capture device includes multiple sensors, and the multiple sensors are selected from at least one of inertial sensors, optical sensors, magnetic sensors, and ultrasonic sensors, which are arranged at multiple key joints of the affected limb for capturing the second motion data of each of the key joints in real time to constitute the first motion data of the affected limb.

[0009] Furthermore, in some embodiments of the present invention, the exoskeleton device is configured to generate control instructions for assisting the user's affected limb to perform corresponding movements based on the motor imagery signal and the first motion data collected by the motion capture device.

[0010] Furthermore, in some embodiments of the present invention, the mixed reality display device includes a YOLOv8 module for tracking the position of the exoskeleton device in the first motion image to determine the first motion data of the affected limb.

[0011] Furthermore, in some embodiments of the present invention, the mixed reality display device includes a video restoration module, which is configured to: determine a target area for video restoration based on the first motion data; generate a filled image of the healthy limb in the target area through optical flow calculation, feature propagation and content generation algorithms; and remove the original image of the target area from the first motion image, and insert the filled image to generate the second motion image.

[0012] Furthermore, in some embodiments of the present invention, the assisted rehabilitation system also includes: a controller, used to determine the corresponding motion scene, target action, and / or the auxiliary strength of the exoskeleton device according to the rehabilitation stage of the affected limb.

[0013] Further, in some embodiments of the present invention, the affected limb is a hand, and the controller is configured to: determine the rehabilitation stage of the affected limb; in response to the affected limb being in the first rehabilitation stage, display a first motion scene of grasping at least one first target object via the mixed reality display device, and provide a larger first torque auxiliary force to the affected limb according to the user's motion imagination signal via the exoskeleton device to help the user complete the first target action of grasping; in response to the affected limb being in the second rehabilitation stage, display a second motion scene of rotating and / or picking up at least one second target object via the mixed reality display device, and provide a moderate second torque auxiliary force to the affected limb according to the user's motion imagination signal via the exoskeleton device to help the user complete the second target action of rotating and / or picking up; and in response to the affected limb being in the third rehabilitation stage, display a third motion scene simulating daily life via the mixed reality display device, and provide a smaller third torque auxiliary force to the affected limb according to the user's motion imagination signal via the exoskeleton device to help the user complete at least one third target action related to daily life.

[0014] Furthermore, in some embodiments of the present invention, the controller is also configured to: in response to completing a training session, generate a rehabilitation report indicating the rehabilitation stage of the affected limb based on first motion data collected by a motion capture device arranged at multiple key joints of the affected limb.

[0015] Furthermore, in some embodiments of the present invention, the controller is also configured to: in response to the affected limb being in the first rehabilitation stage, control the mixed reality display device to display the first motion scene with a higher first frame rate and a smaller first resolution; in response to the affected limb being in the second rehabilitation stage, control the mixed reality display device to display the second motion scene with a higher second frame rate and a smaller second resolution; and in response to the affected limb being in the third rehabilitation stage, control the mixed reality display device to display the third motion scene with a lower third frame rate and a larger third resolution.

[0016] Furthermore, in some embodiments of the present invention, the assisted rehabilitation system also includes a human-computer interaction interface, and the controller is further configured to: obtain user instructions via the human-computer interaction interface; and adjust the amount of auxiliary force provided by the exoskeleton device to the affected limb, and / or the display frame rate and / or resolution of the mixed reality display device according to the user instructions.

[0017] In addition, a computer program product is provided according to the second aspect of the present invention. It includes computer instructions. When the computer instructions are executed by the processor, the following steps are implemented: obtaining the user's motor imagery signal via the brain-computer interface; assisting the user's affected limb to perform corresponding movements according to the motor imagery signal via the exoskeleton device; acquiring the first motion image of the affected limb via the image acquisition device; and processing the first motion image according to the first motion data of the affected limb via the mixed reality display device to generate a second motion image of the healthy limb, and displaying the second motion image to the user, so as to perform a two-way closed-loop re-learning of the user from the central nervous system to the peripheral nerves.

[0018] In addition, according to the computer-readable storage medium provided in the third aspect of the present invention, computer instructions are stored thereon. When the computer instructions are executed by the processor, the following steps are implemented: obtaining the user's motor imagery signal via the brain-computer interface; assisting the user's affected limb to perform corresponding movements according to the motor imagery signal via the exoskeleton device; acquiring the first motion image of the affected limb via the image acquisition device; and processing the first motion image according to the first motion data of the affected limb via the mixed reality display device to generate a second motion image of the healthy limb, and displaying the second motion image to the user, so as to perform a bidirectional closed-loop re-learning of the user from the central nervous system to the peripheral nerves. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

[0020] Figure 1 A schematic flow chart of an assisted rehabilitation system provided according to some embodiments of the present invention is shown.

[0021] Figure 2A to Figure 2D A schematic diagram of feature extraction results of CSP under SVM with different γ values ​​provided according to some embodiments of the present invention is shown.

[0022] Figure 3 A schematic diagram of a process of mixed reality display provided according to some embodiments of the present invention is shown.

[0023] Figure 4 A flow chart of a method for bidirectional closed-loop relearning provided according to some embodiments of the present invention is shown.

[0024] Figure 5A schematic structural diagram of an MR audio-visual feedback module provided according to some embodiments of the present invention is shown. DETAILED DESCRIPTION

[0025] The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.

[0026] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0027] In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0028] It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below may be referred to as a second component, region, layer and / or part without departing from some embodiments of the present invention.

[0029] As mentioned above, in stroke rehabilitation, traditional motor imagery combined with the application of brain-computer interfaces can capture brain signals and convert them into control commands, but lacks the support of mixed reality technology, making it difficult to adjust training programs in a timely manner based on real-time monitoring data and provide personalized feedback and guidance. Therefore, the traditional motor imagery combined with the application of brain-computer interfaces often lacks an immersive experience, and patients may lack enthusiasm and motivation for training, reducing participation and training effects. At the same time, the rehabilitation needs and abilities of each stroke patient are different, so the lack of a customized training program may limit the effectiveness of rehabilitation.

[0030] In order to overcome the above-mentioned defects of the prior art, the present invention provides an auxiliary rehabilitation technology for an affected limb, a computer program product, and a computer-readable storage medium, which are used to provide the user with real-time feedback of a second motion image of the healthy limb while assisting the user in exercising, thereby improving the rehabilitation effect of the affected limb.

[0031] In some non-limiting embodiments, the auxiliary rehabilitation system for affected limbs provided in the first aspect of the present invention can be implemented by configuring the computer program product provided in the second aspect of the present invention and the computer-readable storage medium provided in the third aspect of the present invention.

[0032] The core of this personalized rehabilitation training system lies in evaluation and diagnosis, rehabilitation goal setting, training content customization, and real-time feedback and adjustment. Specifically. First, the patient's motor ability and brain activity are preliminarily evaluated using brain-computer interfaces and motion capture devices to determine the patient's rehabilitation needs and current functional status. Combined with medical imaging and clinical evaluation, the patient's hand dysfunction and nerve damage are analyzed in detail to provide basic data for the formulation of personalized rehabilitation plans. Secondly, short-term rehabilitation goals that are easy to achieve are set to help patients gradually restore hand function and improve their confidence in rehabilitation. According to the patient's overall rehabilitation process, long-term rehabilitation goals are formulated, covering fine hand movements and overall motor coordination. Thirdly, task-oriented training content is customized, and training tasks related to the patient's daily life, such as grasping objects, rotating bottle caps, etc., are designed to enhance the patient's motivation and participation. According to the patient's rehabilitation progress, the difficulty and complexity of the training are gradually increased to ensure the effectiveness and sustainability of the rehabilitation training. Finally, the exoskeleton device and motion capture system are used to monitor the patient's motor performance and rehabilitation progress in real time, and data is collected for analysis. According to the real-time monitoring data, the training plan is adjusted in time, and personalized feedback and guidance are provided to ensure that the patient obtains the best rehabilitation effect.

[0033] The present invention combines exoskeleton equipment and mixed reality technology to create a mixed reality rehabilitation system. By hiding the exoskeleton equipment and presenting real hand image information in a virtual environment, the patient's rehabilitation experience is improved. The system combines technology with clinical needs to provide a more attractive and effective rehabilitation training method for stroke patients. It uses mixed reality technology combined with video removal and repair algorithms to provide a more realistic and effective rehabilitation experience. The system cooperates with traditional brain-computer interface rehabilitation courses to provide new ideas and methods for stroke rehabilitation training, and is expected to achieve good results in clinical applications.

[0034] Please refer to Figure 1 , Figure 1 A schematic flow chart of an assisted rehabilitation system provided according to some embodiments of the present invention is shown.

[0035] like Figure 1 As shown, the auxiliary rehabilitation system for the affected limb includes a brain-computer interface, an exoskeleton device, an image acquisition device and a motion capture device. The brain-computer interface is used to obtain the user's motor imagination signal. The exoskeleton device is used to assist the user's affected limb to perform corresponding movements according to the motor imagination signal; the image acquisition device is used to capture the first motion image of the affected limb; the mixed reality display device is used to process the first motion image according to the first motion data of the affected limb to generate a second motion image of the healthy limb, and display the second motion image to the user, so as to perform a two-way closed-loop re-learning from the central nervous system to the peripheral nerves for the user.

[0036] Preferably, the brain-computer interface is used to obtain the user's motor imagery signal. Feature extraction is a key step in the field of brain-computer interface, and the common spatial pattern algorithm is one of the commonly used feature extraction methods. The principle of this algorithm is quite clever: it focuses on solving the problem of spatial filtering feature extraction under two-classification tasks. In this process, it can extract the spatial distribution components of each category from multi-channel brain-computer interface data. The core idea is to use the diagonalization of the matrix to design the spatial filter, and to maximize the difference in variance values ​​of the two types of signals by finding a set of optimal spatial filters for projection. In this way, we can obtain feature vectors with high discrimination. Sending these feature vectors to the classifier for classification can help us distinguish different categories of signals more accurately. In this way, the common spatial pattern algorithm provides us with an effective means to process brain-computer interface data and provides strong support for subsequent classification tasks.

[0037] Furthermore, in this algorithm, it is assumed that there are two classification imagined movement tasks, corresponding to the spatiotemporal signal matrices x and x of multi-channel evoked responses. The dimensions of these two matrices are both N*T, where N represents the number of EEG channels and T represents the number of samples collected for each channel. In order to calculate their covariance matrices, we assume that a mathematical model of a composite source is used to describe the EEG signal and ignore the influence of noise.

[0038]

[0039] In formula (1): x and x can be expressed as shown in formula (1), where S and S represent two types of tasks, and S_M represents the common source signal under these two tasks. It is assumed that these source signals are linearly independent of each other, and C and C are composed of common spatial patterns related to S and S. These spatial patterns are N-dimensional vectors, which are used to represent the distribution weights of a single source signal on N leads.

[0040] The goal of the CSP algorithm is to design spatial filters F and F to obtain the spatial factor W. Then, through the normalized covariance matrices R and R, we can obtain the mixed spatial covariance matrix R:

[0041]

[0042] In formula (2): X T represents the transpose of the X matrix, and trace(X) represents the sum of the elements on the diagonal of the matrix. Then find the mixed space covariance matrix R:

[0043]

[0044] In formula (3): are the average covariance matrices of Task 1 and 2 experiments respectively.

[0045] The eigenvalue decomposition of the mixed space covariance matrix R is performed according to the formula:

[0046] R=UλU T (4)

[0047] In formula (4), U is the eigenvector matrix of matrix λ, and λ is the diagonal matrix of the corresponding eigenvalues. The eigenvalues ​​are arranged in descending order, and the whitened value matrix is:

[0048]

[0049] Then do principal component decomposition on S1 and S2 to get:

[0050] S1=PR1P T , S2=PR2P T (6)

[0051]

[0052] By observing formula (6) and formula (7), it can be proved that the eigenvector matrix of matrix S1 is equal to the eigenvector matrix of matrix S2. At the same time, the sum of the diagonal matrices λ1 and λ2 of the two eigenvalues ​​is the unit matrix, that is:

[0053] B1=B2=V (8)

[0054] λ1+λ1=I (9)

[0055] Since the sum of the eigenvalues ​​of the two types of matrices is always 1, the eigenvector corresponding to the maximum eigenvalue of S1 makes S2 have the smallest eigenvalue, and vice versa. If the eigenvalues ​​in λ1 are arranged in descending order, the corresponding eigenvalues ​​in λ2 are arranged in ascending order. Based on this, it can be inferred that λ1 and λ2 have the following form:

[0056] Since the sum of the eigenvalues ​​of the two types of matrices is always 1, the eigenvector corresponding to the maximum eigenvalue of S1 makes S2 have the smallest eigenvalue, and vice versa. Arranging the eigenvalues ​​in λ1 in descending order, it can be inferred that the eigenvalues ​​corresponding to λ2 are arranged in ascending order. Based on this, we can get λ1 and λ2 to have a specific form, as shown in formula (10).

[0057] λ1=diag(I1σ M 0),λ2=diag(0σ M I2) (10)

[0058] The transformation of the whitening matrix to the eigenvector corresponding to the largest eigenvalue in λ1 and λ2 is optimal for separating the variances in the two signal matrices. Therefore, the projection matrix W is the required spatial filter, which is calculated as shown in formula (11).

[0059] W=B T P (11)

[0060] The motion imagery matrix of the training set is filtered through the corresponding filter W to obtain the feature Z L , Z R for:

[0061] Z L =W×X L

[0062] Z R =W×X R (12)

[0063] According to the definition of CSP algorithm in multi-electrode EEG signal feature extraction, this study selected f L and f RThe feature vectors for imagined left and imagined right are defined as follows:

[0064]

[0065] For the test data X i For example, its eigenvector f i The extraction method is as follows, and with f L and f R A comparison is made to determine whether the i-th image is to the left or to the right.

[0066]

[0067] Finally, find the test f i features for subsequent classification.

[0068] Furthermore, in EEG feature classification, support vector machine (SVM) classifier is a commonly used machine learning method. The basic idea of ​​SVM is to find a separating hyperplane that can divide the training data set in the feature space and maximize the interval. For a linearly separable data set, such a hyperplane is unique, although there are infinitely many separating hyperplanes (i.e., perceptrons). Suppose a training data set in a feature space is given, which contains m samples (feature vectors), represented by T = {(x1, y1), (x2, y2), ..., (x N ,y N )}in, y i ∈{+1,-1},i=1,2,...N,x i is the i-th eigenvector, y i is the class label. When it is equal to +1, it is a positive example; when it is -1, it is a negative example. Assuming that the training data set is linearly separable, the minimum geometric interval of the hyperplane about all sample points is γ. In fact, this distance is what we call the distance from the support vector to the hyperplane. According to the above definition, the solution of the SVM model can be expressed as the following constrained optimization problem:

[0069] max w，b γ

[0070]

[0071] Dividing both sides of the constraint by γ, we get Because maximizing γ is equivalent to maximizing This is equivalent to minimizing 1 / 2 is for the sake of simplicity in the subsequent derivation and does not affect the results. Therefore, the problem of solving the maximum segmentation hyperplane of the SVM model can be expressed as the following constrained optimization problem:

[0072]

[0073] sty i (w·x i +b)≥1,i=1,2,...,N (16)

[0074] First, we transform the original objective function with constraints into the newly constructed Lagrangian objective function without constraints:

[0075]

[0076] Among them, α i is a Lagrange multiplier, and α i ≥ 0. Now we let When the sample point does not meet the constraints, that is, it is outside the feasible solution area: y i (w·x i +b)<1. At this time, α i If θ(w) is set to infinity, then θ(w) is also infinite. When the full point satisfies the constraints, that is, it is in the feasible solution region: i (w·x i +b)≥1. At this time, θ(w) is the original function itself. Therefore, combining the two cases can give us our new objective function:

[0077]

[0078] So the original constraint problem is equivalent to:

[0079]

[0080] First find the maximum value, then the minimum value. In this case, we first have to face the equation with the parameters w and b that need to be solved, and α i This is another inequality constraint, and the solution process is not easy. Therefore, we need to use the duality of the Lagrangian function and swap the positions of the minimum and maximum, so that it becomes:

[0081]

[0082] To have p * =d * , two conditions need to be met: 1. The optimization problem is a convex optimization problem. 2. The KKT condition is met. The KKT condition is as follows:

[0083]

[0084] In order to obtain the specific form of solving the dual problem, let the partial derivative of L(w, b, a) with respect to w and b be 0, we can get and Substituting the above two equations into the Lagrangian objective function and eliminating w and b, we get:

[0085]

[0086] Add a minus sign to the target equation to convert the maximum solution to the minimum solution:

[0087]

[0088] We can get α through the optimization algorithm * , and then according to α * , we can solve for w and b, and then achieve our original goal: to find the hyperplane, that is, the "decision plane".

[0089] Please refer to Table 1 and Figure 2A to Figure 2D ,Table 1 shows the process of SVM algorithm. Figure 2A to Figure 2D A schematic diagram of feature extraction results of CSP under SVM with different γ values ​​provided according to some embodiments of the present invention is shown.

[0090] Table 1 SVM algorithm process

[0091]

[0092] like Figure 2A to Figure 2D As shown in the figure, the CSP feature extraction algorithm has excellent linear characteristics of the hyperplane under SVM. In the case of high γ values, although the dividing line does not converge, the distribution is more reasonable and the classification accuracy is higher.

[0093] Preferably, the exoskeleton device is used to assist the user's affected limb to perform corresponding movements according to the motor imagery signal. The exoskeleton device can generate control instructions to assist the user's affected limb to perform corresponding movements according to the motor imagery signal and the first motion data collected by the motion capture device. Through closed-loop control, the error between the control amount of the exoskeleton device and the actual measurement is reduced to improve the accuracy of the affected limb's movement, thereby improving the rehabilitation effect of the affected limb from the peripheral nervous system end. The image acquisition device is used to capture the first motion image of the affected limb.

[0094] Preferably, the mixed reality display device is used to process the first motion image according to the first motion data of the affected limb to generate a second motion image of the healthy limb, and display the second motion image to the user to perform bidirectional closed-loop re-learning from the central nervous system to the peripheral nerves for the user.

[0095] Please refer to Figure 3 , Figure 3 A schematic diagram of a process of mixed reality display provided according to some embodiments of the present invention is shown.

[0096] like Figure 3As shown, the mixed reality display device may include a YOLOv8 module and a video restoration module. The YOLOv8 module is used to track the position of the exoskeleton device in the first motion image to determine the first motion data of the affected limb. The video restoration module may first determine the target area of ​​the video restoration based on the first motion data, and then generate a filled image of the healthy limb in the target area through optical flow calculation, feature propagation and content generation algorithms, and finally remove the original image of the target area from the first motion image, and fill in the filled image to generate a second motion image.

[0097] Optionally, the motion data of the affected limb can also be directly determined based on the control amount of the exoskeleton device, thereby obtaining the motion data more directly.

[0098] Furthermore, the core of this two-way closed-loop relearning is to establish an effective communication channel between the central nervous system (CNS) and the peripheral nervous system (PNS), and promote neuroplasticity and functional recovery through continuous training and feedback. Here, the central nervous system mainly refers to the brain and spinal cord, and the brain's motor intention is monitored through the brain-computer interface. The peripheral nervous system mainly refers to the nerves extending from the spinal cord to the limbs, and the movement status of the limbs is fed back through exoskeleton equipment and sensors.

[0099] Please refer to Figure 4-5 , Figure 4 A flow chart of a method for bidirectional closed-loop relearning provided according to some embodiments of the present invention is shown. Figure 5 A schematic structural diagram of an MR audio-visual feedback module provided according to some embodiments of the present invention is shown.

[0100] like Figure 4-5 As shown, the auxiliary rehabilitation system may first perform step S1: capturing and transmitting movement intention.

[0101] Specifically, the brain-computer interface (BCI) uses EEG and other technologies to capture the movement intention signals in the patient's brain in real time. These signals represent the hand or limb movements that the patient wants to make. Then, intention decoding is performed, and the captured movement intention signals are decoded into specific movement instructions through algorithms and transmitted to the exoskeleton device.

[0102] Afterwards, the assisted rehabilitation system may proceed to step S2: exoskeleton device execution.

[0103] Specifically, after the exoskeleton device receives the movement command, it performs corresponding movements according to the command to help the patient complete the movements of the hands or limbs. In the early stage of rehabilitation, the exoskeleton device provides a large auxiliary force to help the patient complete basic movements; in the middle and late stages of rehabilitation, the auxiliary force is gradually reduced to enhance the patient's autonomous control ability.

[0104] Afterwards, the assisted rehabilitation system may proceed to step S3: motion capture and feedback.

[0105] Specifically, the motion capture system captures the actual motion data of the patient's hands and limbs in real time, including motion trajectory, speed, angle, etc. The feedback signal transmits the motion capture data back to the brain-computer interface and exoskeleton control system to achieve real-time feedback, adjust the motion instructions, and ensure the accuracy and consistency of the movement.

[0106] Finally, the assisted rehabilitation system can proceed to step S4: video restoration and visual feedback.

[0107] Specifically, the assisted rehabilitation system can first use YOLOv8 to track the exoskeleton device in real time in the video stream to ensure that the target area of ​​the video restoration is accurate. Then, the E2FGVI algorithm captures the motion vector of each pixel in the video frame through optical flow calculation, estimates the motion between video frames, and ensures the coherence of motion during the video restoration process. Based on the optical flow calculation, the feature propagation module propagates the feature information in the time sequence to ensure the temporal consistency during the video restoration process. Feature propagation helps to maintain the consistency of visual effects between multiple video frames and avoid the incoherence of the restored video. Finally, the content generation module generates the content of the missing area after removing the exoskeleton device based on the context information. This module uses deep learning to generate models, fills the blank areas in the video through context understanding and content generation, and ensures that the generated video picture is natural and seamless. The restored video stream is transmitted to the mixed reality device. After the video restoration is completed, the generated video stream no longer contains the image of the exoskeleton device, showing the effect of the natural movement of the patient's hand.

[0108] Here, through two-way closed-loop real-time interaction and feedback, the connection between the central nervous system and the peripheral nervous system is constantly stimulated and strengthened, the reconnection of neurons and functional recovery are promoted, and the concentration and effect of rehabilitation training are improved. As the training progresses, the patient gradually recovers the autonomous control ability of the hands and limbs and achieves full functional recovery. At the same time, the repaired video stream is fed back to the QUEST3 device in real time, and the patient sees the natural movement image of his own hand in the virtual environment, rather than the image of the mechanical exoskeleton. This visual effect allows patients to experience real control, effectively calms patients' emotions, enhances the realism and effect of the rehabilitation experience, and enhances the attractiveness of training.

[0109] Preferably, the motion capture device is used to collect the first motion data of the affected limb, wherein the motion capture device includes multiple sensors, the multiple sensors are selected from at least one of an inertial sensor, an optical sensor, a magnetic sensor, and an ultrasonic sensor, which are arranged at multiple key joints of the affected limb to capture the second motion data of each key joint in real time to form the first motion data of the affected limb. Through closed-loop control, the error between the control amount of the exoskeleton device and the actual measurement is reduced to improve the accuracy of the second motion image, thereby improving the rehabilitation effect of the affected limb from the central nervous system end.

[0110] Furthermore, the motion capture device first uses sensor technology to collect the first motion data. Specifically, the motion capture device is usually composed of multiple sensors, which can be inertial sensors (IMU), optical sensors, magnetic sensors or ultrasonic sensors. The sensors are installed at the patient's key joints and limbs to capture their motion data in real time. Here, inertial sensors may include accelerometers, gyroscopes and magnetometers, which can measure the acceleration, angular velocity and magnetic field changes of objects and provide posture (attitude and position) data. Optical sensors use cameras and markers (such as reflective balls or LED lights) to obtain accurate motion trajectories by calculating the position changes of markers. Magnetic sensors and ultrasonic sensors determine the spatial position and motion trajectory of the sensor by measuring changes in magnetic fields or sound waves.

[0111] Afterwards, the motion capture device transmits the motion data collected by the sensor to a computer or processing unit in real time. These data usually include joint angles, postures, speeds, accelerations, etc. Through data fusion algorithms, the data from different sensors are processed comprehensively to improve the accuracy and reliability of motion capture.

[0112] Finally, the motion capture system transmits the processed motion data to the rehabilitation system for real-time control of the exoskeleton device and the virtual scene in mixed reality. Patients can see real-time virtual images of their hands and bodies through mixed reality devices (such as QUEST3), thus gaining a more intuitive rehabilitation training experience.

[0113] Motion capture devices can accurately capture the patient's hand and body movement data, reflecting their movement status and movement trajectory in real time. These data are crucial for evaluating the patient's rehabilitation progress and adjusting the training plan. Through the motion data obtained by the motion capture device, the exoskeleton device can accurately perform the patient's intended movements. For example, when the patient tries to grasp, the motion capture device can capture the subtle movements of the hand and transmit them to the exoskeleton to help the patient complete the grasping action. The real-time motion data collected by the motion capture device can be used for rendering and interaction of mixed reality scenes. For example, when the patient is doing hand rehabilitation training in a virtual environment, he can see the virtual image of his hand, which is generated based on the real-time data of the motion capture device, thus providing a more realistic interactive experience. The motion data collected by the motion capture device can be used for detailed rehabilitation analysis and evaluation. The system can analyze the patient's movement trajectory, speed, angle and other parameters, generate a rehabilitation report, and help doctors and patients understand the rehabilitation progress and adjust the training plan. Motion capture data can also be used to identify the patient's movement patterns and potential problems, provide personalized rehabilitation advice and feedback, and optimize the rehabilitation effect.

[0114] When conducting user testing on the system, especially for stroke rehabilitation patients, in order to evaluate its effectiveness and user experience in the actual rehabilitation process, we made a lot of comfort adjustments based on the feedback collected. For example, based on test feedback, we optimized the comfort of the QUEST3 device and the quality of the video stream to ensure that patients feel comfortable during use and the visual experience is as realistic as possible; taking into account the different needs and rehabilitation stages of patients, we adjusted the auxiliary strength of the exoskeleton device and the details of the video restoration in real time; at the same time, when designing the system, in order to protect the security and privacy of patients' video data, the system implemented basic data encryption and anonymization measures, and only authorized medical personnel can access sensitive information; after the system is deployed, it can continuously monitor its performance and the patient's rehabilitation progress and use data analysis tools to evaluate the rehabilitation effect, and promptly identify and solve possible problems.

[0115] In addition, the assisted rehabilitation system may also include a controller and a human-computer interaction interface. The controller may be used to determine the corresponding motion scene, target action, and / or the auxiliary force of the exoskeleton device according to the rehabilitation stage of the affected limb. The user instruction is obtained via the human-computer interaction interface; and according to the user instruction, the magnitude of the auxiliary force provided by the exoskeleton device to the affected limb and / or the display frame rate and / or resolution of the mixed reality display device are adjusted.

[0116] The brain-computer interface rehabilitation system combined with MR technology can also build a more portable and flexible rehabilitation training environment. Patients can wear MR headsets for rehabilitation training, and perform standing, sitting or lying training according to the needs of different rehabilitation stages, which improves the portability and flexibility of training and thus improves the rehabilitation effect. Such MR-MI-BCI rehabilitation training system provides a new model for the rehabilitation process, which can help stroke patients achieve the transition from passive limb to active neural rehabilitation, bringing innovation to the field of clinical rehabilitation.

[0117] The working principle of the above-mentioned auxiliary rehabilitation system will be described below in conjunction with some embodiments in which the affected limb is a hand. Those skilled in the art will understand that these embodiments in which the affected limb is a hand are only some non-limiting implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working modes of the auxiliary rehabilitation system. Similarly, the embodiments in which the affected limb is a hand are also only some non-limiting implementation methods provided by the present invention, and do not constitute a limitation on the execution subject or execution order of each step in these processors.

[0118] Specifically, the affected limb is the hand, which may include disabilities of hand muscles and / or hand nerves caused by physical injuries, as well as hand function disabilities caused by stroke. In a specific rehabilitation training, motion capture devices can be installed on the patient's wrists, fingers, and arms. When the patient performs grasping training, the motion capture device captures the motion data of each joint of the hand, and converts this data into control signals of the exoskeleton device through the data processing module to help the patient complete the grasping movement. At the same time, this data is also transmitted to the mixed reality system, and the patient can see the real-time virtual image of his hand in the QUEST3 device, further enhancing the immersion and effectiveness of the rehabilitation training.

[0119] The controller can first determine the rehabilitation stage of the affected limb. In response to the affected limb being in the first rehabilitation stage, a first motion scene of grasping at least one first target object is displayed via a mixed reality display device, and a larger first torque auxiliary force is provided to the affected limb according to the user's motion imagination signal via an exoskeleton device to help the user complete the first target action of grasping. In response to the affected limb being in the second rehabilitation stage, a second motion scene of rotating and / or picking up at least one second target object is displayed via a mixed reality display device, and a moderate second torque auxiliary force is provided to the affected limb according to the user's motion imagination signal via an exoskeleton device to help the user complete the second target action of rotating and / or picking up. In response to the affected limb being in the third rehabilitation stage, a third motion scene simulating daily life is displayed via a mixed reality display device, such as: putting on and taking off clothes, cooking, washing, cleaning, and writing, and a smaller third torque auxiliary force is provided to the affected limb according to the user's motion imagination signal via an exoskeleton device to help the user complete at least one third target action related to daily life. Here, the actual torque value of each stage is set differently according to different people. The actual value of the first torque is about 2Nm~5Nm; the second torque will be halved in the middle stage; and the third torque in the late stage is about 0.5Nm~1Nm.

[0120] Specifically, the auxiliary rehabilitation system is a two-way closed-loop relearning from the central nervous system to the peripheral nerves. In the initial stage, the brain-computer interface captures the patient's brain's motor intention signal about the right hand grasping. The exoskeleton device receives the motor intention and provides greater auxiliary force to help the patient complete the grasping action. The motion capture system records the hand's grasping trajectory and strength, and feeds the data back to the control system for adjustment. Through video repair, patients can see the image of their right hand naturally grasping in the mixed reality device, which enhances their confidence in training.

[0121] In the mid-term stage, the brain-computer interface continues to capture movement intention signals, decodes them and transmits them to the exoskeleton device. Exoskeleton execution: Reduce the auxiliary strength of the exoskeleton and increase the proportion of patient's autonomous control. Motion capture and feedback: Real-time capture and feedback of patients' movement data to ensure the accuracy of training movements. Visual feedback: Provide a video stream with medium picture quality and frame rate, and patients can perform more complex fine movement training in a virtual environment.

[0122] In the later stage, the brain-computer interface captures detailed movement intentions, such as grasping small objects, pinching thin wires, etc. Exoskeleton execution: The exoskeleton provides minimal auxiliary force and relies more on the patient's autonomous control. Motion capture and feedback: Accurately capture the subtle movements of the hand and provide high-frequency feedback adjustments. Visual feedback: Provide high-quality and high-frame rate video streams to ensure that the patient's visual experience during complex operations is real and natural.

[0123] Furthermore, the auxiliary rehabilitation system can also take into account the display effect and real-time performance under limited data processing capabilities to achieve the best two-way closed-loop re-learning effect. Specifically, in response to the affected limb being in the first rehabilitation stage, the mixed reality display device is controlled to display the first motion scene at a higher first frame rate and a smaller first resolution (for example: 540P, 516×552). In response to the affected limb being in the second rehabilitation stage, the mixed reality display device is controlled to display the second motion scene at a higher second frame rate and a smaller second resolution (for example: 1080P, 1032×1104). In response to the affected limb being in the third rehabilitation stage, the mixed reality display device is controlled to display the third motion scene at a lower third frame rate and a larger third resolution (for example: 2k, 2064×2208).

[0124] Please refer to Table 2, which shows the auxiliary strength and video restoration details of the motion capture device provided in each rehabilitation stage according to some embodiments of the present invention.

[0125] Table 2 Assistance intensity and video restoration details at each rehabilitation stage

[0126]

[0127] As shown in Table 2, in the initial stage of rehabilitation, patients mainly need to restore basic hand motor functions, such as grasping, releasing and other basic movements. At this time, the patient's hand muscle strength and coordination ability are weak, and a larger auxiliary force is required to complete these movements. The torque adjustment of the manipulator is large to ensure that patients can get enough support when performing basic movements, helping them to gradually restore hand strength and control. The refresh rate of video repair is set to 60fps, and a low image quality of 540P is used. This stage mainly focuses on the recovery of motor function, and the visual effect requirements of the video are relatively low. Here, the higher frame rate and lower image quality are more suitable for the basic needs of rehabilitation training to improve processing speed while reducing system load.

[0128] Entering the mid-stage of rehabilitation, patients need to enhance their fine motor skills, such as picking up small objects and twisting bottle caps. At this stage, the patient's hand control ability has improved, but still requires a certain amount of assistance. The torque of the manipulator is adjusted to medium to provide moderate assistance to help patients get the necessary support when performing fine motor training, while gradually reducing their dependence on assistance. The refresh rate of video repair is set to 30fps, and the medium image quality of 1080P is used. Here, compared with the initial stage, the higher video clarity in the mid-stage can ensure that patients can get better visual feedback and experience when performing fine motor training.

[0129] In the later stages of rehabilitation, patients need to restore their ability to take care of themselves in daily life, such as using chopsticks to eat, writing and other complex hand operations. At this stage, the patient's hand strength and coordination are basically restored, and less assistance is required. The torque of the manipulator is adjusted to a smaller level to provide minimal assistance, helping patients to get slight support when performing complex hand operations and further enhance their autonomy. The refresh rate of the video repair is set to 20fps, and 2k high-definition is used. Here, the high-definition video stream can provide more realistic visual effects, thereby helping patients to get a better sense of immersion and reality when performing complex operations, and improving the training effect.

[0130] Therefore, for patients with special needs, such as those who need complex fine operation training, the system can flexibly adjust the auxiliary strength and video restoration details according to the specific situation. Not only can the auxiliary strength of the manipulator be adjusted dynamically. According to the specific needs and real-time feedback of the patient, the torque of the manipulator can be flexibly adjusted to provide the most appropriate auxiliary support, but also the algorithm output quality and algorithm operation frame rate can be dynamically adjusted. According to the specific needs of the patient and the training content, the video refresh rate and image quality can be flexibly adjusted to optimize system performance while ensuring the visual experience.

[0131] Furthermore, after the auxiliary rehabilitation system completes a training session, the controller may generate a rehabilitation report indicating the rehabilitation stage of the affected limb based on first motion data collected by a motion capture device disposed at a plurality of key joints of the affected limb.

[0132] Those skilled in the art will appreciate that the embodiment of the hand-affected limb is only an implementation method of a non-limiting signal provided by the present invention. Optionally, for some rehabilitation training that requires coordinated whole-body movement, the motion capture device can cover the patient's main joints and limbs, such as shoulders, elbows, waist and knees. When the patient performs sports training (such as walking or weightlifting) in a virtual environment, the motion capture device captures the whole-body motion data and feeds it back to the mixed reality system in real time. In this way, patients can not only see their own motion images in the virtual environment, but also obtain auxiliary support from the exoskeleton device to help them complete more complex rehabilitation movements.

[0133] At present, this stroke motor rehabilitation system combining brain-computer interface and exoskeleton manipulator has been maturely applied, mainly for patients whose affected limb motor function is impaired due to stroke. This system has been successful in the rehabilitation process of hundreds of patients, and the relevant motor function indicators have been improved by more than 30% on average.

[0134] Those skilled in the art can take the rehabilitation training of 55-year-old Ms. Zhang as an example. Ms. Zhang suffered a stroke that caused partial loss of function of her right hand and was unable to complete daily grasping movements. The preliminary evaluation of the auxiliary rehabilitation system showed that Ms. Zhang's right hand grip was weak and the fine movements of her fingers were limited. Brain-computer interface analysis showed that the control ability of her brain's motor cortex over her right hand needs to be restored. The short-term goal is to restore basic grasping ability and be able to grasp and lift small objects. The long-term goal is to restore the ability to control fine movements of the right hand, such as using chopsticks to eat.

[0135] On this basis, the assisted rehabilitation system customized the training content for Ms. Zhang: Phase 1: Using mixed reality technology to display virtual objects, Ms. Zhang was asked to try to grasp objects of different sizes and shapes, gradually improving her grasping strength. Phase 2: Increasing the difficulty of training, requiring Ms. Zhang to complete fine movements such as rotating bottle caps and pinching small beads. Phase 3: Simulating daily life scenes, such as using the right hand to perform simple cooking operations in a virtual kitchen.

[0136] After each training session, the system automatically generates a rehabilitation report and analyzes Ms. Zhang’s training data. Based on the data feedback, the system adjusts the training content and difficulty for the next stage and provides personalized guidance.

[0137] Furthermore, the brain-computer interface rehabilitation system combined with MR technology can also build a more portable and flexible rehabilitation training environment. Patients can wear MR headsets for rehabilitation training, and perform standing, sitting or lying training according to the needs of different rehabilitation stages, which improves the portability and flexibility of training and thus improves the rehabilitation effect. Such MR-MI-BCI rehabilitation training system provides a new model for the rehabilitation process, which can help stroke patients achieve the transition from passive limb to active neural rehabilitation, bringing innovation to the field of clinical rehabilitation.

[0138] In summary, this system aims to provide an exoskeleton real hand motion perception system based on mixed reality technology for hand perception rehabilitation of stroke patients. The system uses mixed reality technology combined with exoskeleton equipment, and cooperates with traditional brain-computer interface rehabilitation treatment to provide patients with a more realistic and effective rehabilitation experience. The real-time video stream of the patient wearing the exoskeleton is uploaded to the server through the QUEST3 device, and then the SAM function of YOLOv8 is used to track the selected exoskeleton target and remove the exoskeleton device from the original video. The posture of each part of the exoskeleton is determined by the motion capture device, and the E2FGVI end-to-end video restoration framework is adopted. The core idea of ​​the algorithm includes optical flow calculation, feature propagation and content generation modules, and the real-time video stream is repaired to fill in the blanks or missing areas. The purpose is that the patient can better soothe his mood for rehabilitation exercise by watching the image of his own arm without the robot in the QUEST3 device, and assist him to get a more realistic experience.

[0139] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0140] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An auxiliary rehabilitation system for an affected limb, characterized in that: include: Brain-computer interface, used to obtain the user's motor imagery signals; An exoskeleton device, used to assist the user's affected limb to perform corresponding movements according to the motor imagery signal; An image acquisition device, used for acquiring a first motion image of the affected limb; as well as A mixed reality display device is used to process the first motion image according to the first motion data of the affected limb to generate a second motion image of the healthy limb, and display the second motion image to the user to perform bidirectional closed-loop relearning from the central nervous system to the peripheral nerves of the user.

2. The assisted rehabilitation system according to claim 1, characterized in that: Also includes: A motion capture device is used to collect the first motion data of the affected limb, wherein the motion capture device includes multiple sensors, and the multiple sensors are selected from at least one of inertial sensors, optical sensors, magnetic sensors, and ultrasonic sensors. The sensors are arranged at multiple key joints of the affected limb to capture the second motion data of each of the key joints in real time to form the first motion data of the affected limb.

3. The assisted rehabilitation system according to claim 2, characterized in that: The exoskeleton device is configured to: A control instruction for assisting the user's affected limb to perform corresponding movement is generated based on the motor imagery signal and the first motion data collected by the motion capture device.

4. The assisted rehabilitation system according to claim 1, characterized in that: The mixed reality display device includes a YOLOv8 module for tracking the position of the exoskeleton device in the first motion image to determine the first motion data of the affected limb.

5. The assisted rehabilitation system according to claim 1, 2 or 4, characterized in that: The mixed reality display device includes a video repair module, which is configured as follows: Determining a target area for video restoration according to the first motion data; Generate a filled image of the healthy limb in the target area through optical flow calculation, feature propagation and content generation algorithms; and The original image of the target area is removed from the first motion image, and the filled image is inserted to generate the second motion image.

6. The assisted rehabilitation system according to claim 1, characterized in that: Also includes: The controller is used to determine the corresponding movement scene, target action, and / or the auxiliary force of the exoskeleton device according to the rehabilitation stage of the affected limb.

7. The assisted rehabilitation system according to claim 6, characterized in that: The affected limb is a hand, and the controller is configured as follows: determining the rehabilitation stage of the affected limb; In response to the affected limb being in the first rehabilitation stage, displaying a first motion scene of grasping at least one first target object via the mixed reality display device, and providing a large first torque auxiliary force to the affected limb via the exoskeleton device according to the user's motion imagination signal, so as to help the user complete the first target action of grasping; In response to the affected limb being in the second rehabilitation stage, a second motion scene of rotating and / or picking up at least one second target object is displayed via the mixed reality display device, and a moderate second torque auxiliary force is provided to the affected limb via the exoskeleton device according to the user's motor imagery signal, so as to help the user complete the second target action of rotating and / or picking up; as well as In response to the affected limb being in the third rehabilitation stage, a third motion scene simulating daily life is displayed via the mixed reality display device, and a smaller third torque auxiliary force is provided to the affected limb via the exoskeleton device according to the user's motion imagination signal, so as to help the user complete at least one third target action related to daily life.

8. The assisted rehabilitation system according to claim 7, characterized in that: The controller is also configured to: In response to completing a training session, a rehabilitation report indicating the rehabilitation stage of the affected limb is generated based on first motion data collected by motion capture devices disposed at multiple key joints of the affected limb.

9. The assisted rehabilitation system according to claim 7, characterized in that: The controller is also configured to: In response to the affected limb being in the first rehabilitation stage, controlling the mixed reality display device to display the first motion scene at a higher first frame rate and a lower first resolution; In response to the affected limb being in the second rehabilitation stage, controlling the mixed reality display device to display the second motion scene at a higher second frame rate and a smaller second resolution; as well as In response to the affected limb being in the third rehabilitation stage, the mixed reality display device is controlled to display the third motion scene at a lower third frame rate and a larger third resolution.

10. The assisted rehabilitation system according to claim 9, characterized in that: The auxiliary rehabilitation system also includes a human-computer interaction interface, and the controller is further configured to: Obtaining user instructions via the human-computer interaction interface; and According to the user instruction, the magnitude of the assistive force provided by the exoskeleton device to the affected limb and / or the display frame rate and / or resolution of the mixed reality display device are adjusted.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the following steps are implemented: Obtain the user's motor imagination signal via the brain-computer interface; Assisting the user's affected limb to perform corresponding movements according to the motor imagery signal via the exoskeleton device; Acquiring a first motion image of the affected limb via an image acquisition device; as well as Via a mixed reality display device, the first motion image is processed according to the first motion data of the affected limb to generate a second motion image of the healthy limb, and the second motion image is displayed to the user to perform bidirectional closed-loop re-learning from the central nervous system to the peripheral nerves for the user.

12. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the following steps are implemented: Obtain the user's motor imagination signal via the brain-computer interface; Assisting the user's affected limb to perform corresponding movements according to the motor imagery signal via the exoskeleton device; Capturing a first motion image of the affected limb via an image acquisition device; as well as Via a mixed reality display device, the first motion image is processed according to the first motion data of the affected limb to generate a second motion image of the healthy limb, and the second motion image is displayed to the user to perform bidirectional closed-loop re-learning from the central nervous system to the peripheral nerves for the user.

Images