Virtual-real fusion hand function rehabilitation training method and system for autistic children

Through the integration of virtual and real technology and flexible rehabilitation gloves, combined with voice recognition and tactile feedback, and adaptively adjust the training content, the problem of insufficient tactile feedback and emotional attention in hand function rehabilitation training for autistic children is solved, and the training effect and fun are improved, achieving Independent recovery and quantitative assessment of children with autism.

CN118969203AActive Publication Date: 2024-11-15SHANDONG UNIV +2
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411440477.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-16
Publication Date
2024-11-15
Estimated Expiration
2044-10-16

AI Technical Summary

Technical Problem

The existing hand function rehabilitation training methods and systems for children with autism have insufficient requirements in stimulating the hand nerves to generate tactile feedback and adjusting the training content, resulting in poor training results and lack of attention to emotional and social skills.

Method used

Using the method of fusion of virtual and real, the training scenario is constructed through augmented reality technology, combined with flexible rehabilitation gloves and speech recognition technology, brain function, behavior and subjective scale data are collected in real time, and the difficulty and content of the training task are adjusted adaptively, and tactile feedback is provided. and emotional perception improve training effect.

Benefits of technology

It has achieved independent hand function rehabilitation training for children with autism, improved the accuracy of hand movement and nerve regeneration, enhanced the fun and effect of training, quantitatively evaluated rehabilitation progress, and reduced dependence on medical staff.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118969203A_ABST
    Figure CN118969203A_ABST
Patent Text Reader

Abstract

The invention discloses a virtual-real fusion autism child hand function rehabilitation training method and system, and relates to the technical field of child autism rehabilitation training.The method comprises the steps that motion information and a virtual scene corresponding to a current rehabilitation training task are mapped into an augmented reality scene, and an autism child is guided to interact with the augmented reality scene; based on the worn flexible rehabilitation gloves, the children with autism are assisted in completing multiple rounds of current rehabilitation training tasks; in the non-auxiliary state, collecting brain function data, behavioral data and subjective scale data when the autistic child executes the current rehabilitation training task, and performing comprehensive evaluation based on the collected data to obtain a rehabilitation training evaluation result; according to the rehabilitation training evaluation result, the difficulty level of the current rehabilitation training task is adjusted in a self-adaptive mode, and rehabilitation training of the next round is carried out. The difficulty and content of the training task can be flexibly adjusted, a patient is assisted in rehabilitation training by himself, and the rehabilitation effect of the hand function of the autism child is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation training for children with autism, and in particular to a virtual-real integration hand function rehabilitation training method and system for children with autism. Background Art

[0002] Autism Spectrum Disorder (ASD), also known as autism or autism, has an increasing incidence rate year by year. Autism itself has no drug cure and can only be treated with long-term educational intervention. Autism is a pervasive neurodevelopmental disorder that occurs in early childhood development. Fine motor skills refer to the movements of small muscles or small muscle groups in the body. These movements usually require a high degree of coordination and control, and require accurate control of the strength and position of the hands and fingers. The hand grasping ability and fine motor flexibility of autistic children are much lower than those of children of the same age, and hand-eye coordination disorders are common. At present, motor intervention for autistic children has gradually emerged in all walks of life, but most studies focus on improving the problem of poor gross motor skills in autistic children, while there are few special intervention studies on fine motor development disorders. Therefore, it is particularly important to pay attention to the development of fine motor skills in the hands of autistic children and formulate corresponding intervention and training strategies.

[0003] Educational intervention is a traditional rehabilitation training method, which is mostly to train autistic patients through targeted verbal guidance, by breaking down the task movements for demonstration, and providing hand-on assistance, following the principle of easy to difficult, from simple to complex, including grasping, pinching, coordination, skills, etc. However, this method has little effect on the improvement of autistic patients' perception, movement, language and cognitive skills, and the improvement of adaptive behaviors including communication and daily life is also minimal. These educational intervention rehabilitation methods are not very applicable to autistic children.

[0004] The emergence of rehabilitation gloves makes up for the defects of traditional educational rehabilitation treatment programs and is more effective in the rehabilitation, treatment and education of hand function for patients with autism spectrum disorders. Rehabilitation gloves can be divided into rigid rehabilitation gloves and flexible rehabilitation gloves. They drive the patient's hand movements through rigid exoskeletons or flexible materials, thereby replacing medical staff to assist patients in hand rehabilitation exercises, and provide long-term and precise movement stimulation, which helps to restore hand function, reduce medical costs and expand treatment options. However, due to the lack of tactile sensors, rehabilitation gloves cannot stimulate the patient's hand nerves to produce tactile feedback, and due to technical and material limitations and sensor accuracy issues, their training content is boring and the application scenarios are single. It is difficult to accurately obtain the patient's hand posture in real time to judge the recovery level and adjust the rehabilitation plan, which seriously affects the patient's treatment enthusiasm and effect, and limits its application and development in the field of rehabilitation medicine. In addition, current research focuses on hand function rehabilitation training, but ignores the improvement of emotions or social skills, and lacks attention to the psychology of autistic children.

[0005] At present, the use of augmented reality (AR) technology for game interaction is a development trend that is more convenient for the rehabilitation of autistic children. This kind of game interaction is diverse and motivating, which can more easily enhance the rehabilitation interest of autistic children, thereby minimizing anxiety and boredom during the task, so that participants can maintain a high level of participation and learning efficiency. However, the first step in hand function treatment is to establish the correct movement pattern of the hand, good posture control, and improvement of residual function. At present, the rehabilitation of hand function through augmented reality technology requires the assistance of medical staff. It is difficult for autistic children to establish the correct movement pattern and good posture control of the hand alone, and it is difficult to complete the rehabilitation training of hand function. Summary of the invention

[0006] In order to address the deficiencies of the above-mentioned prior art, the present invention provides a virtual-real integration hand function rehabilitation training method and system for autistic children, which constructs a virtual-real integration training scene through augmented reality technology to expand the training scene, and identifies hand postures to obtain joint position information data, which is combined with brain function data and subjective scale data and processed as a quantitative evaluation indicator of rehabilitation effect. On this basis, combined with the set training tasks for hand function rehabilitation, the difficulty and content of the training tasks can be flexibly adjusted to assist patients in conducting rehabilitation training on their own. In addition, voice recognition technology is used to understand and respond to language commands to achieve communication and interaction, while capturing changes in facial expressions and sounds to perceive emotional states, and making corresponding responses according to different situations, greatly improving the rehabilitation effect of hand function for autistic children.

[0007] In a first aspect, the present invention provides a virtual-real integration hand function rehabilitation training method for children with autism.

[0008] A virtual and real integrated hand function rehabilitation training method for children with autism, comprising: Map the motion information and virtual scenes corresponding to the current rehabilitation training task to the augmented reality scene, and guide autistic children to interact with the augmented reality scene; Based on the flexible rehabilitation gloves worn, autistic children are assisted to complete multiple rounds of current rehabilitation training tasks; In a non-assisted state, brain function data, behavioral data, and subjective scale data of autistic children are collected when they perform the current rehabilitation training tasks, and a comprehensive evaluation is performed based on the collected data to obtain the rehabilitation training evaluation results; According to the rehabilitation training evaluation results, the difficulty level of the current rehabilitation training task is adaptively adjusted to carry out the next round of rehabilitation training.

[0009] In a second aspect, the present invention provides a virtual-real integration hand function rehabilitation training system for children with autism.

[0010] A virtual-real integration hand function rehabilitation training system for children with autism, comprising: A mapping unit, used to map the motion information and virtual scene corresponding to the current rehabilitation training task to the augmented reality scene, and guide the autistic child to interact with the augmented reality scene; An auxiliary unit, used to assist autistic children to complete multiple rounds of current rehabilitation training tasks based on the flexible rehabilitation gloves they wear; A collection unit is used to collect brain function data, behavioral data and subjective scale data of autistic children when they perform current rehabilitation training tasks in a non-assisted state; An evaluation unit, used to perform a comprehensive evaluation based on the collected data and obtain a rehabilitation training evaluation result; The adjustment unit is used to adaptively adjust the difficulty level of the current rehabilitation training task according to the rehabilitation training evaluation results, and conduct the next round of rehabilitation training.

[0011] One or more of the above technical solutions have the following beneficial effects: 1. The present invention provides a virtual-real integration hand function rehabilitation training method and system for autistic children. The virtual-real integration training scene is constructed through augmented reality technology to expand the training scene, and the hand posture is identified to obtain joint position information data, which is combined with brain function data and subjective scale data. After processing, it is used as a quantitative evaluation indicator of rehabilitation effect. On this basis, combined with the set training tasks for hand function rehabilitation, the difficulty and content of the training tasks are flexibly adjusted to assist patients in conducting rehabilitation training on their own. In addition, voice recognition technology is used to understand and respond to language commands to achieve communication and interaction, and changes in facial expressions and sounds are captured to perceive emotional states, and corresponding responses are made according to different situations, greatly improving the rehabilitation effect of hand function in autistic children.

[0012] 2. In the present invention, rehabilitation gloves are used to fully exercise the hand muscles and joints, establish the correct corresponding mode of the hand, and provide personalized tactile feedback to stimulate the hand nerves; sports training based on augmented reality helps to promote nerve regeneration and reorganization in the patient's brain area, and through repeated training of movement, strengthens the connection and information transmission ability of damaged neurons, and promotes the patient's hand muscle coordination and movement accuracy; the present invention uses augmented reality technology to combine virtual information with the real world, giving patients real visual and auditory feedback, combined with the tactile feedback of gloves, to achieve an immersive experience, alleviate sensory integration disorders, and improve the hand function rehabilitation effect. In addition, by designing the gamification interface and tasks of rehabilitation training, the interest and enthusiasm of autistic children are stimulated, the fun of training is increased, and the efficiency and effect of hand function rehabilitation are improved.

[0013] 3. The present invention provides a quantitative evaluation index of rehabilitation effect and a personalized rehabilitation plan. During the augmented reality interaction process, the user's hand movements are tracked and the hand postures are identified to obtain joint position information, such as finger bending angle, finger movement trajectory, time taken to complete the task, etc. After processing and analysis, it is used as a quantitative evaluation index of rehabilitation effect. The difficulty and training content can be flexibly adjusted according to the rehabilitation effect evaluation results to meet the needs of different patients.

[0014] 4. In the present invention, rehabilitation gloves are used to replace medical staff to assist patients in completing repetitive hand joint movements. Different control modes are used to drive the hand joints to complete different movements, and the patient's tactile perception is fully stimulated. Through augmented reality technology, the one-to-one rehabilitation training mode between medical staff and patients can be changed to one-to-many, and even completely get rid of dependence on medical staff and realize autonomous rehabilitation training. It can also achieve the goal of patients' home rehabilitation and remote guidance by medical staff, effectively alleviating the pressure of shortage of medical resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0016] Figure 1 This is an overall flow chart of the virtual-real integration hand function rehabilitation training method for children with autism described in an embodiment of the present invention; Figure 2 is a schematic diagram of different modes of rehabilitation training tasks in an embodiment of the present invention; Figure 3 This is a schematic diagram of a virtual-real integration hand function rehabilitation training system for children with autism according to an embodiment of the present invention; Figure 4 is a schematic diagram of a flexible rehabilitation glove used in an embodiment of the present invention; Figure 5 Schematic diagram of feedback pixels of the tactile feedback system of the flexible rehabilitation glove used in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] It should be noted that the following detailed descriptions are exemplary only, are intended to describe specific embodiments, are intended to provide further explanation of the present invention, and are not intended to limit exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0018] Embodiment 1 This embodiment provides a virtual-real fusion hand function rehabilitation training method for autistic children. Figure 1 As shown, the following steps are included: Step S1, mapping the motion information and virtual scene corresponding to the current rehabilitation training task to the augmented reality scene, and guiding the autistic child to interact with the augmented reality scene; Step S2: assisting the autistic child to complete multiple rounds of current rehabilitation training tasks based on the worn flexible rehabilitation gloves; Step S3: In a non-assisted state, collecting brain function data, behavioral data, and subjective scale data of the autistic child when performing the current rehabilitation training task, performing a comprehensive evaluation based on the collected data, and obtaining a rehabilitation training evaluation result; Step S4: According to the rehabilitation training evaluation result, the difficulty level of the current rehabilitation training task is adaptively adjusted to carry out the next round of rehabilitation training.

[0019] In the above step S1, the motion information and virtual scene corresponding to the current rehabilitation training task are mapped to the augmented reality scene, so that the subject (i.e., the autistic child being tested) can obtain the gesture type and motion information of the current rehabilitation training task in advance through the augmented reality device before each round of rehabilitation training task begins. The motion information includes the movement speed, movement trajectory, and task difficulty of the rehabilitation training. On this basis, through the rehabilitation training scene that integrates virtuality and reality, the autistic child is guided to interact with the augmented reality scene to conduct training in hearing, vision, basic sensation, balance, spatial perception, etc.

[0020] In the above step S2, the subject wears a flexible rehabilitation glove and turns on the flexible rehabilitation glove to assist the subject in passively completing the rehabilitation training task. The glove is activated according to the interactive information in the augmented reality scene to drive the subject's different fingers to perform bending / stretching movements, thereby assisting the subject in completing the corresponding gestures.

[0021] In the above step S3, in a non-assisted state, the brain function data, behavioral data and subjective scale data of the autistic child when performing the current rehabilitation training task are collected, and a comprehensive evaluation is performed based on the collected data to obtain the rehabilitation training evaluation results.

[0022] In this embodiment, after the subject (or user) completes each round of rehabilitation training tasks, the collected brain function data during the task is compared and analyzed with the subject's resting state (or resting state) brain function data to obtain a brain function evaluation result, which is then combined with the subject's behavioral data and subjective scale data for a comprehensive evaluation to obtain a final comprehensive evaluation result, which is reflected in the form of a score.

[0023] Specifically, first, based on the collected brain function data and the pre-collected resting-state brain function data, the activation degree of different brain regions is obtained to obtain the brain function evaluation result; then, based on the collected behavioral data, the hand movement score is calculated to obtain the behavioral data evaluation result; secondly, based on the obtained subjective scale data, the training performance results are quantified to obtain the subjective scale data evaluation result; finally, based on the brain function evaluation results, the behavioral data evaluation results, and the subjective scale data evaluation results, a weighted average is performed to obtain the final rehabilitation training evaluation result.

[0024] (1) Processing of brain function data In this embodiment, after the subject (or user) completes each round of rehabilitation training tasks, the collected brain function data during the task are compared and analyzed with the subject's resting-state brain function data to determine the relationship between the degree of brain region activation and the threshold size, including determining the subject's current brain activity and the neural activity and connectivity (i.e., correlation) between brain regions by constructing a brain function connection network, thereby determining the specific rehabilitation status of each brain region of the user, and determining the causal relationship between the interactions between the user's brain regions by constructing a brain effect connection network, while quantitatively evaluating the subject's current condition to obtain brain function evaluation results.

[0025] First, the brain function data of autistic children when they perform the current rehabilitation training task is obtained, and the brain function data of autistic children in a resting state is pre-collected. Specifically, before each data collection, the user's age, gender, height, weight and other information are recorded. The user collects brain function data by wearing a near-infrared device. After wearing, the experiment is carried out. The experiment is divided into two parts: resting state and task state, that is, first collect 20 minutes of near-infrared data from the subject in a quiet environment. These data are the resting state data of the subject. The subject is in a relaxed state of sitting with eyes open throughout the whole process; after resting for 5 minutes, the subject performs the training task as required. At this time, the near-infrared data recorded is the task state data. During this recording process, the user's behavior data is collected synchronously: Open HoloLens2 (existing wearable mixed reality device) to capture hand movement data, and the data collection time is also 20 minutes. The training task uses fNIRS (functional near-infrared spectroscopy) to monitor brain blood oxygen signals. This signal provides feedback on the real-time state of the brain during training. It reflects changes in oxygenated hemoglobin levels in the prefrontal cortex and is used to monitor changes in different brain areas when patients perform tasks, including cognitive, memory, language, movement, emotional and other modules.

[0026] Secondly, the cerebral blood oxygen signal is preprocessed, which includes smoothing, missing value filling and denoising. Specifically, the preprocessing of cerebral blood oxygen signal needs to remove the collected abnormal values ​​as well as the physiological activity noise and motion artifacts therein, so as to facilitate the subsequent coherence analysis between measurement channels and brain functional connectivity analysis. In this embodiment, the time window and sliding average processing are used to smooth the signal; then the missing parts in the data are filled by spline interpolation to maintain the continuity of the signal; then the Butterworth filter is used to remove high-frequency noise, effectively removing the noise components higher than the characteristic frequency of cerebral hemodynamics.

[0027] Then, based on the preprocessed signal data, wavelet phase coherence analysis (WPCO) was used to construct the brain functional connection network, and Granger causality (GC) was used to construct the brain effect connection network.

[0028] For the brain functional connection network, the preprocessed signal data is subjected to wavelet transform to extract the dynamic information characteristics of the phase and amplitude of the oscillation component in the signal, and the signal is converted from the time domain to the frequency domain to obtain the main components of the cerebral blood oxygen signal time series in the frequency domain, which is used to analyze the local intermittent oscillation of the time series. The amplitude after wavelet transform (i.e., the absolute value of the coefficient obtained in the wavelet transform) is used to calculate the activation degree of the brain area. Furthermore, the wavelet phase coherence analysis is used to construct the brain functional connection network, which can quantitatively calculate the degree to which the instantaneous phase between the two time series signals remains consistent throughout the process, thereby evaluating the correlation between the two time series signals. Among them, the above two time series signals refer to the time series signals collected by every two acquisition channels of the infrared device, and the calculation process of the wavelet phase coherence of the data between the two near-infrared device acquisition channels is: the two time series data and After continuous wavelet transform, it is and time The corresponding instantaneous phases are and , calculate the instantaneous phase difference between the two , then and Averaging is performed in the time domain, and the two signals are obtained at the frequency The value of the wavelet phase coherence under is: , these two signals at all frequencies The wavelet phase coherence value (WPCO) between the two signals can be obtained by averaging the wavelet phase coherence values ​​under . The brain regions where the electrodes for collecting brain function data are located are taken as nodes, the connections between nodes are taken as edges, and the inter-regional phase coherence (i.e., the mean of the wavelet phase coherence values ​​of all channel pairs between the two regions) is taken as weights to construct brain functional connection networks for the resting state and the task state, respectively. Based on all the constructed brain functional connection networks, the synergy between different brain regions, i.e., the neural activity and connectivity of brain function, can be analyzed.

[0029] For the brain effect connection network, Granger causality is used to calculate the effective connectivity (EC) within the brain region, that is, to construct a brain effect connection network to reveal the directionality of information transmission between brain regions. It is believed that if a time series Causes time series the occurrence or If there is an impact, then it is related Information knowledge can predict Future values. Specifically, first, the time series signal of any channel is tested for stationarity in turn with other channels. If the two sequences fail the stationarity test, the cointegration test is continued. The stationarity of a time series refers to a time series that has no systematic change in mean (no trend), no systematic change in variance, and strictly eliminates periodic changes over a period of time. Then, the cointegration test is performed. The cointegration test of time series is used to analyze the long-term equilibrium relationship between variables. In the process of cointegration analysis of two variables, if the independent variable and the dependent variable are cointegrated, it can be confirmed that the two variables will not produce spurious regression results and there is a long-term stable relationship between the two variables. Finally, after the above test is passed, the Granger causality between the sequences is analyzed, that is, two sequences are selected. and sequence Establish an autoregressive model; respectively in the sequence and sequence Add the past values ​​of another series to the forecast of , and get the bivariate autoregressive model; Assuming that the two series are Granger causal to each other, use Test and verify the hypothesis. Among them, the quantitative value of Granger causality is calculated based on the prediction error in the autoregressive model and the bivariate autoregressive model: ,in, and Represent the prediction errors in the autoregressive model and the bivariate autoregressive model respectively; and sequence Input the Granger causality test function. If the test result is greater than or equal to the set threshold, the Granger causality quantization value is binarized to 1, otherwise it is binarized to 0. Repeat the above steps until the Granger causality quantization values ​​between any two sequences are binarized, and the brain effect connection network is obtained.

[0030] Based on the above-mentioned constructed brain function connection network and brain effect connection network, the current rehabilitation is evaluated according to its significance level. At the same time, the EEGNet model is used as a rehabilitation scoring model, the EEG signal is input into the pre-trained EEGNet model, and the brain function evaluation result is output. In this embodiment, the EEGNet model is used for training. EEGNet is a general and compact convolutional neural network designed for general EEG recognition tasks. EEGNet consists of 4 layers. The first layer is a convolution layer for simulating the bandpass filtering operation of each channel data; the second layer is a spatial filter layer, which weights the data of each channel by deep convolution; the third layer is an independent convolution layer for extracting category information; the fourth layer is a fully connected layer for classification. Further, the above-mentioned EEGNet model can well evaluate its status based on the fNIRS data (i.e., cerebral blood oxygen signal) of the subject, and its training method is: obtain the fNIRS data of different test subjects and the corresponding test set data of the percentage score, and obtain a rehabilitation scoring model based on fNIRS data by training for 500 cycles.

[0031] As another embodiment, the influence of resting state and task state on brain representation is analyzed by one-way variance analysis, and the current condition of the subject is quantitatively evaluated, and guidance is provided for rehabilitation intervention strategy, and it is verified that brain function data is consistent with behavior and scale evaluation results. In the present embodiment, the comprehensive evaluation is divided into two parts as a whole, and the brain network evaluation is completed by the above-mentioned parts, and a comprehensive score is performed by brain data, behavior and scale, so as to obtain the final comprehensive evaluation result. Among them, the priority of the brain network evaluation result is higher, and the brain network evaluation result is used to optimize the comprehensive score, and the final comprehensive evaluation result is adjusted to obtain the final result, including: when the significance level p<0.05, it is considered to be statistically significant, and the comprehensive evaluation result +5; when the significance level p>=0.05, it is considered not to be statistically significant, and the comprehensive evaluation result -5, wherein the comprehensive evaluation result = subject's final score - threshold. Preferably, Pearson correlation (i.e., Pearson correlation) and Bonferroni correction (i.e., Bonferroni correction) p value can be used to analyze the correlation between scale results, behavior indicators and brain function data to verify the consistency of the evaluation.

[0032] (2) Behavioral data In this embodiment, HoloLens2 is used to collect behavioral data of subjects during the rehabilitation training task, that is, the hand movement images of the subjects, to obtain the three-dimensional coordinates of the hand joint points of the subjects when performing the rehabilitation training task, and then calculate the hand movement score to obtain the behavioral data evaluation result.

[0033] Specifically, the behavioral data of the subjects contain information such as individual behavioral performance, behavioral patterns, behavioral reaction time, behavioral selection, and behavioral preferences, which can help understand the individual's behavioral characteristics, behavioral habits, behavioral changes, etc., thereby revealing the individual's psychological characteristics, cognitive processes, emotional states, etc. HoloLens2 collects the behavioral data of the subjects during the rehabilitation training task, and extracts the position information of the metacarpophalangeal joint (MCP), proximal interphalangeal joint (PIP), and distal interphalangeal joint (DIP) from it. Then, according to the joint point position information, the joint flexion angle, wrist dorsiflexion angle, wrist rotation angle, and Euclidean distance of the joint point are calculated, which are used as task judgment parameters; the total active range of motion of the hand, that is, the finger flexion and extension TAM, is calculated by the formula TAM=flexion degree (MCP1+PIP1+DIP1)-extension deficiency degree (MCP2+PIP2+DIP2), and the wrist rotation angle ROM and wrist dorsiflexion angle ROM are calculated at the same time. These indicators are used as hand function rehabilitation indicators to evaluate the degree of hand function rehabilitation, and then the behavioral data evaluation results are obtained. The above-mentioned extension deficiency and flexion deficiency are the sum of the MCP, PIP and DIP angles in two different states.

[0034] Furthermore, the evaluation criteria of TAM are as follows: excellent, normal flexion and extension, TAM>220°; good, function is more than 75% of healthy fingers, TAM 200~220°; fair, function is 50%~75% of healthy fingers, TAM 180~200°; poor, function is less than 50% of healthy fingers, TAM<180°. Among them, the TAM score of finger flexion and extension is 220° as 100 points, 120° as 0 points, and 1° as 1 point; the ROM score of wrist rotation angle is 160° as 100 points, 60° as 0 points, and 1° as 1 point; the ROM score of wrist dorsiflexion angle is 140° as 100 points, 40° as 0 points, and 1° as 1 point.

[0035] (3) Subjective scale data In this embodiment, after each round of rehabilitation training tasks, the subject will fill in the subjective scale for subjective evaluation (assisted filling), that is, after completing each rehabilitation training task, a self-assessment questionnaire is presented to the subject to collect the subject's subjective feelings during the rehabilitation training task. On the basis of obtaining the subjective scale data, the training performance results are quantified to obtain the subjective scale data evaluation results.

[0036] Specifically, for the subjective scale data of the subjects, the training performance results of the subjects are evaluated according to the scale results, and the scores are converted into a percentage system, including: ① Dorsiflexion of the wrist when the elbow joint is 90 degrees and the shoulder joint is 0 degrees: 0 points: almost unable to dorsiflex the wrist joint; 1 point: able to complete wrist dorsiflexion, but unable to resist resistance; 2 points: able to maintain wrist dorsiflexion with some slight resistance; ②Flexion and extension of the wrist at 90 degrees of the elbow joint and 0 degrees of the shoulder joint: 0 points: cannot move arbitrarily; 1 point: cannot move the wrist joint within the full range of the joint; 2 points: can move smoothly without pause; ③Wrist dorsiflexion at elbow 0 degrees and shoulder 30 degrees: the score is the same as item ①; ④ Wrist flexion and extension at elbow 0 degrees and shoulder 30 degrees: the score is the same as item ②; ⑤ Grip strength 1: The metacarpophalangeal joints are extended and the proximal and distal interphalangeal joints are flexed to test the resistance grip strength: 0 points: unable to maintain the required position; 1 point: weak grip; 2 points: able to withstand considerable resistance to grip; ⑥ Grip 2: When all joints are at position 0, thumb adduction: 0 points: cannot be performed; 1 point: can hold a piece of paper with the thumb, but cannot resist the pulling force; 2 points: can resist the pulling force; ⑦ Grip strength 3: The patient's thumb and index finger can hold a pen: the scoring method is the same as grip strength 2; ⑧ Grip strength 4: The patient can hold a cylindrical object: the scoring method is the same as grip strength 2; ⑨ Grip 5: The patient can grasp a spherical object: the scoring method is the same as Grip 2; ⑩Difficulty of the current training task: 0 points: easy; 1 point: medium; 2 points: difficult.

[0037] After completing the above-mentioned scoring calculation, the brain function data evaluation results and the behavioral data evaluation results, together with the training performance results (i.e., the subjective scale data evaluation results), are obtained. Each subject can obtain three scores, and the three scores are combined for weighted average to calculate the final score as a comprehensive evaluation result. In this embodiment, the brain function data, behavioral data, and training performance result scores account for 40%, 30%, and 30%, respectively. Whether the training effect of the subject has improved is determined by judging whether the score exceeds a certain set threshold. Preferably, the score can be further optimized by the above-mentioned brain network evaluation results to obtain the final evaluation result.

[0038] In the above step S4, according to the rehabilitation training evaluation result, the difficulty level of the current rehabilitation training task is adaptively adjusted to carry out the next round of rehabilitation training, specifically: When the comprehensive evaluation result is much higher than the threshold, the specific result is the subject's final score minus the threshold ≥ 10, and then the training is adjusted to a rehabilitation training scenario for more complex gestures; When 5<the subject's final score-threshold<10, the task difficulty of the rehabilitation training scenario is increased and training is conducted again; When the comprehensive evaluation result is close to the threshold, the specific result is 0≤subject's final score-threshold≤5, repeat training is performed on the current task; When the comprehensive evaluation result is lower than the threshold, specifically -5 < subject's final score - threshold < 0, the task difficulty of the rehabilitation training scene is reduced and training is conducted again. The patient can choose to use rehabilitation gloves to assist rehabilitation. When the comprehensive evaluation result is far below the threshold, the specific result is the subject's final score - threshold ≤ -5. On this basis, the rehabilitation training scenario is adjusted to a simpler gesture for training again, and the rehabilitation gloves are turned on to assist the patient in rehabilitation.

[0039] As another implementation method, the brain function data, behavioral data, and subjective scale data collected by the user end are uploaded to the cloud server, which performs a comprehensive evaluation and transmits the rehabilitation training evaluation results remotely to the doctor end; based on the doctor end, the rehabilitation training adjustment strategy is uploaded to the cloud server, which is fed back to the user end by the cloud server. The user end adjusts the difficulty level of the current rehabilitation training task according to the feedback rehabilitation training adjustment strategy and conducts the next round of rehabilitation training. In this embodiment, the difficulty level can be automatically adjusted according to the score, and the task can also be adjusted according to the doctor's advice. Preferably, the doctor's rehabilitation training adjustment strategy has a higher priority than the patient's automatic adjustment strategy.

[0040] Furthermore, in this embodiment, rehabilitation training tasks with different difficulty levels are set in different modes, wherein the modes include a daily mode and a game mode, such as Figure 2 As shown, the daily mode can provide different rehabilitation training programs according to the user's age and degree of hand dysfunction, and adaptively adjust the training difficulty based on the comprehensive evaluation results. After the glove assistance mode is turned on, the gloves are activated to assist the user in completing the training tasks. The game mode can provide gesture music games in combination with music therapy, and adaptively adjust the game difficulty based on the comprehensive evaluation results. Similarly, after the glove assistance mode is turned on, the gloves are activated to assist the user in completing the training tasks.

[0041] The above-mentioned daily mode includes different scenes in daily life, and provides different rehabilitation training tasks according to the user's age and degree of hand dysfunction. In this training process, the current training task scene is mapped to the augmented reality scene, and the user's attention is attracted by a cartoon picture that integrates virtual and real. When the user is able to accept instructions, the behavior is guided and repeatedly trained with the help of voice instructions; in this process, the user's hand movements are tracked, the hand posture is identified to obtain joint position information, and the total active range of motion of the hand is calculated. The rehabilitation results are evaluated in combination with the task completion status, behavioral data, brain function data and subjective scales, and the difficulty of training is adjusted according to the comprehensive evaluation results. During the training process, the activation tactile feedback actuator of the auxiliary glove gives corresponding tactile feedback. After the glove assistance mode is turned on, when the subject's hand approaches an object, the glove is activated for corresponding interaction (such as assisting the user to grab an object). During the entire training process, if the subject feels any psychological or physical discomfort, the training process can be terminated at any time. For example, the subject can understand and respond to the user's language instructions through voice recognition technology, such as terminating training, adjusting training difficulty, changing training scenes, etc., and interact and communicate with the user; the emotional state can be perceived through changes in facial expressions and voices, and corresponding responses can be made according to different situations: when the user is perceived to be in a low mood, the difficulty of the current training task will be reduced or the training will be terminated; when the user is perceived to be in a high mood, the difficulty of the current training task will be increased. Among them, the user's facial expressions and voice information are all obtained by HoloLens2.

[0042] Furthermore, in order to avoid the monotony of the rehabilitation training environment, this embodiment sets rehabilitation training tasks of different difficulty levels in a variety of different scenarios for the daily mode. Specifically, different training scenarios such as restaurants, bedrooms and beaches are designed. Through different interactive tasks in the scenarios, users are prompted to complete six single hand daily fine movements and their combinations, such as thumb-to-finger, fingertip-to-finger, three-finger pinch, cylindrical grip, side pinch, and fist. Each training movement that reaches the standard can be directly converted into points. Completing all tasks in the level can enter the next challenge or refresh the current level points again.

[0043] Scene 1 is themed on a restaurant. The scene includes models of dining tables, plates, kettles, cups, fruits and cakes. Users need to complete tasks within the specified time, including: picking up the fruits scattered on the table, sorting them and placing them in the designated fruit plate. If the user's placement position exceeds the position of the indicated area, reset the time and the indicated area turns red, otherwise it turns green; pour the water in the kettle into the cup. If the water falls beyond the position of the indicated area, reset the time and the indicated area turns red, otherwise it turns green; cut the cake along the dividing line. If the user's dividing position is far from the dividing line, reset the time and the dividing line turns red, otherwise it turns green.

[0044] Scene 2 is themed on the bedroom. The scene includes models such as toys, drawing boards and paintbrushes. Users need to complete tasks within the specified time, including: picking up toys that have fallen on the ground and throwing them into a box. If the user's throwing point exceeds the warning line, the time is reset and the warning line turns red; stringing beads, stringing and connecting beads of different colors and shapes, and guiding users to use certain rules to present the changes in the bead strings; drawing patterns containing straight lines, origami, curves or circles.

[0045] Scene three is themed on the beach and contains models of shells, crabs and small fish. Users need to complete tasks within the specified time, including: picking up shells scattered on the beach, classifying them and placing them in designated buckets. If the user's placement position exceeds the indicated area, the time is reset and the indicated area turns red, otherwise it turns green; use a fishing net to scoop up crabs and small fish in the sea water and pour them into a bucket. If the user's placement position exceeds the indicated area, the time is reset and the indicated area turns red, otherwise it turns green.

[0046] As another implementation method, the rehabilitation training tasks are customized according to the user's age and degree of hand dysfunction, and tasks of different difficulty levels are provided in different scenarios. For example, for autistic children aged 1-3 or with mild hand dysfunction, training methods such as stringing beads, throwing objects, flipping, grabbing and rubbing are used; for children aged 4 or above or with severe hand dysfunction, training methods such as drawing straight lines, origami, curves or circles are used. The rehabilitation training tasks are divided into three levels of difficulty: elementary, intermediate and advanced according to the fineness of hand movements. The difficulty training is adaptively adjusted based on the comprehensive evaluation results, and the range of motion, speed and tactile stimulation of the joints of the rehabilitation gloves are adjusted at the same time. Specifically, the difficulty adjustment plan for scene one is: adjust the type, quantity and size of fruits; adjust the size of the cup mouth; adjust the number of dividing lines. The difficulty adjustment plan for scene two is: adjust the number and size of toys; adjust the type, quantity and size of beads; adjust the accuracy of line drawing. The difficulty adjustment plan for scene three is: adjust the type, quantity and size of shells; adjust the size of the fishing net, and adjust the type, quantity and size of crabs and small fish.

[0047] The above rehabilitation training tasks can provide targeted training for different key parts of the hand, including: Finger interphalangeal joint flexion function training setting: In scenes one, two and three, the user's finger proximal joints will bend when completing the grasping action. The finger proximal joint flexion can be trained in the process of repeatedly grasping objects. In the design, the interphalangeal joint flexion angle of the current five fingers is detected, and the interphalangeal joint flexion angle characteristic θ value is set when grasping objects. The parameter of θ value must be greater than the normal interphalangeal joint flexion angle, that is, 90°, and the operation is judged to be successful. The Object Manipulator component of MRTK (Mixed Reality Toolkit) is used to make the object movable, scalable and rotatable to grasp the object. During the game, the user needs to grasp objects many times, so the interphalangeal joint flexion action needs to be repeated, which improves the user's finger interphalangeal joint flexion ability to a certain extent, achieving the purpose of training the interphalangeal joints.

[0048] Training setting of wrist joint dorsiflexion and rotation function: In scene 2, the user needs to throw the grasped object into the moving box. Conditional judgment is added in the design. The wrist joint dorsiflexion angle θ value is greater than the normal wrist dorsiflexion angle of 20° before the system can perform the throwing operation. The user can train the wrist dorsiflexion function by repeatedly throwing objects. In scene 1, the user needs to pour the water from the kettle into the cup. Conditional judgment is added in the design. The operation is judged to be successful after the wrist joint rotation angle θ value is greater than the normal wrist dorsiflexion angle of 60°. The system can perform the dumping operation; in scene 3, the user needs to pour the crabs and small fish from the fishing net into the bucket. Conditional judgment is added in the design. The wrist joint rotation angle θ value is greater than the normal wrist dorsiflexion angle of 60° before the system can perform the dumping operation. The user can train the wrist rotation function by repeatedly dumping objects. In scene 2, the user needs to draw patterns containing straight lines, origami, curves or circles. The user can train the wrist dorsiflexion and rotation functions by repeatedly drawing different patterns.

[0049] Thumb-to-finger function training setting: In scene 2, the user needs to string and connect beads of different colors and shapes. During the game, when the patient's thumb and index finger touch the beads and the Euclidean distance between the thumb and index finger is close to 0, the operation is considered successful. While stringing the beads, the user repeats the touch of the thumb and index finger to restore the thumb-to-finger function.

[0050] The above game mode is based on the theme of music games. It automatically generates gesture models following the rhythm of music, including six single actions and their combinations, such as thumb-to-finger, fingertip-to-finger, three-finger pinch, cylindrical grip, side pinch, and fist. When the model reaches the judgment area, the user needs to make a corresponding gesture. At this time, the user's hand is captured to obtain the hand joint position information, the gesture made by the user is calculated, and matched with the generated model. In this training process, the current training task scene is mapped to the augmented reality scene, and the user's attention is attracted by the cartoon picture of virtual and real fusion. When the user can accept the command, the behavior is guided and repeatedly trained with the help of voice commands. Finally, the rehabilitation results are evaluated in combination with the task completion status, behavioral data, brain function data and subjective scales. The difficulty training is adjusted according to the comprehensive evaluation results, and the gesture generation speed and number of gestures are adjusted. After the glove-assisted mode is turned on, the generation point sends the generated information to the server through the communication unit while generating the gesture model, and the rehabilitation gloves are started to assist the user to make the corresponding gesture.

[0051] During the training process of the above-mentioned game mode, the acquisition of behavioral data, brain function data and subjective scale data is the same as that of the daily training mode.

[0052] Embodiment 2 This embodiment provides a virtual-real fusion hand function rehabilitation training system for children with autism. Figure 3 As shown, including: A mapping unit, used to map the motion information and virtual scene corresponding to the current rehabilitation training task to the augmented reality scene, and guide the autistic child to interact with the augmented reality scene; An auxiliary unit, used to assist autistic children to complete multiple rounds of current rehabilitation training tasks based on the flexible rehabilitation gloves they wear; A collection unit is used to collect brain function data, behavioral data and subjective scale data of autistic children when they perform current rehabilitation training tasks in a non-assisted state; An evaluation unit, used to perform a comprehensive evaluation based on the collected data and obtain a rehabilitation training evaluation result; The adjustment unit is used to adaptively adjust the difficulty level of the current rehabilitation training task according to the rehabilitation training evaluation results, and conduct the next round of rehabilitation training.

[0053] Among them, the mapping unit includes an augmented reality device for building an augmented reality scene that integrates virtual and real; the auxiliary unit includes a flexible rehabilitation glove, which is provided with a tactile feedback mechanism; the acquisition unit includes a near-infrared device, a camera device, and a recording device. The near-infrared device is used to collect brain function data, the camera device is used to collect user behavioral data, and the recording device is used to collect sound data; the evaluation unit includes a cloud server, namely a rehabilitation training server, which is used to perform a comprehensive evaluation based on the collected data and obtain the rehabilitation training evaluation results; the adjustment unit includes a controller electrically connected to the flexible rehabilitation glove, which is used to adjust the difficulty level of the current rehabilitation training task according to the rehabilitation training evaluation results.

[0054] In addition, the rehabilitation training server is respectively connected to the augmented reality device, the flexible rehabilitation gloves, the near infrared device and the camera device for communication.

[0055] The above-mentioned augmented reality device uses HoloLens2 to select rehabilitation training programs based on the user's age and degree of hand dysfunction, build a virtual and real integrated training environment, and visually present training tasks to users; The above-mentioned flexible rehabilitation gloves integrate the bionic principle of finger skin wrinkle structure and Hall sensing mechanism, adopt a bionic finger sleeve structure that combines smooth movement and precise perception, and integrate the sensing system, SMA drive and intelligent control system, energy supply and thermal control system and tactile feedback system; The above-mentioned near-infrared device, patients wear the near-infrared device to collect brain function data, where the near-infrared collection channel positions on the human brain include six brain regions: left prefrontal cortex (LPFC), right prefrontal cortex (RPFC), left motor cortex (LMC), right motor cortex (RMC), left occipital lobe (LOL) and right occipital lobe (ROL), and the electrode positions are set according to the international standard 10 / 10; The above-mentioned camera device uses the built-in TOF depth camera of HoloLens2 to collect user behavioral data during the process of the user completing the rehabilitation training task, and uploads the collected data to the virtual-real fusion autistic children's hand function rehabilitation training server; in addition, the camera device is also used to collect the user's facial image during the training task, and upload the collected image data to the virtual-real fusion autistic children's hand function rehabilitation training server; The above-mentioned recording device collects the user's voice when the user completes the rehabilitation training task, and uploads the collected data to the virtual-real fusion autism children's hand function rehabilitation training server.

[0056] Furthermore, the system also includes a recognition unit, which interacts with the hand function rehabilitation training server for autistic children. The recognition unit is used to recognize facial expressions, sound changes, and voice information during the training process based on the received facial image data and sound data, using image recognition technology and voice recognition technology, so as to make corresponding responses.

[0057] Further, such as Figure 3 As shown, the system also includes a communication unit for uploading brain function data, behavioral data, and subjective scale data obtained by the user end to the cloud server, and uploading the rehabilitation training adjustment strategy generated by the doctor end to the cloud server; it is also used to remotely transmit the rehabilitation training evaluation results to the doctor end, and feed back the rehabilitation training adjustment strategy to the user end.

[0058] The user end, i.e. the patient end, manages and controls data collection equipment and rehabilitation training equipment; uploads the subject's brain function data, behavioral data, and subjective scale data to the server; receives information from the doctor end to achieve remote communication between doctors and patients; The doctor receives the evaluation results from the server and adjusts the patient's rehabilitation plan based on the evaluation results; the doctor receives information from the patient to achieve remote communication between doctors and patients.

[0059] The above-mentioned rehabilitation training server conducts a comprehensive evaluation of the subject's behavioral data, brain function data, and subjective scale, and adjusts the subject's hand function rehabilitation training task plan and training difficulty according to the comprehensive evaluation results and the control process of Example 1. During the training process, the auxiliary glove activates the tactile feedback actuator to give corresponding tactile feedback. In the auxiliary mode, the rehabilitation glove is controlled to assist the user in completing the task. In the process of the user completing the rehabilitation training task, the emotional state is perceived through changes in facial expressions and voices, and corresponding responses are made according to different situations; the patient-side information is sent to the doctor's side to realize remote communication between doctors and patients. After the user completes the rehabilitation training task, the training and evaluation results are sent to the doctor's terminal, and the doctor's terminal information is received to adjust the rehabilitation plan.

[0060] In this embodiment, a virtual-real fusion hand function rehabilitation training environment is built based on an augmented reality device, specifically including: the user wears a near-infrared device, and an experiment is conducted after the wearing is completed. The experiment is divided into two parts: resting state and task state. First, the user's resting state brain function indicators are collected, and the acquired data is preprocessed, including removing outliers, physiological activity noise and motion artifacts in the near-infrared data; after completing the resting state data collection, the subject wears an augmented reality head display and a rehabilitation training device and starts training. The augmented reality head display will display the rehabilitation training task. After the user selects the task difficulty, the current round of rehabilitation training begins. In the auxiliary mode, the gloves are started to assist the user to complete the training task and give corresponding tactile feedback. In the non-assisted mode, the glove auxiliary function is turned off. During the training, the near-infrared device collects the subject's brain function data in real time, the camera device collects the user's behavioral data when the subject performs the rehabilitation training task, and the recording device collects the user's voice. The system uses voice recognition technology to understand and respond to the user's language instructions to achieve interaction and communication with the user; perceives the emotional state through changes in facial expressions and voice, and responds accordingly according to different situations. After each round of rehabilitation training tasks, the user will fill in the subjective scale.

[0061] In this embodiment, the augmented reality training environment is developed using Unity+MRTK, and the project is deployed on an augmented reality helmet, namely Hololens2, for users to conduct rehabilitation training; the difficulty of the training program can be selected through button events or voice control events developed by Unity+MRTK; during the training process, Hololens2 will provide real-time feedback on parameters such as training difficulty, task completion status, brain function data and behavioral data; after completing each training task, the subject needs to fill in a subjective scale through buttons or voice control to feedback the subjective feelings during the training process.

[0062] The above-mentioned near-infrared device uses the NirSmart portable near-infrared brain function imaging system, and takes the brain area where the electrode position for collecting data is located as a node, that is, there are a total of 6 nodes, representing 6 brain areas respectively, and takes the connection between the nodes as an edge, and calculates the wavelet phase coherence between the areas as the weight of the edge; in the channel of the NirSmart, taking the left prefrontal cortex (LPFC) and the right prefrontal cortex (RPFC) brain areas as examples, the wavelet phase coherence result between the two brain areas is the average of the wavelet phase coherence values ​​of their channel pairs, thereby constructing a brain function connection network to analyze the neural activity and connectivity between different brain areas of the brain. After each round of rehabilitation training tasks, the collected brain function data during the task and the resting state data are analyzed to determine the relationship between the degree of brain area activation and the threshold value, including determining the patient's current brain activity and the neural activity and connectivity between brain areas through the brain function connection network, thereby determining the specific rehabilitation status of each brain area of ​​the patient, and determining the causal relationship of the interaction between the patient's brain areas through the brain effect connection network, and quantitatively evaluating the current condition of the subject.

[0063] like Figure 4 As shown in the figure, the driving system of the flexible rehabilitation gloves adopts a muscle-tendon bionic structure design, driven by a high power-to-weight ratio shape memory alloy (SMA) spring, which is placed in the arm assembly. The SMA drive deformation is transmitted to the finger sleeve structure using a rope mechanism with multi-stage force amplification, driving the fingers to achieve bending / stretching movements. Furthermore, a joint sensing system based on Hall sensor elements is designed. By adapting to the non-uniform stiffness structure of the flexible finger sleeve, the interference of the finger sleeve movement on the sensing system is reduced, and stable and accurate finger state perception is achieved.

[0064] Preferably, a tactile feedback system is added to the glove, including: Vibration feedback, embedding voice coil actuators in rehabilitation gloves, so that rehabilitation gloves can present realistic button clicks and impact simulation. A vibration motor is installed in the palm to simulate large-scale impacts, and a vibration motor is installed on each fingertip to simulate small-scale impacts; Pressure feedback: Inflatable pneumatic actuators are embedded in the lining of rehabilitation gloves to create precise pressure by inflating and deflating bubbles, allowing the hand to feel the sensation of touching real-world objects. When the user grabs a virtual object, the pneumatic actuators on the fingers are inflated and hardened to simulate resistance, while pulling the skin to simulate the effect of gravity. Texture feedback, which weaves a large number of dynamic tactile feedback actuators into the flexible material at the fingertips, is designed to provide users with high-resolution touch delivered directly to each fingertip. Each fingertip tactile array is equipped with bubble-shaped "pixels" containing fluid, which fills and stretches the bubble when activated; each pixel is a dedicated electrically controlled pump, only a few hundred microns thick, which contains no moving parts and operates on the principle of electroosmosis, directly attracting the electric charge in the fluid to make it flow. Figure 5 As shown, a large number of "pixels" adapt to objects of various shapes, sizes and textures, thereby providing personalized tactile feedback and stimulating fingertip nerves to assist in treatment.

[0065] The steps involved in the above embodiment 2 correspond to those in the method embodiment 1. For the specific implementation method, please refer to the relevant description part of the embodiment 1.

[0066] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0067] The above description is only a preferred embodiment of the present invention. Although the specific implementation mode of the present invention is described in conjunction with the accompanying drawings, it is not a limitation of the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A virtual-real integration hand function rehabilitation training method for children with autism, characterized by: include: Map the motion information and virtual scenes corresponding to the current rehabilitation training task to the augmented reality scene, and guide autistic children to interact with the augmented reality scene; Based on the flexible rehabilitation gloves worn, autistic children are assisted to complete multiple rounds of current rehabilitation training tasks; In a non-assisted state, brain function data, behavioral data, and subjective scale data of autistic children are collected when they perform the current rehabilitation training tasks, and a comprehensive evaluation is performed based on the collected data to obtain the rehabilitation training evaluation results; According to the rehabilitation training evaluation results, the difficulty level of the current rehabilitation training task is adaptively adjusted to carry out the next round of rehabilitation training.

2. The virtual-real integration hand function rehabilitation training method for autistic children according to claim 1, characterized in that: The brain function data, behavioral data, and subjective scale data collected by the user are uploaded to the cloud server, which conducts a comprehensive evaluation and remotely transmits the rehabilitation training evaluation results to the doctor. Based on the doctor side, the rehabilitation training adjustment strategy is uploaded to the cloud server, which then feeds back to the user side. The user side adjusts the difficulty level of the current rehabilitation training task based on the feedback of the rehabilitation training adjustment strategy and conducts the next round of rehabilitation training.

3. The virtual-real integration hand function rehabilitation training method for autistic children according to claim 1, characterized in that: Comprehensive evaluation is performed based on the collected data to obtain rehabilitation training evaluation results, including: According to the collected brain function data and the pre-collected resting state brain function data, the activation degree of different brain regions is obtained to obtain the brain function evaluation results; According to the collected behavioral data, the hand movement score is calculated to obtain the behavioral data evaluation result; According to the acquired subjective scale data, the training performance results are quantified to obtain the subjective scale data evaluation results; Based on the brain function assessment results, behavioral data assessment results, and subjective scale data assessment results, a weighted average was performed to obtain the final rehabilitation training assessment results.

4. The virtual-real integration hand function rehabilitation training method for autistic children according to claim 3, characterized in that: According to the collected brain function data and the pre-collected resting state brain function data, the activation degree of different brain regions is obtained, and the brain function evaluation results are obtained, including: Acquire brain function data of autistic children when they perform current rehabilitation training tasks, and pre-collect brain function data of autistic children in a resting state; the brain function data includes brain blood oxygen signals of the prefrontal cortex; Preprocessing the brain blood oxygen signal; the preprocessing includes smoothing, missing value filling and denoising; Based on the preprocessed brain blood oxygen signal, wavelet phase coherence analysis and Granger causality were used to construct the brain function connection network and brain effect connection network respectively; at the same time, the preprocessed brain blood oxygen signal was input into the pre-trained EEGNet model to output the brain function evaluation results; Based on the significance levels of the brain functional connection network and the brain effect connection network, the effectiveness of the current rehabilitation training task is evaluated, and the final rehabilitation training evaluation results are optimized according to the significance levels.

5. The virtual-real integration hand function rehabilitation training method for autistic children according to claim 3, characterized in that: Also includes: According to the collected behavioral data, the hand movement score is calculated to obtain the behavioral data evaluation results, including: Based on behavioral data, the position information of the metacarpophalangeal joints, proximal interphalangeal joints, and distal interphalangeal joints of the hand during task execution was extracted; According to the extracted joint point position information, the joint flexion angle, wrist dorsiflexion angle, wrist rotation angle and the Euclidean distance of the joint point are calculated, and then the total active range of motion of the hand, wrist rotation degree and dorsiflexion angle are calculated; at the same time, according to the change of the extracted joint point position information per unit time, the movement speed of the hand joint point is calculated; Based on the total active range of motion of the hand, wrist rotation, dorsiflexion angle and movement speed of the hand joints, the degree of hand function recovery and hand movement rate were evaluated to obtain behavioral data evaluation results.

6. The virtual-real integration hand function rehabilitation training method for autistic children as claimed in claim 3, characterized in that: Based on the acquired subjective scale data, the training performance results are quantified to obtain the subjective scale data evaluation results, including: According to the rotation angles of the elbow joint, shoulder joint, and wrist joint as well as the grip strength in the subjective scale data, the training performance results are quantified to obtain the subjective scale data evaluation results.

7. The virtual-real integration hand function rehabilitation training method for autistic children according to claim 1, characterized in that: Set rehabilitation training tasks of different difficulty levels in different modes; The modes include a daily mode and a game mode, wherein the daily mode includes rehabilitation training tasks of different difficulty levels in different scenarios.

8. A virtual-real fusion hand function rehabilitation training system for children with autism, characterized by: include: A mapping unit, used to map the motion information and virtual scene corresponding to the current rehabilitation training task to the augmented reality scene, and guide the autistic child to interact with the augmented reality scene; An auxiliary unit, used to assist autistic children to complete multiple rounds of current rehabilitation training tasks based on the flexible rehabilitation gloves they wear; A collection unit is used to collect brain function data, behavioral data and subjective scale data of autistic children when they perform current rehabilitation training tasks in a non-assisted state; An evaluation unit, used to perform a comprehensive evaluation based on the collected data and obtain a rehabilitation training evaluation result; The adjustment unit is used to adaptively adjust the difficulty level of the current rehabilitation training task according to the rehabilitation training evaluation results, and conduct the next round of rehabilitation training.

9. The virtual-real fusion hand function rehabilitation training system for children with autism as claimed in claim 8, characterized in that: The mapping unit includes an augmented reality device for building an augmented reality scene that integrates virtuality and reality; The auxiliary unit comprises a flexible rehabilitation glove, and the flexible rehabilitation glove is provided with a tactile feedback mechanism; The acquisition unit includes a near-infrared device, a camera, and a recording device. The near-infrared device is used to collect brain function data, the camera is used to collect user behavior data, and the recording device is used to collect sound data. The evaluation unit includes a cloud server for performing a comprehensive evaluation based on the collected data to obtain a rehabilitation training evaluation result; The adjustment unit includes a controller electrically connected to the flexible rehabilitation glove, and is used to adjust the difficulty level of the current rehabilitation training task according to the rehabilitation training evaluation result.

10. The virtual-real fusion hand function rehabilitation training system for children with autism as claimed in claim 9, characterized in that: It also includes a communication unit for uploading brain function data, behavioral data and subjective scale data obtained by the user end to the cloud server, and uploading the rehabilitation training adjustment strategy generated by the doctor end to the cloud server; it is also used to remotely transmit the rehabilitation training evaluation results to the doctor end, and feed back the rehabilitation training adjustment strategy to the user end.

Citation Information

Patent Citations

  • Multi-source data fusion user rehabilitation level evaluation system

    CN115778323A

  • Motion decoding method based on data enhancement and multi-scale feature extraction convolutional neural network

    CN117520935A

  • Upper limb rehabilitation training system based on augmented reality

    CN117894435A

  • Self-adaptive upper limb rehabilitation training control method and system based on multi-source data

    CN118098507A

  • Children autism rehabilitation evaluation system based on multi-source data driving

    CN118098600A