A virtual-real integrated method and system for hand function rehabilitation training in children with autism

Abstract

This invention discloses a virtual-real integrated method and system for hand function rehabilitation training in children with autism, belonging to the field of autism rehabilitation training technology. The method involves: mapping the motor information and virtual scene corresponding to the current rehabilitation training task into an augmented reality scene, guiding the child with autism to interact with the augmented reality scene; using a flexible rehabilitation glove, assisting the child with autism in completing multiple rounds of the current rehabilitation training task; collecting brain function data, behavioral data, and subjective scale data of the child with autism while performing the current rehabilitation training task in a non-assisted state, conducting a comprehensive evaluation based on the collected data to obtain rehabilitation training evaluation results; and adaptively adjusting the difficulty level of the current rehabilitation training task according to the rehabilitation training evaluation results for the next round of rehabilitation training. This invention can flexibly adjust the difficulty and content of the training task, assisting patients to conduct rehabilitation training independently and improving the rehabilitation effect of hand function in children with autism.

Description

Technical Field

[0001] This invention relates to the field of rehabilitation training technology for children with autism, and in particular to a virtual-real integrated method and system for hand function rehabilitation training for children with autism. Background Technology

[0002] Autism Spectrum Disorder (ASD), also known as autism spectrum disorder or autism spectrum disorder, has an increasing incidence rate. There is no cure for autism; treatment relies on long-term educational interventions. Autism is a pervasive neurodevelopmental disorder that appears in early childhood. Fine motor skills refer to movements involving small muscles or muscle groups, which typically require high coordination and control, including precise control of the strength and position of the hands and fingers. Children with autism generally have significantly lower grasping abilities and fine motor dexterity than their peers, and hand-eye coordination disorders are common. Currently, interventions for motor skills in children with autism are gaining traction, but most research focuses on improving gross motor skills, with limited studies specifically addressing fine motor development disorders. Therefore, it is crucial to pay attention to the development of fine motor skills in children with autism and to develop corresponding intervention and training strategies.

[0003] Educational intervention is a traditional rehabilitation training method, often involving targeted verbal instruction to train autistic individuals. It involves demonstrating task movements and providing hands-on assistance, following a principle of progression from easy to difficult and from simple to complex, including training in grasping, finger manipulation, coordination, and skills. However, this approach has limited effectiveness in improving the perceptual, motor, language, and cognitive skills of autistic individuals, and its improvement in adaptive behaviors, including communication and daily living, is negligible. Therefore, these educational intervention methods are not highly applicable to children with autism.

[0004] The emergence of rehabilitation gloves has filled the gaps in traditional educational rehabilitation treatment programs, proving more effective in the rehabilitation, treatment, and education of hand function in patients with autism spectrum disorder. Rehabilitation gloves can be divided into rigid and flexible types. They use rigid exoskeletons or flexible materials to drive hand movements, thus replacing medical staff in assisting patients with hand rehabilitation exercises and providing long-term, precise motor stimulation, which helps restore hand function, reduces medical costs, and expands treatment options. However, due to the lack of tactile sensors, rehabilitation gloves cannot stimulate the nerves in the patient's hand to produce tactile feedback. Furthermore, due to technological, material limitations, and sensor accuracy issues, the training content is often monotonous and the application scenarios are limited. It is difficult to accurately assess the recovery level and adjust the rehabilitation program by acquiring real-time information about the patient's hand posture, severely impacting the patient's motivation and treatment effectiveness, thus limiting their application and development in the field of rehabilitation medicine. In addition, current research focuses primarily on hand function rehabilitation training, neglecting the improvement of emotional or social skills, and lacking attention to the psychological well-being of children with autism.

[0005] Currently, using augmented reality (AR) technology for interactive games is a growing trend that is more suitable for the rehabilitation of children with autism. This type of interactive game is diverse and motivating, making it easier to increase the interest of children with autism in rehabilitation, thereby minimizing anxiety and boredom during tasks and maintaining a high level of participation and learning efficiency. However, the primary aspect of hand function therapy is establishing correct hand movement patterns, good postural control, and improving residual function. Currently, hand function rehabilitation using augmented reality technology requires the assistance of medical staff. Children with autism often struggle to independently establish correct hand movement patterns and good postural control, making it difficult for them to complete hand function rehabilitation training. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a virtual-real integrated hand function rehabilitation training method and system for children with autism. It utilizes augmented reality technology to construct a virtual-real integrated training scenario, expanding the training environment and identifying hand postures to obtain joint position information data. This data, combined with brain function data and subjective scale data, is processed and used as a quantitative evaluation indicator of rehabilitation effectiveness. Furthermore, it incorporates various hand function rehabilitation training tasks, flexibly adjusting the difficulty and content of the tasks to assist patients in self-directed rehabilitation training. Additionally, it employs speech recognition technology to understand and respond to verbal commands, enabling interactive communication. Simultaneously, it captures facial expressions and vocal changes to perceive emotional states and respond accordingly to different situations, significantly improving the rehabilitation effect of hand function in children with autism.

[0007] In a first aspect, the present invention provides a method for hand function rehabilitation training of autistic children that integrates virtual and real elements.

[0008] A virtual-real integrated method for hand function rehabilitation training in children with autism includes:

[0009] The movement information and virtual scene corresponding to the current rehabilitation training task are mapped to the augmented reality scene to guide children with autism to interact with the augmented reality scene;

[0010] The flexible rehabilitation gloves worn by children with autism help them complete multiple rounds of current rehabilitation training tasks.

[0011] In a non-assisted state, brain function data, behavioral data, and subjective scale data of autistic children are collected when they perform the current rehabilitation training task. A comprehensive assessment is conducted based on the collected data to obtain the rehabilitation training assessment results.

[0012] Based on the rehabilitation training assessment results, the difficulty level of the current rehabilitation training task is adaptively adjusted before proceeding to the next round of rehabilitation training.

[0013] Secondly, the present invention provides a virtual-real integrated hand function rehabilitation training system for children with autism.

[0014] A virtual-real integrated hand function rehabilitation training system for children with autism includes:

[0015] The mapping unit is used to map the motion information and virtual scene corresponding to the current rehabilitation training task to the augmented reality scene, guiding autistic children to interact with the augmented reality scene;

[0016] The assistive unit is designed to help children with autism complete multiple rounds of current rehabilitation training tasks based on the flexible rehabilitation gloves they wear.

[0017] The data acquisition unit is used to collect brain function data, behavioral data, and subjective scale data of autistic children when they perform the current rehabilitation training task in a non-assisted state.

[0018] The assessment unit is used to conduct a comprehensive assessment based on the collected data to obtain rehabilitation training assessment results;

[0019] The adjustment unit is used to adaptively adjust the difficulty level of the current rehabilitation training task based on the rehabilitation training assessment results, so as to carry out the next round of rehabilitation training.

[0020] The above one or more technical solutions have the following beneficial effects:

[0021] 1. This invention provides a virtual-real integrated hand function rehabilitation training method and system for children with autism. It utilizes augmented reality technology to construct a virtual-real integrated training scenario to expand the training environment and identifies hand postures to obtain joint position information data. This data, combined with brain function data and subjective scale data, is processed and used as a quantitative evaluation indicator of rehabilitation effectiveness. Based on this, various hand function rehabilitation training tasks are designed, flexibly adjusting the difficulty and content of the training tasks to assist patients in self-directed rehabilitation training. Furthermore, speech recognition technology is used to understand and respond to language commands, enabling interactive communication. It also captures changes in facial expressions and voice to perceive emotional states and respond accordingly to different situations, greatly improving the rehabilitation effect of hand function in children with autism.

[0022] 2. In this invention, rehabilitation gloves are used to fully exercise the muscles and joints of the hand, establishing a correct correspondence pattern in the hand and providing personalized tactile feedback to stimulate the nerves in the hand. Augmented reality-based exercise training helps promote nerve regeneration and reorganization in the patient's brain regions. Through repeated exercise training, the connection and information transmission ability of damaged neurons are strengthened, improving the coordination and precision of the patient's hand muscles. This invention utilizes augmented reality technology to combine virtual information with the real world, providing patients with realistic visual and auditory feedback. Combined with the tactile feedback of the gloves, an immersive experience is achieved, alleviating sensory integration dysfunction and improving the rehabilitation effect of hand function. Furthermore, by designing gamified interfaces and tasks for rehabilitation training, the interest and enthusiasm of children with autism are stimulated, increasing the enjoyment of training and thus improving the efficiency and effectiveness of hand function rehabilitation.

[0023] 3. This invention provides a quantitative evaluation index for rehabilitation effect and a personalized rehabilitation plan. During augmented reality interaction, the user's hand movements are tracked and hand postures are identified to obtain joint position information, such as finger bending angle, finger movement trajectory, and time taken to complete the task. After processing and analysis, these are used as quantitative evaluation indicators for rehabilitation effect. The difficulty and training content are then flexibly adjusted according to the rehabilitation effect evaluation results to meet the needs of different patients.

[0024] 4. In this invention, rehabilitation gloves are used to assist patients in completing repetitive hand joint movements instead of medical staff. Different control modes drive the hand joints to complete different actions and fully stimulate the patient's tactile perception. Through augmented reality technology, the one-on-one rehabilitation training mode between medical staff and patients can be transformed into one-to-many, or even completely eliminate the dependence on medical staff and achieve autonomous rehabilitation training. Moreover, it can also achieve the goal of patients undergoing home rehabilitation and medical staff providing remote guidance, effectively alleviating the pressure of scarce medical resources. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0026] Figure 1 This is an overall flowchart of the virtual-real integrated hand function rehabilitation training method for autistic children as described in this embodiment of the invention;

[0027] Figure 2 This is a schematic diagram of different modes of rehabilitation training tasks in embodiments of the present invention;

[0028] Figure 3 This is a schematic diagram of the virtual-real integrated hand function rehabilitation training system for autistic children as described in an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of the flexible rehabilitation glove used in an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of the feedback pixels of the tactile feedback system of the flexible rehabilitation glove used in this embodiment of the invention. Detailed Implementation

[0031] It should be noted that the following detailed descriptions are exemplary and are intended only to describe specific embodiments and to provide further explanation of the invention, and are not intended to limit the scope of exemplary embodiments of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0032] Example 1

[0033] This embodiment provides a virtual-real integrated method for hand function rehabilitation training in children with autism, such as... Figure 1 As shown, it includes the following steps:

[0034] Step S1: Map the movement information and virtual scene corresponding to the current rehabilitation training task to the augmented reality scene, and guide the autistic child to interact with the augmented reality scene;

[0035] Step S2: Based on the flexible rehabilitation gloves worn, assist autistic children in completing multiple rounds of the current rehabilitation training tasks;

[0036] Step S3: In a non-assisted state, collect brain function data, behavioral data, and subjective scale data of autistic children when they perform the current rehabilitation training task. Based on the collected data, conduct a comprehensive assessment to obtain the rehabilitation training assessment results.

[0037] Step S4: Based on the rehabilitation training assessment results, adaptively adjust the difficulty level of the current rehabilitation training task and proceed to the next round of rehabilitation training.

[0038] In step S1 above, the movement information and virtual scene corresponding to the current rehabilitation training task are mapped onto the augmented reality scene. This allows the subject (i.e., the autistic child in the experiment) to learn in advance, through the augmented reality device, about the gesture type and movement information of the current rehabilitation training task before the start of each round of rehabilitation training. This movement information includes the movement speed, movement trajectory, and task difficulty of the rehabilitation training. Based on this, through the virtual-real integrated rehabilitation training scene, the autistic child is guided to interact with the augmented reality scene to conduct training in areas such as hearing, vision, basic senses, balance, and spatial perception.

[0039] In step S2 above, the subject wears a flexible rehabilitation glove and activates it to assist the subject in passively completing rehabilitation training tasks. The glove is activated based on interactive information in the augmented reality scene, causing different fingers of the subject to bend / extend, assisting the subject in completing the corresponding gestures.

[0040] In step S3 above, in a non-assisted state, brain function data, behavioral data, and subjective scale data of autistic children are collected when they perform the current rehabilitation training task. A comprehensive evaluation is conducted based on the collected data to obtain the rehabilitation training evaluation results.

[0041] In this embodiment, after each round of rehabilitation training tasks, the brain function data collected during the task is compared and analyzed with the brain function data of the subject in the resting state (or resting state) to obtain the brain function assessment results. Then, the subject's behavioral data and subjective scale data are combined for comprehensive evaluation to obtain the final comprehensive assessment result, which is presented in the form of a score.

[0042] Specifically, firstly, based on the collected brain function data and the pre-collected resting-state brain function data, the activation levels of different brain regions are obtained to obtain brain function assessment results; then, based on the collected behavioral data, hand movement scores are calculated to obtain behavioral data assessment results; secondly, based on the acquired subjective scale data, training performance results are quantified to obtain subjective scale data assessment results; finally, based on the brain function assessment results, behavioral data assessment results, and subjective scale data assessment results, a weighted average is performed to obtain the final rehabilitation training assessment result.

[0043] (1) Processing of brain function data

[0044] In this embodiment, after each round of rehabilitation training, the brain function data collected during the task is compared and analyzed with the subject's resting-state brain function data to determine the relationship between the degree of brain region activation and the threshold size. This includes constructing a brain function connectivity network to determine the subject's current brain activity level and the neural activity and connectivity (i.e., correlation) between brain regions, thereby determining the specific rehabilitation status of each brain region, and constructing a brain effect connectivity network to determine the causal relationship between the interactions between the user's brain regions. At the same time, the subject's current status is quantitatively assessed to obtain brain function assessment results.

[0045] First, brain function data of autistic children performing current rehabilitation training tasks was acquired, and brain function data of autistic children in a resting state was pre-collected. Specifically, before each data collection, information such as the user's age, gender, height, and weight was recorded. The user collected brain function data by wearing a near-infrared device. After wearing the device, the experiment was conducted, which consisted of two parts: a resting state and a task state. First, near-infrared data was collected from the subject in a quiet environment for 20 minutes. This data represents the subject's resting state data, and the subject remained in a relaxed state with eyes open and seated throughout. After a 5-minute rest, the subject performed the training task as required. The near-infrared data recorded at this time represents the task state data. During this recording process, the user's behavioral data was simultaneously collected: HoloLens 2 (an existing wearable mixed reality device) was turned on to capture hand movement data, and the data collection time was also 20 minutes. The training task uses fNIRS (functional near-infrared spectroscopy) to monitor brain oxygenation signals. This signal provides feedback on the real-time state of the brain during training and reflects changes in the oxygenated hemoglobin level in the prefrontal cortex. It is used to monitor changes in different brain regions, including cognitive, memory, language, motor, and emotional modules, when patients perform tasks.

[0046] Secondly, the cerebral oxygenation signal undergoes preprocessing, which includes smoothing, missing value imputation, and noise reduction. Specifically, the preprocessing removes outliers, physiological noise, and motion artifacts from the acquired data to facilitate subsequent coherence analysis between measurement channels and brain functional connectivity analysis. In this embodiment, time windows and moving averages are used to smooth the signal; then, spline interpolation is used to imput missing parts of the data to maintain signal continuity; finally, a Butterworth filter is used to remove high-frequency noise, effectively removing noise components above the characteristic frequencies of cerebral hemodynamics.

[0047] Then, based on the preprocessed signal data, wavelet phase coherence (WPCO) was used to construct the brain functional connectivity network, and Granger causality (GC) was used to construct the brain effect connectivity network.

[0048] For brain functional connectivity networks, wavelet transform is performed on preprocessed signal data to extract dynamic information features of the phase and amplitude of oscillating components, transforming the signal from the time domain to the frequency domain to obtain the main components of the brain blood oxygenation signal time series in the frequency domain, which are used to analyze local intermittent oscillations in the time series. The activation level of brain regions is calculated using the amplitude after wavelet transform (i.e., the absolute value of the coefficients obtained from the wavelet transform). Furthermore, wavelet phase coherence analysis is used to construct brain functional connectivity networks, which can quantitatively calculate the degree to which the instantaneous phase between two time series signals remains consistent throughout the process, thereby assessing the correlation between the two time series signals. Here, the two time series signals refer to the time series signals acquired by each pair of acquisition channels of the infrared device. The calculation process of wavelet phase coherence between the data acquired by the two near-infrared device acquisition channels is as follows: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] and After continuous wavelet transform, it reaches a certain frequency and time The corresponding instantaneous phases at time are respectively and Calculate the instantaneous phase difference between the two. Then and Averaging is performed in the time domain, resulting in the two signals at different frequencies. The wavelet phase coherence value is: These two signals at all frequencies The wavelet phase coherence value (WPCO) between two signals is obtained by averaging the wavelet phase coherence values. Using the brain regions where the electrodes collecting brain function data are located as nodes, the connections between nodes as edges, and the inter-regional phase coherence (i.e., the average of the wavelet phase coherence values ​​of all channel pairs between two regions) as weights, brain functional connectivity networks are constructed for both resting and task states. Based on all constructed brain functional connectivity networks, the synergistic effects between different brain regions can be analyzed, i.e., the neural activity and connectivity of brain function.

[0049] For brain effect connectivity networks, Granger causality is used to calculate effective connectivity (EC) within brain regions, i.e., to construct brain effect connectivity networks and reveal the directionality of information transmission between brain regions. This approach suggests that if a time series... Cause time series The occurrence or impact If it has an impact, then it is relevant. Information and knowledge can predict The future value. Specifically, firstly, the time series signal of any channel is sequentially tested for stationarity with other channels. If two series fail the stationarity test, a cointegration test is performed. Stationarity of a time series means that over a period of time, the mean and variance of the time series do not change systematically (no trend), and periodic variations are strictly eliminated. Next, a cointegration test is performed. The cointegration test of time series is used to analyze the long-term equilibrium relationship between variables. In the process of cointegration analysis of two variables, if the independent and dependent variables are cointegrated, it can be confirmed that these two variables will not produce spurious regression results and that there is a long-term stable relationship between them. Finally, after the above tests are passed, the Granger causality relationship between the series is analyzed, i.e., two series are selected. and sequence Establish an autoregressive model; respectively in the series and sequence By incorporating past values ​​of another sequence into the prediction, a bivariate autoregressive model is obtained; assuming the two sequences are Granger causally related to each other, a... The hypotheses are tested and verified. Specifically, the quantification of Granger causality is calculated based on the prediction errors in the autoregressive and bivariate autoregressive models. ,in, and These represent the prediction errors in the autoregressive model and the bivariate autoregressive model, respectively; the series... and sequence Input the Granger causality test function. If the test result is greater than or equal to a set threshold, the Granger causality quantization value is binarized to 1; otherwise, it is binarized to 0. Repeat the above steps until the Granger causality quantization values ​​between any two sequences are binarized, thus obtaining the brain effect connectivity network.

[0050] Based on the constructed brain functional connectivity network and brain effect connectivity network, the effectiveness of the current rehabilitation is assessed according to their saliency level. Simultaneously, the EEGNet model is used as the rehabilitation scoring model. EEG signals are input into the pre-trained EEGNet model, and the brain function assessment results are output. In this embodiment, the EEGNet model is used for training. EEGNet is a general and compact convolutional neural network designed for specific general EEG recognition tasks. EEGNet consists of four layers: the first layer is a convolutional layer used to simulate bandpass filtering operations on the data of each channel; the second layer is a spatial filtering layer that weights the data of each channel through depthwise convolution; the third layer is an independent convolutional layer for extracting category information; and the fourth layer is a fully connected layer for classification. Furthermore, the above-mentioned EEGNet model can effectively assess the state of the subjects based on their fNIRS data (i.e., brain oxygenation signals). Its training method involves obtaining a test set of data including fNIRS data and corresponding percentage scores of different test subjects, and training for 500 iterations to obtain a rehabilitation scoring model based on fNIRS data.

[0051] As another implementation method, a one-way ANOVA is used to quantitatively assess the current status of the subjects by analyzing the influence of resting-state and task-state conditions on brain representation, providing guidance for rehabilitation intervention strategies and verifying the consistency between brain function data and behavioral and scale assessment results. In this embodiment, the comprehensive assessment is divided into two parts. The brain network assessment is completed through the above-mentioned part, and a comprehensive score is obtained by combining brain data, behavioral data, and scales to obtain the final comprehensive assessment result. Among them, the brain network assessment result has a higher priority and is used to optimize the comprehensive score, adjusting the final comprehensive assessment result to obtain the final result, including: when the significance level p < 0.05, it is considered statistically significant, and the comprehensive assessment result is +5; when the significance level p >= 0.05, it is considered not statistically significant, and the comprehensive assessment result is -5, where the comprehensive assessment result = subject's final score - threshold. Preferably, Pearson correlation and Bonferroni correction p-values ​​can be used to analyze the correlation between scale results, behavioral indicators, and brain function data to verify the consistency of the assessment.

[0052] (2) Regarding behavioral data

[0053] In this embodiment, HoloLens2 is used to collect behavioral data of the subject during the completion of the rehabilitation training task, namely images of the subject's hand movements, to obtain the three-dimensional coordinates of the hand joints when the subject performs the rehabilitation training task, and then calculate the hand movement score to obtain the behavioral data evaluation result.

[0054] Specifically, behavioral data from test subjects contains information such as individual behavioral performance, behavioral patterns, behavioral reaction time, behavioral choices, and behavioral preferences. This data helps to understand an individual's behavioral characteristics, habits, and changes, thereby revealing their psychological characteristics, cognitive processes, and emotional states. Behavioral data from the HoloLens 2 during the completion of rehabilitation training tasks was collected. The positional information of the metacarpophalangeal joints (MCP), proximal interphalangeal joints (PIP), and distal interphalangeal joints (DIP) was extracted. Based on the joint positional information, the joint flexion angle, wrist dorsiflexion angle, wrist rotation angle, and Euclidean distance of the joint points were calculated as task assessment parameters. The total active range of motion of the hand, i.e., finger flexion-extension TAM, was calculated using the formula TAM = flexion degree (MCP1 + PIP1 + DIP1) - extension deficiency degree (MCP2 + PIP2 + DIP2). Simultaneously, the wrist rotation angle ROM and wrist dorsiflexion angle ROM were calculated. These indicators were used as hand function rehabilitation indicators to assess the degree of hand function rehabilitation, thus obtaining the behavioral data assessment results. Among them, the above-mentioned straightening deficiency degree and buckling degree are the sum of MCP, PIP, and DIP angles in two different states.

[0055] Furthermore, the assessment criteria for TAM are as follows: Excellent, normal flexion and extension range, TAM > 220°; Good, function is above 75% of the healthy finger, TAM 200~220°; Average, function is 50%~75% of the healthy finger, TAM 180~200°; Poor, function is below 50% of the healthy finger, TAM < 180°. Specifically, the TAM score for finger flexion and extension is 100 points for 220°, 0 points for 120°, with 1 point for each degree; the score for wrist rotation angle (ROM) is 100 points for 160°, 0 points for 60°, with 1 point for each degree; the score for wrist dorsiflexion angle (ROM) is 100 points for 140°, 0 points for 40°, with 1 point for each degree.

[0056] (3) Regarding subjective scale data

[0057] In this embodiment, after each round of rehabilitation training tasks, the subjects will complete a subjective scale for subjective evaluation (assisted completion). That is, after completing each rehabilitation training task, the subjects will be presented with a self-assessment questionnaire to collect their subjective feelings during the performance of the rehabilitation training tasks. Based on the obtained subjective scale data, the training performance results are quantified to obtain the subjective scale data evaluation results.

[0058] Specifically, for the subjective scale data of the subjects, the training performance results of the subjects are evaluated based on the scale results, and the scores are converted into a percentage system, including:

[0059] ① When the elbow is at 90 degrees and the shoulder is at 0 degrees, wrist dorsiflexion is achieved: 0 points: almost no wrist dorsiflexion is possible; 1 point: wrist dorsiflexion can be completed, but not against resistance; 2 points: wrist dorsiflexion can still be maintained with some slight resistance.

[0060] ② When the elbow joint is at 90 degrees and the shoulder joint is at 0 degrees, flexion and extension of the wrist: 0 points: cannot move voluntarily; 1 point: cannot move the wrist joint within the full range of motion; 2 points: can perform the movement smoothly and without pause.

[0061] ③ When the elbow joint is at 0 degrees and the shoulder joint is at 30 degrees, the wrist is dorsiflexed: the scoring is the same as item ①;

[0062] ④ When the elbow joint is at 0 degrees and the shoulder joint is at 30 degrees, wrist flexion and extension: scoring is the same as item ②;

[0063] ⑤ Grip strength 1: With metacarpophalangeal joints extended and proximal and distal interphalangeal joints flexed, test the resistance grip strength: 0 points: unable to maintain the required position; 1 point: weak grip strength; 2 points: able to resist considerable resistance to grip.

[0064] ⑥ Grip strength 2: With all joints in the 0 position, thumb adduction: 0 points: cannot perform; 1 point: can pinch a piece of paper with the thumb, but cannot resist the pull; 2 points: can resist the pull.

[0065] ⑦ Grip strength 3: The patient can hold a pen between their thumb and index finger: the scoring method is the same as grip strength 2;

[0066] ⑧ Grip strength 4: The patient can grasp a cylindrical object: the scoring method is the same as grip strength 2;

[0067] ⑨ Grip strength 5: The patient can grasp a spherical object: The scoring method is the same as grip strength 2;

[0068] ⑩ Current training task difficulty: 0 points: easy; 1 point: medium; 2 points: difficult.

[0069] After completing the above scoring calculations, the brain function data assessment results, behavioral data assessment results, and training performance results (i.e., subjective scale data assessment results) are obtained. Each subject receives three scores, which are then weighted and averaged to calculate the final score, which serves as the comprehensive evaluation result. In this embodiment, the brain function data, behavioral data, and training performance result scores account for 40%, 30%, and 30% of the total score, respectively. Whether the score exceeds a certain set threshold is used to determine whether the subject's training effect has improved. Preferably, the score can be further optimized using the aforementioned brain network assessment results to obtain the final evaluation result.

[0070] In step S4 above, based on the rehabilitation training assessment results, the difficulty level of the current rehabilitation training task is adaptively adjusted for the next round of rehabilitation training. Specifically:

[0071] When the comprehensive assessment result is much higher than the threshold, specifically when the subject's final score minus the threshold is ≥10, the training will be adjusted to target rehabilitation training scenarios with more complex gestures and then conducted again.

[0072] When 5 < subject's final score - threshold < 10, the difficulty of the rehabilitation training scenario is increased and the training is repeated.

[0073] When the comprehensive evaluation result is close to the threshold, specifically 0 ≤ subject's final score - threshold ≤ 5, the current task is repeatedly trained.

[0074] When the comprehensive assessment result is below the threshold, specifically -5 < subject's final score - threshold < 0, the difficulty of the rehabilitation training scenario is reduced and the training is repeated. Rehabilitation gloves can be turned on to assist the patient in rehabilitation.

[0075] When the comprehensive assessment result is far below the threshold, specifically when the subject's final score minus the threshold is ≤ -5, the training is adjusted to focus on simpler gestures in rehabilitation training scenarios, and rehabilitation gloves are used to assist the patient in rehabilitation.

[0076] As another implementation method, brain function data, behavioral data, and subjective scale data collected by the user are uploaded to a cloud server. The cloud server performs a comprehensive evaluation and remotely transmits the rehabilitation training evaluation results to the doctor's end. Based on the doctor's end, a rehabilitation training adjustment strategy is uploaded to the cloud server, which then feeds back to the user's end. The user's end adjusts the difficulty level of the current rehabilitation training task according to the feedback rehabilitation training adjustment strategy and proceeds to the next round of rehabilitation training. In this embodiment, the difficulty level can be automatically adjusted based on the score, or the task can be adjusted based on the doctor's suggestions. Preferably, the doctor's rehabilitation training adjustment strategy has a higher priority than the patient's automatic adjustment strategy.

[0077] Furthermore, in this embodiment, rehabilitation training tasks of different difficulty levels are set under different modes, including a daily mode and a game mode, such as... Figure 2 As shown, the daily mode can provide different rehabilitation training programs based on the user's age and degree of hand dysfunction, and adaptively adjust the training difficulty based on the comprehensive assessment results. Moreover, when the glove assistance mode is turned on, the gloves are activated to assist the user in completing the training tasks. The game mode can provide gesture music games in combination with music therapy, and adaptively adjust the game difficulty based on the comprehensive assessment results. Similarly, when the glove assistance mode is turned on, the gloves are activated to assist the user in completing the training tasks.

[0078] The aforementioned daily routine encompasses various scenarios from daily life, providing different rehabilitation training tasks tailored to the user's age and degree of hand dysfunction. During this training, the current training task scenario is mapped into an augmented reality environment, attracting the user's attention with a blended virtual and real cartoon visual. Voice commands provide guidance and repeated training when the user is able to accept instructions. Throughout this process, the system tracks the user's hand movements, identifies hand postures to obtain joint position information, and calculates the total active range of motion of the hand. The rehabilitation outcome is assessed by combining task completion, behavioral data, brain function data, and subjective scales, and the difficulty of training is adjusted based on the comprehensive assessment results. During training, the assistive glove's haptic feedback actuator provides corresponding haptic feedback. When the glove's assistive mode is activated, the glove initiates corresponding interactions (such as assisting the user in grasping objects) when the subject's hand approaches an object. Throughout the training process, if the subject experiences any psychological or physiological discomfort, the training can be terminated at any time. For example, speech recognition technology can understand and respond to the user's verbal commands, such as terminating training, adjusting the training difficulty, changing the training scenario, etc., enabling interaction and communication with the user. The system can also perceive the user's emotional state through changes in facial expressions and voice, and react accordingly: when the system detects the user's low mood, it will reduce the difficulty of the current training task or terminate the training; when the system detects the user's high mood, it will increase the difficulty of the current training task. All information regarding the user's facial expressions and voice is acquired by HoloLens2.

[0079] Furthermore, to avoid a monotonous rehabilitation training environment, this embodiment sets up various rehabilitation training tasks of different difficulty levels in different scenarios for the daily mode. Specifically, different training scenarios such as a restaurant, bedroom, and beach are designed. Through different interactive tasks in the scenarios, users are prompted to complete six single hand fine motor skills and their combinations, including thumb-to-finger, fingertip-to-finger, three-finger pinch, cylindrical grip, side pinch, and fist. Each training action that meets the standard can be directly converted into points. Completing all tasks in a level allows users to enter the next challenge or refresh the score for the current level.

[0080] Scene 1 is themed around a restaurant and includes models of tables, plates, kettles, cups, fruits, and cakes. Users must complete tasks within a specified time, including: picking up fruits scattered on the table, sorting them, and placing them in a designated fruit plate. If the user's placement exceeds the indicated area, the timer resets, and the indicated area turns red; otherwise, it turns green. Next, pour water from the kettle into a cup. If the water falls beyond the indicated area, the timer resets, and the indicated area turns red; otherwise, it turns green. Finally, cut the cake along the dividing line. If the user's cutting point is far from the dividing line, the timer resets, and the dividing line turns red; otherwise, it turns green.

[0081] Scene Two is themed around a bedroom and includes models of toys, drawing boards, and paintbrushes. Users need to complete tasks within a specified time, including: picking up toys that have fallen to the ground and throwing them into a box; if the user's throwing point exceeds the warning line, the timer resets and the warning line turns red; stringing beads, connecting beads of different colors and shapes, and guiding users to use certain patterns to present changes in the bead string; and drawing patterns that include straight lines, origami, curves, or circles.

[0082] Scene 3 is themed around a beach and includes models of seashells, crabs, and small fish. Users need to complete tasks within a specified time, including: picking up seashells scattered on the beach, sorting them, and placing them in a designated bucket. If the user's placement exceeds the indicated area, the timer resets, and the indicated area turns red; otherwise, it turns green. They also need to use a fishing net to scoop up crabs and small fish from the seawater and pour them into the bucket. If the user's placement exceeds the indicated area, the timer resets, and the indicated area turns red; otherwise, it turns green.

[0083] As another implementation method, rehabilitation training tasks are personalized based on the user's age and the degree of hand dysfunction, providing tasks of different difficulty levels for different scenarios. For example, for children aged 1-3 years or with mild hand dysfunction and autism, training methods such as stringing beads, throwing objects, flipping over objects, grasping, and rubbing are used; for children over 4 years old or with severe hand dysfunction, training methods such as drawing straight lines, folding paper, drawing curves, or circles are used. Rehabilitation training tasks are divided into three levels of difficulty—beginner, intermediate, and advanced—based on the fineness of hand movements. The difficulty is adaptively adjusted based on comprehensive assessment results, while the range of motion, speed, and tactile stimulation of the rehabilitation gloves are also adjusted. Specifically, the difficulty adjustment plan for Scenario 1 includes: adjusting the type, quantity, and size of fruits; adjusting the size of the cup opening; and adjusting the number of dividing lines. The difficulty adjustment plan for Scenario 2 includes: adjusting the number and size of toys; adjusting the type, quantity, and size of beads; and adjusting the required accuracy of line drawing. The difficulty adjustment plan for Scenario 3 includes: adjusting the type, quantity, and size of seashells; adjusting the size of the fishing net; and adjusting the type, quantity, and size of crabs and small fish.

[0084] The above rehabilitation training tasks can provide targeted training for different key parts of the hand, including:

[0085] Finger interphalangeal joint flexion training setup: In scenarios one, two, and three, the user's proximal interphalangeal joints bend when performing a grasping action. Repeatedly grasping objects trains this proximal interphalangeal joint flexion. The design detects the current interphalangeal joint flexion angle of the five fingers and the characteristic θ value of this angle when grasping an object. The θ value is set to be greater than the normal interphalangeal joint flexion angle (90°) for a successful operation. The Object Manipulator component of MRTK (Mixed Reality Toolkit) is used to make the object movable, scalable, and rotatable to facilitate grasping. During the game, the user needs to grasp objects multiple times, thus requiring repeated interphalangeal joint flexion movements. To a certain extent, this improves the user's finger interphalangeal joint flexion ability, achieving the goal of training the interphalangeal joints.

[0086] Wrist extension and rotation function training settings: In scenario two, the user needs to throw a grasped object into a moving box. A conditional judgment is included in the design: the wrist extension angle θ is greater than the normal wrist extension angle by 20° before the system can perform the throwing operation. The user trains wrist extension function by repeatedly throwing objects. In scenario one, the user needs to pour water from a kettle into a cup. A conditional judgment is included in the design: the operation is considered successful only if the wrist rotation angle θ is greater than the normal wrist extension angle by 60° before the system can perform the pouring operation. In scenario three, the user needs to pour crabs and small fish from a fishing net into a bucket. A conditional judgment is included in the design: the wrist rotation angle θ is greater than the normal wrist extension angle by 60° before the system can perform the pouring operation. The user trains wrist rotation function by repeatedly pouring objects. In scenario two, the user needs to draw patterns containing straight lines, origami, curves, or circles. The user trains wrist extension and rotation function by repeatedly drawing different patterns.

[0087] Thumb-opposition function training setup: In scenario two, the user needs to string and connect beads of different colors and shapes. During the game, the operation is considered successful when the patient's thumb and forefinger touch the beads and the Euclidean distance between the thumb and forefinger is close to 0. While stringing beads, the user repeats the thumb-forefinger touch to restore thumb-opposition function.

[0088] The aforementioned game mode is themed around music games, automatically generating gesture models in sync with the music rhythm. These models include six single actions and their combinations: thumb-to-finger, fingertip-to-finger, three-finger pinch, cylindrical grip, side pinch, and fist. When the model reaches the judgment area, the user must make the corresponding gesture. The system captures the user's hand to obtain hand joint position information, calculates the gesture, and matches it with the generated model. During this training process, the current training task scene is mapped into an augmented reality scene. A cartoon-like visual style blending virtual and real elements attracts the user's attention. Voice commands provide guidance and repeated training when the user is able to accept instructions. Finally, rehabilitation results are assessed based on task completion, behavioral data, brain function data, and subjective scales. The difficulty of training is adjusted based on the comprehensive assessment results, including the gesture generation speed and the number of gestures. When the glove-assisted mode is activated, the generation point sends the generated information to the server via the communication unit while generating the gesture model, activating the rehabilitation glove to assist the user in making the corresponding gesture.

[0089] During the training process of the aforementioned game mode, the acquisition of behavioral data, brain function data, and subjective scale data is the same as that of the daily training mode.

[0090] Example 2

[0091] This embodiment provides a virtual-real integrated hand function rehabilitation training system for children with autism, such as... Figure 3 As shown, it includes:

[0092] The mapping unit is used to map the motion information and virtual scene corresponding to the current rehabilitation training task to the augmented reality scene, guiding autistic children to interact with the augmented reality scene;

[0093] The assistive unit is designed to help children with autism complete multiple rounds of current rehabilitation training tasks based on the flexible rehabilitation gloves they wear.

[0094] The data acquisition unit is used to collect brain function data, behavioral data, and subjective scale data of autistic children when they perform the current rehabilitation training task in a non-assisted state.

[0095] The assessment unit is used to conduct a comprehensive assessment based on the collected data to obtain rehabilitation training assessment results;

[0096] The adjustment unit is used to adaptively adjust the difficulty level of the current rehabilitation training task based on the rehabilitation training assessment results, so as to carry out the next round of rehabilitation training.

[0097] The system comprises the following components: a mapping unit, which includes augmented reality equipment to create a virtual-real integrated augmented reality scene; an auxiliary unit, which includes a flexible rehabilitation glove equipped with a tactile feedback mechanism; a data acquisition unit, which includes a near-infrared device, a camera, and a recording device, with the near-infrared device collecting brain function data, the camera collecting user behavioral data, and the recording device collecting sound data; an evaluation unit, which includes a cloud server, i.e., a rehabilitation training server, used to perform a comprehensive evaluation based on the collected data and obtain rehabilitation training evaluation results; and an adjustment unit, which includes a controller electrically connected to the flexible rehabilitation glove, used to adjust the difficulty level of the current rehabilitation training task according to the rehabilitation training evaluation results.

[0098] In addition, the rehabilitation training server is connected to augmented reality devices, flexible rehabilitation gloves, near-infrared devices, and camera devices.

[0099] The aforementioned augmented reality device, using HoloLens 2, selects rehabilitation training programs based on the user's age and degree of hand dysfunction, and builds a training environment that blends the virtual and real worlds to visually present training tasks to the user.

[0100] The aforementioned flexible rehabilitation gloves combine the biomimetic principle of finger skin folds and the Hall effect sensing mechanism. They adopt a biomimetic finger sleeve structure that integrates compliant movement and precise perception, and integrate a sensing system, an SMA drive and intelligent control system, a power supply and thermal control system, and a tactile feedback system.

[0101] The aforementioned near-infrared device allows patients to collect brain function data by wearing it. The near-infrared acquisition channels on the human brain include six brain regions: left prefrontal cortex (LPFC), right prefrontal cortex (RPFC), left motor cortex (LMC), right motor cortex (RMC), left occipital lobe (LOL), and right occipital lobe (ROL). The electrode positions are set according to the international standard 10 / 10.

[0102] The aforementioned camera device uses the built-in TOF depth camera of HoloLens2 to collect user behavioral data during the user's completion of rehabilitation training tasks, and uploads the collected data to the virtual-real fusion autism children's hand function rehabilitation training server; in addition, the camera device is also used to collect facial images of the user during the execution of training tasks, and uploads the collected image data to the virtual-real fusion autism children's hand function rehabilitation training server.

[0103] The aforementioned recording device collects the user's voice during the process of the user completing rehabilitation training tasks and uploads the collected data to a virtual-real integrated autism children's hand function rehabilitation training server.

[0104] Furthermore, the system also includes a recognition unit that interacts with the hand function rehabilitation training server for children with autism. This recognition unit is used to identify facial expressions and voice changes and speech information during the training process based on the received facial image data and sound data, using image recognition technology and speech recognition technology, and thus make corresponding responses.

[0105] Furthermore, such as Figure 3 As shown, the system also includes a communication unit, which is used to upload brain function data, behavioral data and subjective scale data obtained by the user terminal to the cloud server, and to upload the rehabilitation training adjustment strategy generated by the doctor terminal to the cloud server; it is also used to remotely transmit the rehabilitation training assessment results to the doctor terminal, and to feed back the rehabilitation training adjustment strategy to the user terminal.

[0106] The user end, or patient end, manages and controls data collection and rehabilitation training equipment; uploads subject brain function data, behavioral data, and subjective scale data to the server; and receives information from the doctor end to enable remote communication between doctors and patients.

[0107] On the doctor's end, the system receives assessment results from the server and adjusts the patient's rehabilitation plan accordingly; it also receives information from the patient's end to enable remote communication between doctors and patients.

[0108] The aforementioned rehabilitation training server comprehensively evaluates the subject's behavioral data, brain function data, and subjective scales. Based on the comprehensive evaluation results, it adaptively adjusts the subject's hand function rehabilitation training task plan and training difficulty using the control process described in Example 1. During training, the assistive glove activates a tactile feedback actuator to provide corresponding tactile feedback. In assistive mode, the rehabilitation glove assists the user in completing the task. While the user completes the rehabilitation training task, the server perceives the emotional state through changes in facial expressions and voice, and responds accordingly to different situations. Information from the patient's end is sent to the doctor's end, enabling remote communication between doctor and patient. After the user completes the rehabilitation training task, the training and evaluation results are sent to the doctor's terminal, and the server simultaneously receives information from the doctor's terminal to adjust the rehabilitation plan.

[0109] In this embodiment, a virtual-real fusion hand function rehabilitation training environment is constructed using augmented reality (AR) devices. Specifically, the user wears a near-infrared device and conducts an experiment after donning it. The experiment consists of two parts: a resting state and a task state. First, the user's resting state brain function indicators are collected. The acquired data is preprocessed, including removing outliers, physiological noise, and motion artifacts from the near-infrared data. After completing the resting state data collection, the subject wears an AR headset and the rehabilitation training device and begins training. The AR headset displays the rehabilitation training task, and the user selects the task difficulty before entering the current round of rehabilitation training. In assisted mode, gloves are activated to assist the user in completing the training task and provide corresponding tactile feedback. In non-assisted mode, the glove assistance function is disabled. Simultaneously, the near-infrared device collects the subject's brain function data in real time, a camera captures behavioral data of the subject performing the rehabilitation training task, and a recording device captures the user's voice. The system uses speech recognition technology to understand and respond to the user's language commands, enabling interaction and communication; it perceives emotional states through changes in facial expressions and voice, and reacts accordingly to different situations. After each round of rehabilitation training, the user completes a subjective questionnaire.

[0110] In this embodiment, the augmented reality training environment is developed using Unity+MRTK, and the project is deployed on the augmented reality headset, HoloLens2, for users to conduct rehabilitation training. The difficulty of the training program can be selected through button events or voice control events developed using Unity+MRTK. During the training process, HoloLens2 will provide real-time feedback on parameters such as training difficulty, task completion status, brain function data, and behavioral data. After completing each training task, the subject needs to fill out a subjective scale through button or voice control to provide feedback on their subjective feelings during the training process.

[0111] The aforementioned near-infrared device uses the NirSmart portable near-infrared brain functional imaging system. It uses the brain regions where the electrodes collecting data are located as nodes, resulting in a total of 6 nodes representing 6 brain regions. The connections between these nodes are used as edges, and the wavelet phase coherence between regions is calculated as the edge weight. In the NirSmart channels, taking the left prefrontal cortex (LPFC) and right prefrontal cortex (RPFC) as examples, the wavelet phase coherence result between the two regions is the average of the wavelet phase coherence values ​​for their respective channel pairs. This constructs a brain functional connectivity network to analyze the neural activity and connectivity between different brain regions. After each round of rehabilitation training, the collected task-time brain functional data and resting-state data are analyzed to determine the relationship between brain region activation levels and threshold values. This includes using the brain functional connectivity network to determine the patient's current brain activity level and the neural activity and connectivity between brain regions, thereby determining the specific rehabilitation status of each brain region, and using the brain effect connectivity network to determine the causal relationship between the interactions between brain regions and quantitatively assess the subject's current condition.

[0112] like Figure 4 As shown, the drive system of the flexible rehabilitation glove adopts a biomimetic structure design similar to a muscle-tendon, driven by a high power-to-weight ratio shape memory alloy (SMA) spring, which is placed in the arm assembly. A rope mechanism with multi-stage force amplification transmits the SMA drive deformation to the finger sleeve structure, enabling the fingers to perform flexion / extension movements. Furthermore, a joint sensing system based on Hall effect sensors is designed. By adapting to the non-uniform stiffness structure of the flexible finger sleeve, the interference of finger sleeve movement on the sensing system is reduced, achieving stable and accurate finger status perception.

[0113] Preferably, a haptic feedback system is added to the glove, including:

[0114] Vibration feedback is incorporated into the rehabilitation gloves by embedding voice coil actuators, enabling them to simulate realistic button clicks and impacts. A vibration motor is mounted in the palm to simulate large-amplitude impacts, while each fingertip has a vibration motor to simulate small-amplitude impacts.

[0115] Pressure feedback involves embedding inflatable pneumatic actuators in the lining of the rehabilitation glove. These actuators create precise pressure by inflating and deflating air bubbles, allowing the user's hand to feel the sensation of touching a real-world object. When the user grasps a virtual object, the pneumatic actuators in their fingers inflate to harden and simulate resistance, while simultaneously pulling on the skin to simulate the effect of gravity.

[0116] Texture feedback weaves numerous dynamic haptic feedback actuators into a flexible material at the fingertips, aiming to provide users with a high-resolution touch directly delivered to each fingertip. Each fingertip haptic array contains bubble-like "pixels" containing fluid that, when activated, fills and stretches the bubbles; each pixel is a dedicated, electrically controlled pump, only a few hundred micrometers thick. These pumps contain no moving parts and operate on the principle of electroosmosis, directly attracting electrical charges within the fluid to cause its flow. Figure 5 As shown, a large number of "pixels" are adapted to objects of various shapes, sizes and textures, thereby providing personalized tactile feedback and stimulating the nerves in the fingertips to assist in treatment.

[0117] The steps and methods involved in the above embodiment two correspond to those in embodiment one. For specific implementation details, please refer to the relevant description section of embodiment one.

[0118] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0119] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention has been described in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the present invention.

Claims

1. A virtual-real integrated method for hand function rehabilitation training in children with autism, characterized in that, include: The movement information and virtual scene corresponding to the current rehabilitation training task are mapped to the augmented reality scene to guide children with autism to interact with the augmented reality scene; The flexible rehabilitation gloves worn by children with autism help them complete multiple rounds of current rehabilitation training tasks. In a non-assisted state, brain function data, behavioral data, and subjective scale data of autistic children are collected when they perform the current rehabilitation training task. A comprehensive assessment is conducted based on the collected data to obtain the rehabilitation training assessment results. Based on the rehabilitation training assessment results, the difficulty level of the current rehabilitation training task is adaptively adjusted, and the next round of rehabilitation training is carried out. The comprehensive evaluation based on the collected data to obtain rehabilitation training evaluation results includes: Based on the collected brain function data and the pre-collected resting state brain function data, the activation level of different brain regions is obtained, and brain function assessment results are obtained. Based on the collected behavioral data, hand movement scores are calculated to obtain behavioral data evaluation results; Based on the obtained subjective scale data, the training performance results are quantified to obtain the subjective scale data evaluation results; The final rehabilitation training assessment result is obtained by weighting the results of brain function assessment, behavioral data assessment, and subjective scale data assessment. The daily fine motor skills of the hands include thumb-to-finger, fingertip-to-finger, three-finger pinch, cylindrical grip, lateral pinch, and fist clenching. Behavioral data was collected from subjects during the completion of rehabilitation training tasks using HoloLens 2. Based on the collected behavioral data, hand movement scores were calculated to obtain behavioral data assessment results, including: Based on behavioral data, the position information of the metacarpophalangeal joint (MCP), proximal interphalangeal joint (PIP), and distal interphalangeal joint (DIP) of the hand during task execution was extracted. Based on the extracted joint position information, the joint flexion angle, wrist extension angle, wrist rotation angle, and Euclidean distance of the joint are calculated. Then, the total active range of motion of the hand, wrist rotation range (ROM), and extension angle (ROM) are calculated. At the same time, the hand joint movement velocity is calculated based on the changes of the extracted joint position information per unit time. Among them, the total active range of motion of the hand, i.e., finger flexion-extension TAM, is calculated by the formula TAM = flexion degree (MCP1 + PIP1 + DIP1) - extension deficiency degree (MCP2 + PIP2 + DIP2). Based on the total active range of motion of the hand, wrist joint rotation, dorsiflexion angle, and hand joint movement speed, the degree of hand function rehabilitation and hand movement rate are assessed, and behavioral data assessment results are obtained. Specifically, based on the collected brain function data and the pre-collected resting-state brain function data, the activation levels of different brain regions are obtained to acquire brain function assessment results, including: The brain function data of autistic children during the current rehabilitation training task is acquired, and brain function data of autistic children in the resting state is pre-collected; the brain function data includes cerebral blood oxygenation signals in the prefrontal cortex; The cerebral blood oxygenation signal is preprocessed; the preprocessing includes smoothing, missing value imputation, and noise reduction. Based on the preprocessed cerebral blood oxygenation signal, wavelet phase coherence analysis and Granger causality were used to construct brain functional connectivity network and brain effect connectivity network, respectively; at the same time, the preprocessed cerebral blood oxygenation signal was input into the pre-trained EEGNet model to output brain function assessment results. The effectiveness of the current rehabilitation training task is assessed based on the saliency level of brain functional connectivity networks and brain effect connectivity networks, and the final rehabilitation training assessment results are optimized based on the saliency level. The construction of the brain effect connectivity network is as follows: The time series signal of any channel is sequentially tested for stationarity with other channels. If two sequences fail the stationarity test, a cointegration test is performed. The cointegration test of the time series is used to analyze the long-term equilibrium relationship between variables. In the process of cointegration analysis of two variables, if the independent and dependent variables are cointegrated, it can be confirmed that these two variables will not produce spurious regression results and that there is a long-term stable relationship between them. Finally, after the above tests are passed, the Granger causality relationship between the sequences is analyzed by selecting two sequences... and sequence Establish an autoregressive model, and respectively in the series and sequence By incorporating past values ​​of another sequence into the prediction, a bivariate autoregressive model is obtained; assuming the two sequences are Granger causally related to each other, the model is then used... To test and verify the hypothesis, the quantification value of Granger causality is calculated based on the prediction errors in the autoregressive and bivariate autoregressive models, as follows: , and These represent the prediction errors in the autoregressive model and the bivariate autoregressive model, respectively; the series... and sequence Input the Granger causality test function. If the test result is greater than or equal to the set threshold, the Granger causality quantization value is binarized to 1, otherwise it is binarized to 0. Repeat the above steps until the Granger causality quantization values ​​between any two sequences are binarized to obtain the brain effect connectivity network. This also includes: using one-way ANOVA to quantitatively assess the current status of subjects by evaluating the impact of resting and task states on brain representation, providing guidance for rehabilitation intervention strategies, and verifying the consistency between brain function data and behavioral and scale assessment results; brain network assessment results have higher priority and are used to optimize the comprehensive score, adjusting the final comprehensive assessment result to obtain the final result, including: when the significance level p < 0.05, it is considered statistically significant, and the comprehensive assessment result is +5; when the significance level p >= 0.05, it is considered not statistically significant, and the comprehensive assessment result is -5, where the comprehensive assessment result = subject's final score - threshold; using Pearson correlation and Bonferroni corrected p-value to analyze the correlation between scale results, behavioral indicators, and brain function data to verify the consistency of the assessment; The system senses the user's emotional state through changes in facial expressions and voice, and reacts accordingly to different situations: when it senses that the user is in a low mood, it will reduce the difficulty of the current training task or terminate the training; when it senses that the user is in a high mood, it will increase the difficulty of the current training task. The user's facial expressions and voice information are all acquired by HoloLens2.

2. The virtual-real integrated hand function rehabilitation training method for autistic children as described in claim 1, characterized in that, Brain function data, behavioral data, and subjective scale data collected by the user are uploaded to the cloud server, which then conducts a comprehensive evaluation and remotely transmits the rehabilitation training evaluation results to the doctor. Based on the doctor's end, the rehabilitation training adjustment strategy is uploaded to the cloud server, which then feeds back to the user's end. The user's end adjusts the difficulty level of the current rehabilitation training task according to the feedback of the rehabilitation training adjustment strategy, and proceeds to the next round of rehabilitation training.

3. The virtual-real integrated hand function rehabilitation training method for autistic children as described in claim 1, characterized in that, Based on the acquired subjective scale data, the training performance results are quantified to obtain the subjective scale data evaluation results, including: Based on the rotation angles of the elbow, shoulder, and wrist joints and the grip strength in the subjective scale data, the training performance results are quantified, and the subjective scale data evaluation results are obtained.

4. The virtual-real integrated hand function rehabilitation training method for autistic children as described in claim 1, characterized in that, Set up rehabilitation training tasks of different difficulty levels under different modes; The modes include a daily mode and a game mode. The daily mode includes rehabilitation training tasks of different difficulty levels in different scenarios.

5. A virtual-real integrated hand function rehabilitation training system for autistic children, employing the virtual-real integrated hand function rehabilitation training method for autistic children as described in any one of claims 1-4, characterized in that, include: The mapping unit is used to map the motion information and virtual scene corresponding to the current rehabilitation training task to the augmented reality scene, guiding autistic children to interact with the augmented reality scene; The assistive unit is designed to help children with autism complete multiple rounds of current rehabilitation training tasks based on the flexible rehabilitation gloves they wear. The data acquisition unit is used to collect brain function data, behavioral data, and subjective scale data of autistic children when they perform the current rehabilitation training task in a non-assisted state. The assessment unit is used to conduct a comprehensive assessment based on the collected data to obtain rehabilitation training assessment results; The adjustment unit is used to adaptively adjust the difficulty level of the current rehabilitation training task based on the rehabilitation training assessment results, so as to carry out the next round of rehabilitation training.

6. The virtual-real integrated hand function rehabilitation training system for autistic children as described in claim 5, characterized in that, The mapping unit includes an augmented reality device for building an augmented reality scene that blends the virtual and real worlds; The auxiliary unit includes a flexible rehabilitation glove, which is equipped with a tactile feedback mechanism. The acquisition unit includes a near-infrared device, a camera device, and a recording device. The near-infrared device is used to acquire brain function data, the camera device is used to acquire user behavioral data, and the recording device is used to acquire sound data. The assessment unit includes a cloud server, which is used to perform a comprehensive assessment based on the collected data and obtain rehabilitation training assessment results; The adjustment unit includes a controller electrically connected to the flexible rehabilitation glove, used to adjust the difficulty level of the current rehabilitation training task based on the rehabilitation training assessment results.

7. The virtual-real integrated hand function rehabilitation training system for autistic children as described in claim 6, characterized in that, It also includes a communication unit, which is used to upload brain function data, behavioral data and subjective scale data obtained by the user terminal to the cloud server, and to upload the rehabilitation training adjustment strategy generated by the doctor terminal to the cloud server; it is also used to remotely transmit the rehabilitation training assessment results to the doctor terminal, and to feed back the rehabilitation training adjustment strategy to the user terminal.

Images