Multi-sensory cooperation man-machine interaction method and system for motion cognition combined rehabilitation training

Abstract

The invention relates to the technical field of virtual reality and medical rehabilitation, and provides a multi-sensory collaborative man-machine interaction method and system for motion cognition combined rehabilitation training. According to the method, the attention level, the fixation point position, the arm motion vector and the gesture state of a subject are quantized by collecting and analyzing electroencephalogram, near infrared, eye movement, optical marking, myoelectricity and other multi-mode physiological signals of the subject so as to represent the motion and cognitive intention of the subject. On the basis of the structured result, interaction instructions and performance data for controlling locking, lifting, moving, grabbing and releasing of objects in the virtual rehabilitation environment are generated according to composite triggering conditions; and finally, according to the instruction execution and the internal state of the virtual environment, determining an interaction event, synchronously driving a motion platform and a visual display and sound system to provide multi-channel fusion feedback, and according to performance data, adaptively adjusting the training difficulty. Multi-modal physiological signals are adopted, the immersion feeling of a patient in the rehabilitation training process is enhanced, the understanding of tasks is improved, and combined rehabilitation of the motion cognition function is achieved.

Description

Technical Field

[0001] This invention relates to the fields of virtual reality, medical rehabilitation, and human-computer interaction, and in particular to a multi-sensory collaborative human-computer interaction method and system for motor cognition combined rehabilitation training. This method is particularly suitable for rehabilitation training of patients with stroke, nerve injury, or movement disorders, improving patients' immersion and participation, thereby enhancing the effect of rehabilitation training. Background Technology

[0002] With the development of modern medicine and technology, rehabilitation therapy has evolved from simple physical therapy to multi-dimensional training that integrates psychological and cognitive aspects. Traditional rehabilitation training methods typically rely on physical equipment and simple exercises, which can help patients restore physical function to some extent, but still have limitations in improving cognitive function and concentration. Especially for patients with nerve damage, physical rehabilitation training alone cannot effectively improve their attention, reaction ability, and other cognitive functions. Therefore, how to organically combine cognitive training with physical rehabilitation has become an important issue in the field of rehabilitation medicine.

[0003] Current rehabilitation training methods primarily rely on the guidance of physical therapists, typically focusing on passive movement recovery, where patients perform passive exercises with the therapist's assistance. While this approach can help patients gradually regain some motor function, the limited active participation of patients often results in a slow rehabilitation process, failing to fully motivate them to engage in self-training. Furthermore, passive training lacks real-time feedback and personalized adjustments, making it difficult to dynamically adjust according to the patient's actual performance, thus failing to maximize rehabilitation effectiveness.

[0004] With the rapid development of brain-computer interface (BCI) technology, interactive rehabilitation training programs combining multi-sensory input have gradually attracted widespread attention. BCI can monitor a patient's brain activity in real time by collecting and analyzing neural activity signals such as electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), and translate this into control commands for external devices. This technology allows patients to participate in rehabilitation training through attention and intention, thereby improving their initiative and sense of involvement. Furthermore, interactive systems combining electromyography (EMG) signals with visual tracking technologies (such as eye tracking) can further enhance patients' motor control abilities in virtual environments, particularly in the manipulation of virtual objects and path planning, effectively improving hand-eye coordination, motor accuracy, and cognitive function.

[0005] However, many interactive rehabilitation systems still have some shortcomings. On the one hand, most systems lack comprehensive patient perception, often relying on single biosignal inputs such as electroencephalogram (EEG), electromyography (EMG), or gesture recognition signals, making it difficult to comprehensively assess a patient's motor performance and cognitive state. This single feedback mechanism limits the system's responsiveness, hindering the full effectiveness of rehabilitation training. On the other hand, existing systems have relatively simple feedback modes, often limited to visual or auditory feedback, lacking multi-sensory collaborative input-output design, making it difficult to provide patients with an immersive and comprehensive rehabilitation experience. This limitation prevents patients from receiving sufficient sensory stimulation and training challenges during rehabilitation, affecting the comprehensive recovery of cognitive attention and motor function. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a multi-sensory collaborative human-computer interaction method and system for motor cognitive rehabilitation training. This method utilizes electroencephalogram (EEG), functional near-infrared spectroscopy (fNIRS), eye tracking, marker localization, and EMG signals to identify the attention level and motor state of human subjects. This information serves as input for interaction with a virtual reality game environment. Based on game rules and interaction logic, it provides subjects with multi-sensory collaborative feedback (visual, auditory, tactile, and bodily) through VR devices, an audio system, and a motion platform. This enhances the patient's immersion during rehabilitation training, improves task comprehension, and achieves integrated rehabilitation of motor cognitive functions.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A multi-sensory collaborative human-computer interaction method for motor cognitive rehabilitation training includes the following steps:

[0009] S1. Acquire multimodal physiological signals of the subject during rehabilitation training, including electroencephalogram (EEG), functional near-infrared spectroscopy (fNIRS), eye tracking signals, marker localization signals, and forearm electromyography (EMG).

[0010] S2. Process and analyze the multimodal physiological signals to quantify the subject's real-time state and interaction intentions, and generate structured analysis results; the structured analysis results include attention level, fixation point position, arm motion vector, and gesture state;

[0011] S3. Execute the rehabilitation training task logic based on the structured analysis results, and generate corresponding interactive control instructions and task performance data; the interactive control instructions include at least operation instructions for controlling the locking, lifting, spatial movement, grasping and releasing of target objects in the virtual environment constructed for rehabilitation training.

[0012] S4. Based on the execution of interactive control commands and the internal state of the virtual environment, determine the interactive events in the virtual environment, and drive the motion platform, visual display system and audio system to provide synchronous multi-channel fusion feedback to the subject; and adaptively adjust the training difficulty according to the task performance data.

[0013] Furthermore, the specific implementation process of step S2 includes:

[0014] S21. Signal preprocessing: The acquired multimodal physiological signals are sequentially bandpass filtered to remove DC bias and motion noise, and downsampling is performed; wherein, the sampling rate of the EEG signal and functional near-infrared spectroscopy signal is reduced to a predetermined first frequency, the sampling rate of the eye tracking signal and marker positioning signal is reduced to a predetermined second frequency, and the sampling rate of the forearm electromyography signal is reduced to a predetermined third frequency.

[0015] S22. Signal Analysis: Based on the preprocessed multimodal physiological signals, the following analysis operations are performed in parallel:

[0016] a. Quantitative analysis of attention levels: This is achieved through at least one of the following methods:

[0017] The energy ratio of the Theta band to the Beta band is calculated based on EEG signals to obtain the attention level characterization value TBR.

[0018] Based on functional near-infrared spectral signals, the concentration change signals of HbO and HbR are calculated according to the Beer-Lambert law, and the concentration change signals are input into the long short-term memory network model to evaluate and output attention level values ​​in real time.

[0019] b. Calculate the gaze point position: Based on eye-tracking signals, calculate the subject's gaze point position in the VR environment;

[0020] c. Calculate the arm motion vector: Based on the marker point positioning signal, calculate the arm motion vector representing the arm direction by mapping the coordinate system with the marker point positioning signal;

[0021] d. Gesture pattern recognition: Based on forearm electromyography signals, the system segments the signals, extracts features, and inputs them into a long short-term memory network model to identify the subject's gesture state.

[0022] Furthermore, the generation of interactive control commands in step S3 depends on a composite triggering condition consisting of gaze point position, arm motion vector, gesture state, and attention level; wherein:

[0023] The locking and selection of the target object is triggered by the coincidence of the gaze point position and the arm motion vector.

[0024] Grasping and lifting a target object is a combination of conditions where the gesture state and the level of attention exceed a first threshold.

[0025] The release and placement of the target object are based on a combination of conditions, namely, the gesture state and the attention level exceeding the second threshold.

[0026] Furthermore, the rehabilitation training task logic in S3 is a sequential state machine process, including:

[0027] Object selection state: triggered by a combination of conditions, namely, the gaze point coincides with the target object and the arm motion vector intersects with the object;

[0028] Object grasping state: In the object selection state, it is triggered by a combination of conditions, namely the gesture state indicating a first predetermined gesture and the attention level exceeding a first threshold.

[0029] Region selection state: In the object grasping state, it is triggered by a combination of conditions: the gaze point coincides with the target region and the arm motion vector intersects with the region.

[0030] Object placement state: In the area selection state, it is triggered by a combination of conditions, namely, the gesture state indicates a second predetermined gesture and the attention level exceeds a second threshold.

[0031] Furthermore, the triggering of interactive feedback in step S4 is based on the real-time detection of specific interactive events in the virtual environment, including collisions between virtual objects and obstacles.

[0032] Furthermore, the interactive feedback includes haptic feedback provided by the motion platform, visual feedback provided by the visual display system, and auditory feedback provided by the audio system; and when the specific interactive event is triggered, the motion platform, visual display system, and audio system are configured to operate synchronously to provide the subject with fused multi-channel feedback for the same interactive event.

[0033] A multi-sensory collaborative human-computer interaction system for motor cognitive rehabilitation training includes: an interactive data acquisition module, an interactive data analysis module, a rehabilitation training interaction module, and an interactive training feedback module.

[0034] The interactive data acquisition module synchronously acquires multimodal physiological signals of the subject during rehabilitation training and sends them to the interactive data analysis module; wherein, the multimodal physiological signals include electroencephalogram (EEG) signals, functional near-infrared spectroscopy (fNIRS) signals, eye-tracking signals, marker localization signals, and forearm electromyography (EMG) signals.

[0035] The interactive data analysis module processes and analyzes the received multimodal physiological signals in real time, quantifies the subject's real-time state and interaction intention, and generates structured analysis results including attention level, fixation point position, arm movement vector and gesture state, which are then sent to the rehabilitation training interactive module.

[0036] The rehabilitation training interaction module executes rehabilitation training task logic based on the received structured analysis results and generates corresponding interactive control instructions and task performance data, which are then sent to the interactive training feedback module. The interactive control instructions include at least operation instructions for locking, lifting, spatially moving, grasping, and releasing target objects in the virtual environment constructed for rehabilitation training.

[0037] The interactive training feedback module includes a motion platform for providing haptic feedback, a non-immersive or immersive VR display system for providing visual and tactile feedback, and a sound system for providing auditory feedback. Based on the execution of interactive control commands and the internal state of the virtual environment, the interactive training feedback module determines interactive events in the virtual environment and drives the motion platform, visual display system, and sound system to provide synchronous multi-channel fused feedback to the subject. It also adaptively adjusts the training difficulty based on the task performance data.

[0038] Compared with existing technologies, this invention collects multimodal physiological signals, analyzes and processes them to generate interactive control commands, which are used to control the locking, lifting, spatial movement, grasping and releasing of target objects in a virtual environment, thereby constructing a precise and efficient motor cognitive rehabilitation training platform. This platform provides patients with an immersive interactive experience, accelerates the rehabilitation process, and strengthens patients' attention control, hand-eye coordination, and motor skills, helping patients achieve simultaneous recovery of cognitive and motor functions through progressively challenging training. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the interaction architecture of a multi-sensory collaborative human-computer interaction system as an example.

[0040] Figure 2 This is a schematic diagram illustrating a scenario where a subject is training on a motion platform in a multi-sensory collaborative human-computer interaction method as described in the embodiment.

[0041] Figure 3 This is a flowchart illustrating the analysis and processing of multimodal physiological signals in a multisensory collaborative human-computer interaction method, as described in this embodiment.

[0042] Figure 4 This is a logic diagram of a rehabilitation training task using a multi-sensory collaborative human-computer interaction method as an example. Detailed Implementation

[0043] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0044] like Figure 1 As shown, the present invention provides a multi-sensory collaborative human-computer interaction system for motor cognition combined rehabilitation training, comprising: an interactive data acquisition module 110, an interactive data analysis module 120, a rehabilitation training interaction module 130, and an interactive training feedback module 140.

[0045] The interactive data acquisition module 110 is used to synchronously acquire multimodal physiological signals of the subject during rehabilitation training and send them to the interactive data analysis module 120; wherein, the multimodal physiological signals include electroencephalogram (EEG) signals, functional near-infrared spectroscopy (fNIRS) signals, eye-tracking signals, marker localization signals, and forearm electromyography (EMG) signals.

[0046] The interactive data analysis module 120 is used to process and analyze the received multimodal physiological signals in real time, quantify the subject's real-time state and interaction intention, and generate structured analysis results including attention level, gaze point position, arm movement vector and gesture state, which are then sent to the rehabilitation training interactive module.

[0047] The rehabilitation training interaction module executes rehabilitation training task logic based on the received structured analysis results and generates corresponding interactive control instructions and task performance data, which are then sent to the interactive training feedback module. The interactive control instructions include at least operation instructions for locking, lifting, spatially moving, grasping, and releasing target objects in the virtual environment constructed for rehabilitation training.

[0048] The interactive training feedback module includes a motion platform for providing haptic feedback, a non-immersive or immersive VR display system for providing visual and tactile feedback, and a sound system for providing auditory feedback. Based on the execution of interactive control commands and the internal state of the virtual environment, the interactive training feedback module determines interactive events in the virtual environment and drives the motion platform, visual display system, and sound system to provide synchronous multi-channel fused feedback to the subject. It also adaptively adjusts the training difficulty based on the task performance data.

[0049] In some embodiments, the interactive data acquisition module 100 comprises an EEG signal acquisition unit 110, an fNIRS signal acquisition unit 110, an eye-tracking unit 130, an EMG signal acquisition unit 140, and a marker point positioning unit 150, wherein:

[0050] The EEG signal acquisition unit 110 mainly includes a 32-channel EEG cap and a 32-channel 16 kHz amplifier. The EEG cap is wired to the amplifier and transmits the acquired EEG data to the interactive data analysis unit 200.

[0051] The fNIRS signal acquisition unit 120 is a portable fNIRS device that transmits the acquired fNIRS signals from the prefrontal cortex of the subject to the interactive data analysis unit 200.

[0052] The eye-tracking unit 130 is a head-mounted eye tracker device that transmits the collected eye-tracking signals to the interactive data analysis unit 200.

[0053] The EMG signal acquisition unit 140 consists of a surface electromyography sensor attached between the radial and ulnar flexor carpi radialis muscles of the test subject, which acquires surface electromyography signals near the palmaris longus muscle and transmits them to the interactive data analysis unit 200.

[0054] The marker positioning unit 150 consists of two markers, M1 and M2, respectively attached to the upper side of the subject's right elbow joint and the upper side of the right wrist, and a motion capture system. It is used to collect the position data of markers M1 and M2, which are then transmitted to the interactive data analysis unit as marker positioning signals.

[0055] In some embodiments, the rehabilitation training interaction module is a rehabilitation training game.

[0056] Based on the above system, this embodiment also provides a multi-sensory collaborative human-computer interaction method for motor cognition combined rehabilitation training, including the following steps:

[0057] S1. Acquire multimodal physiological signals of the subject during rehabilitation training, including electroencephalogram (EEG), functional near-infrared spectroscopy (fNIRS), eye tracking signals, marker localization signals, and forearm electromyography (EMG).

[0058] S2. Process and analyze the multimodal physiological signals to quantify the subject's real-time state and interaction intentions, and generate structured analysis results; the structured analysis results include attention level, fixation point position, arm motion vector, and gesture state;

[0059] S3. Execute the rehabilitation training task logic based on the structured analysis results, and generate corresponding interactive control instructions and task performance data; the interactive control instructions include at least operation instructions for controlling the locking, lifting, spatial movement, grasping and releasing of target objects in the virtual environment constructed for rehabilitation training.

[0060] S4. Based on the execution of interactive control commands and the internal state of the virtual environment, determine the interactive events in the virtual environment, and drive the motion platform, visual display system, and audio system to provide synchronous multi-channel fused feedback to the subject; and adaptively adjust the training difficulty according to the task performance data. It should be noted that the task performance data in this embodiment is the statistical result of the subject's goal achievement rate in the training task, which is implemented using existing technology and will not be elaborated here.

[0061] In some embodiments, such as Figure 3 As shown, the specific operation steps of step S2 include:

[0062] S21. Signal preprocessing: The acquired multimodal physiological signals are sequentially bandpass filtered to remove DC bias and motion noise, and downsampling is performed; wherein, the sampling rate of the EEG signal and functional near-infrared spectroscopy signal is reduced to 240Hz, the sampling rate of the eye tracking signal and marker positioning signal is reduced to 60Hz, and the sampling rate of the forearm electromyography signal is reduced to 600Hz.

[0063] S22. Signal Analysis: Based on the preprocessed multimodal physiological signals, the following analysis operations are performed in parallel:

[0064] a. Quantitative analysis of attention levels: This is achieved through at least one of the following methods:

[0065] The energy ratio of the Theta band to the Beta band is calculated based on EEG signals to obtain a value representing the attention level; the calculation formula is as follows:

[0066] (1);

[0067] Wherein, TBR is a value representing the level of attention. The capability for the theta band within the most recent 800 sampling points; This represents the energy of the Beta band within the most recent 800 sampling points;

[0068] Based on functional near-infrared spectral signals, the concentration change signals of HbO and HbR are calculated according to the Beer-Lambert law, and the concentration change signals are input into a long short-term memory network (LSTM) model to evaluate and output attention level values ​​in real time.

[0069] b. Calculate the gaze point position: Based on eye-tracking signals, calculate the subject's gaze point position in the VR environment.

[0070] c. Calculate the arm motion vector: The arm motion vector, also known as the arm pointing vector, is used to represent the direction of the subject's right forearm, thereby determining the subject's target pointing intention. This allows the subject to achieve natural and intuitive interaction through arm movements. The specific steps are as follows:

[0071] c1. The positioning signals of markers M1 and M2 are mapped from the world coordinate system to the VR environment coordinate system through the transformation matrix, as shown in formula (2).

[0072] ;

[0073] in, This indicates the position coordinates of marker point M12 in the VR environment coordinate system. This represents the position coordinates of M2 in the VR environment coordinate system, and ; This represents the transformation matrix from the world coordinate system to the VR environment coordinate system; This represents the environmental coordinates of marker point M1 in the world coordinate system, acquired by the marker point positioning system. The environmental coordinates of point M2 in the world coordinate system are obtained by the marker positioning system, and .

[0074] C2. Calculate the arm motion vector V using formula (3):

[0075] ;

[0076] d. Gesture pattern recognition: The purpose is to identify the subject's fist clenching and opening movements based on electromyographic signals. The specific steps are as follows:

[0077] d1: Segment the electromyographic signal into 200ms windows with a 50% overlap.

[0078] d2: Extract features such as absolute mean, zero crossover, power spectral density, and frequency center for each feature window;

[0079] d3: Use an LSTM model to classify the extracted features and output the gesture status in real time.

[0080] In some embodiments, the generation of interactive control instructions in step S3 depends on a composite triggering condition consisting of gaze position, arm motion vector, gesture state, and attention level, and the rehabilitation training task logic consists of four state machine processes performed sequentially: object selection state → object grasping state → region selection state → object placement state.

[0081] like Figure 4As shown, before training, the subject stands on the motion platform 410, with markers and an electromyography (EMG) sensor on their right arm, and an EEG signal acquisition device, a portable fNIRS device, and a head-mounted eye tracker on their head, facing the circular VR screen to prepare for training. During training, the object selection state is first triggered when the gaze point coincides with the target object and the arm motion vector intersects with the object. At this time, the object is first highlighted when the gaze point coincides, and then highlighted to a greater extent and slightly enlarged when the arm pointing vector intersects to complete the selection. Next, in the object selection state, the subject triggers the object grasping state by clenching their right fist and exceeding the first threshold of attention. The object is lifted and moved with the arm motion vector. Subsequently, in the object grasping state, the region selection state is triggered when the gaze point coincides with the target area and the arm motion vector intersects with the area. The region is highlighted when the gaze point coincides, and then highlighted to a greater extent when the arm pointing vector intersects to complete the selection. Finally, in the region selection state, the subject triggers the object placement state by opening their right hand and exceeding the second threshold of attention. The object is transferred to the target area.

[0082] Throughout the interaction, the rehabilitation training interaction module 300 calls the interaction training feedback module to provide visual, auditory, tactile and somatosensory feedback. Subjects need to concentrate within a specified time, control objects to bypass obstacles and place them in the designated area as much as possible to achieve motor cognitive rehabilitation training.

[0083] In some embodiments, step S4 is implemented as follows:

[0084] In this step, the triggering of interactive feedback is based on the real-time detection of specific interactive events in the virtual environment, including collisions between virtual objects and obstacles. Feedback is provided through an interactive feedback module 400, which includes a motion platform 410, a non-immersive VR system 420, and an audio feedback system 430. The motion platform 410 is driven by one or more motors and can simulate different movement modes and environments. The non-immersive VR system 420 consists of a projection system and a circular screen, displaying virtual training scenes to the subject and dynamically adjusting the displayed content based on the subject's interaction to enhance immersion. The audio feedback system 430 provides corresponding sound effects and feedback based on the subject's performance in the virtual environment, such as operation command prompts and sound cues when a heavy object collides with an obstacle.

[0085] Throughout the training process, the rehabilitation training interaction module 300 continuously monitors the movement trajectory of the virtual weight and the position of surrounding obstacles. When the system detects that the virtual weight collides with an obstacle during its movement, it immediately triggers a feedback mechanism. The motion platform 410, the non-immersive VR system 420, and the audio feedback system 430 are configured to operate synchronously to provide the subject with integrated multi-channel feedback for the same interactive event. Specifically, the motion platform 410 generates physical shaking to simulate the feeling of a collision in reality, the non-immersive VR system 420 displays corresponding visual effects such as vibration and flicker on the screen, and the audio feedback system 430 plays sound effects corresponding to the collision. The combination of these multiple feedback mechanisms not only significantly enhances the subject's sense of immersion but also effectively improves the effect of rehabilitation training by reminding them of errors in their operation in real time.

Claims

1. A multi-sensory collaborative human-computer interaction method for motor cognitive rehabilitation training, characterized in that, Includes the following steps: S1. Acquire multimodal physiological signals of the subject during rehabilitation training, including electroencephalogram (EEG) signals, functional near-infrared spectroscopy (FIR) signals, eye-tracking signals, marker localization signals, and forearm electromyography (EMG) signals. S2. Process and analyze the multimodal physiological signals to quantify the subject's real-time state and interaction intentions, and generate structured analysis results; the structured analysis results include attention level, fixation point position, arm motion vector, and gesture state; S3. Execute the rehabilitation training task logic based on the structured analysis results, and generate corresponding interactive control instructions and task performance data; the interactive control instructions include at least operation instructions for controlling the locking, lifting, spatial movement, grasping and releasing of target objects in the virtual environment constructed for rehabilitation training. S4. Based on the execution of interactive control commands and the internal state of the virtual environment, determine the interactive events in the virtual environment, and drive the motion platform, visual display system and audio system to provide synchronous multi-channel fusion feedback to the subject; and adaptively adjust the training difficulty according to the task performance data.

2. The method according to claim 1, characterized in that, The specific implementation process of step S2 includes: S21. Signal preprocessing: The acquired multimodal physiological signals are sequentially bandpass filtered to remove DC bias and motion noise, and downsampling is performed; wherein, the sampling rate of the EEG signal and functional near-infrared spectroscopy signal is reduced to a predetermined first frequency, the sampling rate of the eye tracking signal and marker positioning signal is reduced to a predetermined second frequency, and the sampling rate of the forearm electromyography signal is reduced to a predetermined third frequency. S22. Signal Analysis: Based on the preprocessed multimodal physiological signals, the following analysis operations are performed in parallel: a. Quantitative analysis of attention levels: This is achieved through at least one of the following methods: The energy ratio of the Theta band to the Beta band is calculated based on EEG signals to obtain the attention level characterization value TBR. Based on functional near-infrared spectral signals, the concentration change signals of HbO and HbR are calculated according to the Beer-Lambert law, and the concentration change signals are input into the long short-term memory network model to evaluate and output attention level values ​​in real time. b. Calculate the gaze point position: Based on eye-tracking signals, calculate the subject's gaze point position in the VR environment; c. Calculate the arm motion vector: Based on the marker point positioning signal, calculate the arm motion vector representing the arm direction by mapping the coordinate system with the marker point positioning signal; d. Gesture pattern recognition: Based on forearm electromyography signals, the system segments the signals, extracts features, and inputs them into a long short-term memory network model to identify the subject's gesture state.

3. The method according to claim 1, characterized in that, The generation of interactive control commands in step S3 depends on a composite triggering condition consisting of gaze position, arm motion vector, gesture state, and attention level; wherein: The locking and selection of the target object is triggered by the coincidence of the gaze point position and the arm motion vector. Grasping and lifting a target object is a combination of conditions where the gesture state and the level of attention exceed a first threshold. The release and placement of the target object are based on a combination of conditions, namely, the gesture state and the attention level exceeding the second threshold.

4. The method according to claim 3, characterized in that, The rehabilitation training task logic in S3 is a sequential state machine process, including: Object selection state: triggered by a combination of conditions, namely, the gaze point coincides with the target object and the arm motion vector intersects with the object; Object grasping state: In the object selection state, it is triggered by a combination of conditions, namely the gesture state indicating a first predetermined gesture and the attention level exceeding a first threshold. Region selection state: In the object grasping state, it is triggered by a combination of conditions: the gaze point coincides with the target region and the arm motion vector intersects with the region. Object placement state: In the area selection state, it is triggered by a combination of conditions, namely, the gesture state indicates a second predetermined gesture and the attention level exceeds a second threshold.

5. The method according to claim 1, characterized in that, The triggering of interactive feedback in step S4 is based on the real-time detection of specific interactive events in the virtual environment, including collisions between virtual objects and obstacles.

6. The method according to claim 1, characterized in that, The interactive feedback includes haptic feedback provided by the motion platform, visual feedback provided by the visual display system, and auditory feedback provided by the audio system; and when the specific interactive event is triggered, the motion platform, visual display system, and audio system are configured to operate synchronously to provide the subject with fused multi-channel feedback for the same interactive event.

7. A multi-sensory collaborative human-computer interaction system for motor cognitive rehabilitation training, comprising: Interactive data acquisition module, interactive data analysis module, rehabilitation training interaction module, and interactive training feedback module; The interactive data acquisition module synchronously acquires multimodal physiological signals of the subject during rehabilitation training and sends them to the interactive data analysis module; wherein, the multimodal physiological signals include electroencephalogram (EEG) signals, functional near-infrared spectroscopy signals, eye-tracking signals, marker point localization signals, and forearm electromyography signals; The interactive data analysis module processes and analyzes the received multimodal physiological signals in real time, quantifies the subject's real-time state and interaction intention, and generates structured analysis results including attention level, fixation point position, arm movement vector and gesture state, which are then sent to the rehabilitation training interactive module. The rehabilitation training interaction module executes rehabilitation training task logic based on the received structured analysis results and generates corresponding interactive control instructions and task performance data, which are then sent to the interactive training feedback module. The interactive control instructions include at least operation instructions for locking, lifting, spatially moving, grasping, and releasing target objects in the virtual environment constructed for rehabilitation training. The interactive training feedback module includes a motion platform for providing haptic feedback, a non-immersive or immersive VR display system for providing visual and tactile feedback, and a sound system for providing auditory feedback. Based on the execution of interactive control commands and the internal state of the virtual environment, the interactive training feedback module determines interactive events in the virtual environment and drives the motion platform, visual display system, and sound system to provide synchronous multi-channel fused feedback to the subject. It also adaptively adjusts the training difficulty based on the task performance data.

Images