An immersive sensory training and evaluation method for special education needs

Abstract

The application belongs to the field of special education assistance and rehabilitation evaluation, and particularly relates to an immersive sensory training and evaluation method for special education needs. A method named "an immersive sensory training and evaluation method for special education needs" is disclosed, aiming to solve the problems of strong subjectivity and coarse data granularity of existing evaluation. The steps include: constructing a virtual scene and defining a standard trajectory; capturing data such as eye movement visual line, head pose and limb end in real time; comparing the actual trajectory with the standard trajectory by using the DTW algorithm, and extracting the offset and lag features; calculating the attention concentration and motion smoothness through visual line collision detection and Jerk value quantification; and finally automatically generating a digital evaluation report containing a space behavior heat map. Through the above scheme, the application realizes the transformation of evaluation from qualitative observation to quantitative analysis, significantly improving the objectivity, accuracy and scientificity of rehabilitation evaluation.

Description

Technical Field

[0001] This invention belongs to the field of special education assistance and rehabilitation assessment, and specifically relates to an immersive sensory training and assessment method for special education needs. Background Technology

[0002] With the deep integration of neurorehabilitation and information technology, sensory integration training for individuals with special educational needs, such as those with autism spectrum disorder (ASD), attention deficit hyperactivity disorder (ADHD), and other developmental delays, has become a core research direction in clinical rehabilitation and special education. The essence of sensory integration training lies in inducing adaptive responses in the trainee's nervous system through structured multisensory stimulation, thereby improving their functional performance in areas such as visual search, attention allocation, proprioception, and limb coordination. Throughout the rehabilitation cycle, accurate assessment of training effectiveness is not only a key indicator for measuring the trainee's current functional status but also the cornerstone for special education teachers and rehabilitation physicians to dynamically adjust intervention strategies and develop subsequent personalized rehabilitation plans.

[0003] For a long time, the assessment paradigm commonly used in the field has been based primarily on behavioral observation and standardized rating scales. In typical training scenarios, qualified special education professionals observe trainees' task execution and, based on pre-set scale dimensions such as the ABC scale or CARS scale, qualitatively or semi-quantitatively grade the trainees' reaction time, adherence to medical advice, and behavioral coordination. While such methods played an important role in a specific historical period lacking digital monitoring tools, their inherent technical limitations and deep-seated contradictions become increasingly apparent when addressing the stringent requirements of data granularity and objectivity in modern precision rehabilitation.

[0004] Specifically, existing technological solutions suffer from an insurmountable physiological perception bottleneck at the underlying logic level: the sampling frequency of the human observer's visual and auditory perception system is far lower than the dynamic evolution rate of neural activity. When monitoring visual fixation drift in ASD trainees or momentary attention loss in ADHD trainees, traditional manual observation struggles to capture millisecond-level changes in gaze paths or extremely subtle attentional "vacuum periods." This coarse observation granularity leads to the loss of numerous micro-behavioral features with critical diagnostic value during the data collection phase. Consequently, assessment results often only reflect the trainee's success or failure at the task's endpoint, failing to trace the cognitive processing pathways during execution. Furthermore, conventional assessment methods exhibit significant "dimensionality reduction" defects when processing complex three-dimensional spatial interaction information. When trainees perform sensory integration tasks, the spatial movement trajectories of their extremities, the sequence of head posture adjustments, and the coherence of hand-eye coordination are essentially complex dynamic vector fields evolving over time in three-dimensional physical space. However, through manual recording or scale scoring, this high-dimensional dynamic spatial information is forcibly abstracted into several isolated, static behavioral descriptions. This not only leads to a serious lack of assessment dimensions but also makes the quantitative analysis of trainees' proprioception and their ability to interact with the environment lack scientific mathematical support. Therefore, the industry urgently needs an immersive sensory training and assessment method tailored to the needs of special education. Summary of the Invention

[0005] This invention provides an immersive sensory training and assessment method for special education needs, aiming to solve the technical problems of strong subjectivity, coarse data granularity, and lack of spatial dimension information in the existing technology for sensory integration training and assessment of people with special needs.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: An immersive sensory training and assessment method for special education needs, characterized by the following steps: Step 1, Environment Calibration and Task Deployment: The system constructs a virtual training scenario and defines the task target object in the three-dimensional world coordinate system of the virtual scenario. The standard motion trajectory of the task target object in the three-dimensional coordinate system is set. The standard motion trajectory consists of a series of discrete spatial coordinate points arranged in a time sequence. The system is constructed and employs a cubic spline interpolation algorithm to ensure the continuity of the trajectory. Before deploying the task, the system calibrates the display refresh rate of the virtual display device and the sensor sampling frequency to ensure that the frame rate of the virtual scene is not less than 90Hz, in order to eliminate motion sickness in trainees and ensure the real-time performance of the data.

[0007] Step 2, Real-time Multimodal Data Capture: During the training task, the system synchronously collects multi-source behavioral data of the trainee through the multimodal perception module integrated into the immersive device. The data collection includes: 1. Eye-tracking gaze vector data: Near-infrared pupil-corneal reflection method is used. The center of the pupil and the position of the corneal reflection spot are captured by an infrared camera with a sampling frequency of not less than 120Hz. The gaze vector of the trainee in three-dimensional space is calculated. .

[0008] 2. Head pose data: The built-in six-axis inertial measurement unit (IMU) is used to collect the three-dimensional Euler angles (pitch, yaw, roll) and acceleration data of the trainee's head in real time, and converts them into a head pose matrix in the global coordinate system through a quaternion algorithm.

[0009] 3. Extremity spatial coordinate data: Real-time coordinates of the trainee's extremities (such as hand controllers or wearable sensors) in a three-dimensional coordinate system are acquired through optical or electromagnetic tracking at a sampling frequency of no less than 60Hz. .

[0010] Step 3, Trajectory Feature Fusion Processing: The system maps the collected multimodal data to the system's main clock axis using a timestamp alignment algorithm. For heterogeneous data with inconsistent sampling frequencies, linear interpolation or Bézier curve completion is used to ensure consistency of data in each dimension across time steps. Subsequently, the Dynamic Time Warping (DTW) algorithm is used to map and compare the trainee's actual limb movement trajectory with a preset standard movement trajectory. The DTW algorithm finds the optimal alignment path by calculating the cumulative distance between two time series; its calculation formula is as follows: in, This represents the Euclidean distance between the actual coordinate point and the standard coordinate point. The algorithm extracts the spatial offset features and temporal hysteresis features of the trainee's motion trajectory.

[0011] Step 4, Attention and Coordination Measurement: Based on the processed multimodal data, the system performs a deep quantitative evaluation: 1. Attention concentration calculation: The system uses gaze vectors. A collision detection mechanism with task targets in the virtual scene determines whether the line of sight falls within the bounding box of the target. Attention concentration is defined. This is the ratio of the cumulative duration of line-of-sight coverage on the target to the total task duration, i.e.: 2. Simultaneously, record the number of times the gaze leaves the target and the average angle of deviation to characterize the stability of attention.

[0012] 3. Evaluation of movement smoothness: based on spatial coordinates of limb extremities. The jerk (Jerk) value of limb movement is calculated using third-order differentiation. The Jerk value reflects the coordination of the trainee's central nervous system in controlling muscle contraction, and its quantification is based on the standardized average jerk (NAJ), as shown in the following formula: in, For the duration of the action, This represents the total length of the trajectory. The smaller the NAJ value, the smoother the movement and the higher the coordination.

[0013] Step 5: Automatic Report Generation: The system integrates the aforementioned attention metrics, motion smoothness metrics, and trajectory deviation data to generate a multi-dimensional digital evaluation report. The report includes: 1. Spatial Behavior Heatmap: Using the Gaussian kernel density estimation algorithm, the trainee's gaze resting point is projected onto the planar unfolded map of the virtual scene. The dense areas of attention allocation are represented by the color intensity, which intuitively shows the blind spots of visual search.

[0014] 2. Task completion score: Based on the standard trajectory overlap, motion smoothness score and task execution time, a weighted scoring model is used to output a comprehensive evaluation score.

[0015] Furthermore, in step one, the environmental calibration process also includes benchmark calibration of the trainee's initial height. The system automatically adjusts the height offset of all interactive elements in the virtual scene by acquiring the vertical height of the head-mounted display device worn by the trainee in a static state, ensuring that the initial height of the task target relative to the trainee's visual axis is consistent, thus eliminating the impact of height differences on the accuracy of the interaction trajectory.

[0016] Furthermore, in step two, the acquisition of the eye-tracking vector data also includes the automatic classification of fixations and saccades. The system sets a velocity threshold of 30° / s. When the angular velocity of the gaze is lower than this threshold and lasts for more than 100ms, it is determined to be a fixation behavior; when the angular velocity is higher than this threshold, it is determined to be a saccade behavior. By analyzing the saccade paths, the search strategy characteristics of the trainee when facing complex visual interference are extracted.

[0017] Furthermore, in step three, the trajectory feature fusion processing also includes feature extraction of micro-limb tremors. The system uses a high-pass filter to filter out low-frequency voluntary movement components in the limb trajectory, retaining tremor signals with frequencies between 4Hz and 12Hz. The power spectral density of the tremor signal is analyzed by Fast Fourier Transform (FFT) to calculate the dominant frequency and amplitude of the tremor, which serve as a quantitative auxiliary indicator of neuromuscular control ability.

[0018] Furthermore, in step four, the calculation of attention concentration also incorporates a "distraction resistance" dimension. Several distracting objects with different acoustic and optical characteristics are preset in the virtual scene. The system monitors in real time the response time and retraction time of the trainee's gaze vector shifting from the target object to the distracting object. If the gaze lingers on the distracting object for more than a preset threshold (e.g., 2.0 seconds), the system identifies it as a valid attention drift event and records the spatial quadrant in which the event occurred.

[0019] Furthermore, in step five, the generation process of the assessment report employs a trend prediction model based on Long Short-Term Memory (LSTM) networks. The system calls upon the multidimensional indicator sequence in the trainee's historical training database, compares it with the data characteristics of the current training, calculates the slope of change for each assessment dimension, thereby predicting the trainee's ability evolution trend within a future rehabilitation cycle, and providing special education teachers with decision support for adjusting task difficulty (such as increasing the speed of the target object or increasing the number of distracting items).

[0020] The present invention has the following beneficial effects: First, it realizes a paradigm shift in special education assessment from "qualitative observation" to "quantitative analysis." Through high-frequency multimodal data capture, this invention can record trainees' attentional fluctuations at the millisecond level and limb movement deviations at the micrometer level, completely eliminating errors caused by observer subjectivity in traditional scale assessments and ensuring the objectivity and rigor of the assessment results.

[0021] Secondly, it provides a multi-dimensional and precise localization capability for sensory integration disorders. By utilizing dynamic time warping algorithms and spatial heatmap analysis, this invention can accurately characterize the blind spots and motor coordination deficiencies of trainees in three-dimensional space. For example, it can identify visual neglect in specific quadrants or ataxia when performing cross-midline movements, providing data support for developing personalized rehabilitation strategies.

[0022] Third, a closed-loop rehabilitation progress tracking system has been established. Through long-term serial monitoring of key indicators such as Jerk score and attention coverage, the system can quantify the small progress of trainees at each stage, which not only improves the scientific nature of rehabilitation assessment, but also provides a standardized data exchange format for the field of special education, facilitating the evaluation and comparison of rehabilitation effects across institutions and regions.

[0023] The present invention will be further described in detail below with reference to specific embodiments.

[0024] In a preferred embodiment, for the assessment of attention deficit disorders commonly found in special education needs, this invention is implemented by constructing a "dynamic visual tracking" training task. First, in the environment calibration stage of step one, the system generates a sphere with a diameter of 0.15 meters in virtual space as the task target object. Its material is set to a high-contrast yellow, and the background is filled with 30 low-saturation cubes as interference. The standard motion trajectory is set as a figure-eight curve in three-dimensional space, defined by the following parametric equation: A, B, and C are dynamically adjusted based on the trainee's arm span range.

[0025] In the data acquisition phase of step two, the trainee wore a virtual reality headset integrated with eye-tracking sensors and held an interactive controller with six-axis spatial positioning capabilities. The infrared eye-tracking module recorded the spatial distance between the trainee's line of sight and the virtual sphere at a frequency of 120Hz, and the controller sensor recorded the path points when the trainee attempted to simulate the figure-eight trajectory at a frequency of 100Hz.

[0026] In step three, the system uses a time alignment algorithm to synchronize the handle trajectory point sequence with the eye-tracking fixation point sequence. Considering potential reaction delays during task execution, the DTW algorithm automatically corrects time axis offsets through an elastic matching mechanism, extracting pure spatial path deviations. If the trainee experiences significant trajectory jerking at a curve bend, the system uses high-pass filtering to identify the jerking frequency at that point, determining whether it indicates a failure in fine motor control.

[0027] In the quantization phase of step four, the system calculates the ratio of the gaze vector to the duration of the ball collision. If the trainee's gaze frequently jumps to background distractions during the ball's movement, and the duration of each jump exceeds 500ms, the system records this as inefficient attention allocation. Simultaneously, the NAJ value for that path segment is calculated by taking the third derivative with respect to the motion handle coordinates. Experimental data shows that the NAJ value of the healthy control group typically remains in a low range, while the NAJ value of trainees with motor integration disorders increases exponentially with increasing task complexity.

[0028] In step five, the generated report not only displays the total score but also compares the envelope range of the standard curve with the actual curve using a 3D trajectory graph. The heatmap clearly reveals the distribution density of the trainee's gaze in each quadrant of space. If the heatmap shows an extremely light color in the left-hand area, it suggests to the special education teacher that the trainee may have weakened left-hand visual search, requiring increased stimulation and guidance of the left-hand space in subsequent training.

[0029] As another embodiment of the present invention, the system also supports sensory-motor integration assessment under multi-task loads. In a "simulated supermarket shopping" scenario, trainees need to find specific items on the shelves (visual search task) and simultaneously grasp and place them in a shopping cart using a handle (fine motor task). During this process, the system monitors the trainee's gaze trajectory in real time before, during, and after the action, assessing their visual-limb coordination prediction ability. Specifically, the system calculates the time difference (Lead-LagTime) between the time the gaze reaches the target object and the time the hand trajectory arrives. For trainees with good sensory integration, the gaze usually leads the hand movement by about 200-400ms, while for individuals with special needs, this time difference often exhibits significant instability. By digitally recording this parameter, the cognitive load tolerance limit of trainees under complex tasks can be deeply analyzed.

[0030] Furthermore, the method described in this invention employs a comprehensive scoring matrix based on eigenvalue normalization at the data processing level. Considering the differences in age and initial ability levels among different trainees, the system will collect raw physical quantities (such as distance deviation)... Time delay accelerometer value The score is converted into a dimensionless score through Z-score standardization. This score matrix is ​​defined as follows: , in To score attention, Score the smoothness of the movement. Scoring is given based on trajectory consistency. The weighting coefficients are set according to the different focuses of the training phase. This parameterized scoring system ensures that the evaluation model can flexibly adapt to the needs of different stages, from primary sensory induction to advanced behavioral intervention.

[0031] Furthermore, to enhance the anti-interference capability of the assessment, this invention introduces an adaptive threshold filtering algorithm in the data preprocessing stage of step two. To address the potential for involuntary blinking or head-mounted device slippage leading to momentary signal loss, the system uses a Kalman filter to predict gaze position and limb coordinates. In cases of signal interruption duration less than 150ms, interpolation is performed using the prediction model to complete the signal, ensuring the continuity of dynamic parameter analysis and avoiding noise interference from outliers in the NAJ calculation.

[0032] In summary, this invention transforms the sensory training process in special education into an observable, modelable, and computable digital process by constructing a high-dimensional, synchronous multimodal sensory space. Through deep feature extraction and algorithmic fusion of eye movements, head movements, and limb movement trajectories, this invention not only solves the problems of subjectivity and coarse granularity in assessment but also provides a novel technical approach with a physiological and neurological basis for the precise rehabilitation of people with special needs, demonstrating significant clinical application value and educational assessment significance. Attached Figure Description

[0033] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating an immersive sensory training and assessment method for special educational needs according to the present invention. Figure 2 This is a schematic diagram of the system architecture and data acquisition scenario in an embodiment of the present invention. Detailed Implementation

[0034] According to the present invention, an immersive sensory training and assessment method for special educational needs integrates high-precision virtual reality technology, multimodal biometric recognition, and deep behavioral analysis algorithms during implementation. To enable those skilled in the art to more clearly understand and implement the technical solution of the present invention, the following will provide a very detailed description of each implementation step, engineering parameter setting, algorithm logic, and specific hardware coordination mechanism.

[0035] In a complete implementation of this invention, the first step is the environmental calibration and task deployment process. The core of this process is establishing a high-fidelity three-dimensional coordinate system benchmark to ensure that every action and every second of gaze deviation by the trainee in the virtual space can be accurately mapped to a unified measurement. The virtual training scenario constructed by the system unfolds in a Cartesian coordinate system based on the right-hand rule, where the origin is typically set as the trainee's initial standing or sitting center point in the virtual environment. When deploying tasks, the system defines the task target in the three-dimensional world coordinate system using a preset logic script. To achieve refined training of the trainee's sensory integration abilities, the task target is endowed with dynamic physical properties. The system sets a standard motion trajectory for the task target in the three-dimensional coordinate system. This trajectory is not a simple straight line or circular motion, but rather a series of discrete spatial coordinate points arranged according to a strict time sequence. The system constructs a complex path. To avoid visual jumps when the target object moves between discrete points, the system employs a third-order or higher-order cubic spline interpolation algorithm. This algorithm constructs a third-order polynomial equation between two adjacent discrete coordinate points, ensuring the continuity of the first derivative (velocity) and second derivative (acceleration) of the trajectory at the connection point, thus providing trainees with a smooth and physically consistent visual inducement source. Furthermore, in the early stages of task deployment, the system must rigorously calibrate the hardware parameters of the virtual display device. This includes obtaining the display refresh rate through the underlying driver interface and locking it with the rendering frame rate of the virtual engine to ensure that the real-time rendering frame rate is no less than 90Hz. Through this high-frequency refresh mechanism, the system can control the motion-to-photon latency to within 20 milliseconds. For trainees with special needs due to vestibular system sensitivity, this is a key technical support for eliminating motion sickness and ensuring long-term training compliance. At the same time, the system will synchronously calibrate the sampling frequency of the sensors and use a high-precision event timer (HPET) as a global clock source to ensure that the timestamp offset of all input devices is less than 1 millisecond.

[0036] During environmental calibration, adaptive calibration to accommodate individual differences among trainees is a prerequisite for ensuring data accuracy. In a preferred embodiment of this invention, the environmental calibration process includes benchmark calibration logic for the trainee's initial height. The system acquires the vertical height component (Z-axis coordinate value) of the head-mounted display device worn by the trainee in real-time during a static standby state, using this as a benchmark reference value. The system's internal logic layer automatically calculates the difference between this height value and the default eye level of the virtual scene, and applies this difference as a global offset vector to the vertical coordinates of all interactive elements, task targets, and interfering objects in the virtual scene. This automated repositioning mechanism ensures that regardless of whether the trainee is a child or an adult, and whether they are standing or in a wheelchair, the initial relative height of the task target relative to the trainee's visual axis remains consistent, thereby eliminating motion trajectory distortion caused by differences in viewing angle.

[0037] When the system enters the second stage of multimodal data real-time capture, the synchronous acquisition of multi-source heterogeneous data constitutes the underlying data stream for evaluation. In this stage, the system acquires the trainee's eye movement vector data in real time through an infrared eye-tracking module integrated within the immersive head-mounted device. Specifically, the system employs near-infrared pupillary corneal reflection (PCCR) and continuously captures infrared reflection images of the trainee's eyes using two sets of high-frame-rate infrared cameras with a frequency of at least 120Hz. The system's internal image processing chip locates the pupil center and the position of the corneal reflection spot generated by the infrared light source in real time. It calculates the deflection angle of the gaze in the eye coordinate system by establishing an eye geometry model and further combines this with the head pose matrix to transform it into a three-dimensional gaze vector in virtual space. To achieve deeper functional assessment, the eye-tracking vector data acquisition process also integrates real-time classification logic for fixation and saccades. The system calculates the angular velocity of the gaze in real time using a sliding window algorithm, and rigidly sets the velocity threshold to 30° / s. Within a specific implementation cycle, when the gaze angular velocity remains below this threshold for more than 100ms, the system marks it as a valid gaze behavior, which is a core indicator for evaluating the trainee's visual information processing depth and attention maintenance ability; conversely, when the angular velocity suddenly exceeds the threshold, it is judged as a saccade behavior. By analyzing the geometric features of the saccade path, such as saccade amplitude, peak velocity, and damped oscillations after saccade, the system extracts the trainee's visual search strategy and neuromuscular feedback characteristics when facing complex visual interference.

[0038] Meanwhile, head pose data is supported by a built-in six-axis inertial measurement unit (IMU). This unit includes a three-axis accelerometer and a three-axis gyroscope. The system fuses the raw data using gradient descent or extended Kalman filtering algorithms, outputting the trainee's three-dimensional Euler angles of head rotation in real time. To avoid gimbal lock at the vertical extreme positions, the system uses quaternions for attitude calculation throughout the internal computation logic, then converts it into pitch, yaw, and roll angles as needed for data display. Acquisition of spatial coordinate data for the extremities relies on an optical or electromagnetic tracking system integrated with the head-mounted device. A handheld controller or wearable sensor continuously feeds back its real-time coordinates in three-dimensional space to the host at a sampling frequency of at least 60Hz. This high-frequency, multi-dimensional data capture system provides millisecond-level temporal resolution and millimeter-level spatial resolution for subsequent sensory integration analysis.

[0039] In the trajectory feature fusion processing stage of step three, the core technical challenge faced by the system lies in how to transform raw data with different sampling frequencies and spatial attributes into feature matrices that can be directly compared by the algorithm. To this end, the system first calls a timestamp alignment algorithm to synchronize and resample eye-tracking data (120Hz), limb data (60Hz), and head-movement data, using the system's master clock as the axis. For the low-frequency acquired limb data, the system employs a four-point third-order Bézier curve completion technique to refine its time step to match that of the eye-tracking data while maintaining the physical continuity of the trajectory. Subsequently, the system uses the Dynamic Time Warping (DTW) algorithm to perform crucial trajectory comparisons. The core value of this algorithm lies in its ability to identify and correct time delays caused by trainees performing tasks. For example, when a trainee's actions lag behind the standard trajectory by 0.5 seconds due to slow cognitive processing, the traditional Euclidean distance algorithm would consider this a large error, while the DTW algorithm, by finding the optimal alignment path with the minimum cumulative distance in the search space, can accurately associate the trainee's actual action points with the corresponding standard trajectory points. By analyzing the slope of the DTW search path, the system can quantify the trainee's reaction time delay and the speed fluctuation pattern during motion execution.

[0040] Furthermore, during the trajectory feature fusion process, the system specifically incorporates a feature extraction mechanism for micro-limb tremors. This mechanism has significant clinical implications when assessing trainees with neurodevelopmental disorders such as autism or cerebral palsy. The system utilizes a high-order Chebyshev high-pass filter with a cutoff frequency set to 4Hz to filter out low-frequency voluntary movement components during large-range pointing movements. The remaining high-frequency signal components contain involuntary tremor information with frequencies ranging from 4Hz to 12Hz. The system then calls a Fast Fourier Transform (FFT) function to convert the time-series signal to the frequency domain, and calculates the dominant frequency components of the tremor and their peak amplitudes by analyzing the power spectral density (PSD) distribution. These quantified tremor features, as auxiliary indicators of neuromuscular control ability, are integrated into the final comprehensive evaluation matrix.

[0041] In step four, the system performs deep quantization evaluation based on the preprocessed multimodal data stream, focusing on the deconstruction of attention and motor coordination. Attention concentration is calculated using the gaze vector. This is achieved through dynamic collision detection with task targets in the virtual scene. The system generates a bounding box that fits the geometry of each task target in the background and determines in real time whether the gaze vector penetrates the bounding box. (Attention concentration) Strictly defined as the ratio of effective fixation time to total task time, this metric also incorporates a "distraction resistance" dimension. Within specific quadrants of the virtual scene, the system periodically deploys distracting objects with different audio-visual characteristics, such as sudden flashes of light or visual fluctuations simulating noise. The system monitors and records in real-time the saccade latency (the time it takes for the trainee's gaze to shift from the target object to the distractor) and the duration of dwelling on the distractor. If the dwelling time exceeds a preset 2.0-second threshold, the system automatically triggers an attention drift event recording. This quantification method based on distraction resistance can deeply characterize the trainee's visual filtering performance in a multi-tasking environment.

[0042] In terms of motion smoothness evaluation, the system utilizes spatial coordinate points at the limb extremities. The system performs continuous third-order numerical differentiation on the sequence data. The first order of the derivative is velocity, the second is acceleration, and the third is jerk, which reflects the mass of the movement. In physics and biomechanics, the magnitude of the Jerk value directly characterizes the coordination of the trainee's central nervous system in planning and correcting muscle movements. The system uses standardized average jerk (NAJ) as the final quantitative indicator. This is further enhanced by incorporating the duration of the movement. Total length of trajectory By weighting Jerk's squared integral, the NAJ value is transformed into a pure coordination index unaffected by range of motion and speed. The closer the NAJ value is to zero, the smoother and more predictive the trainee's movements are; conversely, a large NAJ value reflects a large number of minor corrections and jitters during movement execution, which is quantitative evidence of ataxia or insufficient fine motor control.

[0043] In the automatic generation phase of the assessment report in step five, the system transforms the aforementioned discrete technical indicators into an intuitive and instructive rehabilitation report. Specifically, the generation of the spatial behavior heatmap employs the Gaussian kernel density estimation (KDE) algorithm. The system projects the coordinates of tens of thousands of gaze-holding points throughout the training process onto a planar unfolded map of the virtual scene, smoothing each point using a Gaussian kernel function to form a continuous probability density field. The warm and cool color distribution in the heatmap intuitively demonstrates the trainee's attention allocation preferences, particularly in identifying visual neglect or blind spots in specific quadrants; this visualization method provides unparalleled diagnostic value. Furthermore, the assessment report also incorporates a trend prediction model based on a Long Short-Term Memory (LSTM) network. The system calls upon the trainee's training indicator sequences from multiple rehabilitation cycles, inputting the current NAJ value, attention score, trajectory offset, and other dimensions into the trained LSTM neural network. This model utilizes its internal forgetting and memory gate mechanisms to capture the non-linear characteristics of the trainee's ability evolution and calculate the slope of change in each dimension. Based on this, the system can automatically predict the level of rehabilitation that trainees may achieve in the coming weeks and provide educators with suggestions for adjusting the difficulty of the task, such as suggesting increasing the speed of the target object, expanding the range of motion, or introducing more challenging dynamic interference items.

[0044] To further demonstrate the superiority and reliability of the method described in this invention, a specific embodiment and corresponding comparative experimental data are provided below.

[0045] Example 1: Dynamic visual tracking training for children with attention deficit hyperactivity disorder (ADHD) In this embodiment, the system constructs a specialized training scenario designed to assess and improve trainees’ sustained attention and hand-eye coordination.

[0046] 1. Environment Configuration: The virtual scene is a pure hemispherical space. The objective is a sphere with a diameter of 0.15 meters and a highly saturated yellow color (Hex: #FFD700). The interference consists of 30 randomly distributed low-saturation gray cubes in the background.

[0047] 2. Trajectory Setting: The target object moves along a three-dimensional figure-eight curve, and its parametric equation is: (The height has been calibrated according to the trainee's height). The cycle of the target object's rotation is set to 10 seconds.

[0048] 3. Training process: Trainees need to wear a VR headset with 120Hz eye-tracking function, hold an interactive controller, and try to keep the virtual crosshair of the controller following the moving yellow ball, while also keeping their eyes locked on the ball.

[0049] Comparative Example 1: Sensory Training Assessment Based on Traditional Video Scale Observations In the comparative study, the same group of trainees received similar tracking training in a traditional multimedia classroom. The teacher observed the trainees' actions via an external monitor and manually scored them according to the Sensory Integration Assessment Scale. The scoring criteria were: Excellent (smooth movements, full attention), Good (occasional deviations, generally following along), and Poor (frequent distractions, discontinuous movements).

[0050] The comparison results of the data collected by the system described in this invention with traditional evaluation methods are shown in Table 1 below: Table 1: Performance Comparison of Embodiments of the Invention with Traditional Evaluation Methods As can be seen from the above data comparison, this invention not only achieves a breakthrough in the spatial and temporal dimensions of data, but more importantly, it transforms vague terms (such as "good" and "intermittent") that rely on teachers' subjective judgment in traditional assessments into mathematical indicators with strict physical meaning. For example, the data point of NAJ value 42.15 can accurately reflect the trainee's control failure at the turn of the figure-eight trajectory, which is easily overlooked in traditional observation.

[0051] Furthermore, this invention employs a Z-score-based standardized scoring matrix model in its processing. Considering the significant differences in the initial abilities of individuals with special needs, purely absolute physical quantities cannot objectively evaluate their progress. The system collects normative data from a large number of trainees of the same age group, transforming the trainees' raw performance into standardized scores. (Scoring matrix) The weighting coefficients can be dynamically configured according to specific training objectives. For example, in the early stages of rehabilitation, attention weights can be adjusted. Set it to 0.6, and during the fine motor skill enhancement phase, increase the weight of motion smoothness. Increase. This parameterized evaluation framework makes the method highly flexible and universal.

[0052] Regarding data stability assurance, this invention integrates an adaptive Kalman filter into the underlying algorithm. During actual sensory training, trainees may experience temporary loss of pupil signals from the infrared camera due to frequent blinking or violent shaking of the head-mounted display. The system uses the prediction equation of the Kalman filter, leveraging the previous moment's gaze angular velocity and acceleration, to calculate the predicted gaze point coordinates within a short period (less than 150ms) of signal interruption. This prediction interpolation mechanism effectively prevents abnormal NAJ value calculations caused by data breaks, ensuring the continuity of behavioral feature extraction and the rigor of the evaluation report throughout the entire training cycle.

[0053] In summary, this invention elevates sensory training in special education from the traditional "observation and recording mode" to a "precision measurement mode" by constructing a high-dimensional immersive sensory space and utilizing multimodal sensor fusion technology and advanced motion dynamics algorithms. It not only provides centimeter-level motion capture and millisecond-level attention monitoring but also achieves scientific prediction of rehabilitation trends through deep learning models. The implementation of this invention significantly reduces human interference in the assessment process, providing special education institutions with a standardized, digitalized, and traceable integrated solution for assessment and training. It has extremely high engineering application value and social benefits in areas such as clinical rehabilitation assessment, personalized teaching plan development, and early screening of sensory integration disorders.

Claims

1. An immersive sensory training and assessment method for special education needs, characterized in that, The method is implemented using an integrated system comprising a virtual display device, a multimodal sensing module, and a computing processing unit, and includes the following steps: Step 1, Environment Calibration and Task Deployment: A virtual training scene is constructed using a computing processing unit, and the task target is defined in the three-dimensional world coordinate system of the virtual scene; the standard motion trajectory of the task target in the three-dimensional coordinate system is set, wherein the standard motion trajectory consists of a series of discrete spatial coordinate points arranged in a time sequence. The system constructs a transition path between adjacent discrete spatial coordinate points using a cubic spline interpolation algorithm to ensure the continuity of the first and second derivatives of the trajectory at the connection points. Before deploying the task, the system calibrates the display refresh rate of the virtual display device and locks it to the rendering frame rate. At the same time, it performs time synchronization calibration on the sampling frequency of the sensor to ensure that the real-time frame rate of the virtual scene is not lower than 90Hz. Step 2, Real-time Multimodal Data Capture: During the training task execution, the system synchronously collects multi-source behavioral data of the trainee through the multimodal perception module integrated in the virtual display device. This data collection includes: Eye-tracking gaze vector data: The center of the pupil and the position of the corneal reflective spot of the trainee are captured by an infrared camera with a sampling frequency of no less than 120Hz. The gaze vector of the trainee in three-dimensional space is calculated based on the near-infrared pupil-corneal reflectance method. ; Head pose data: The built-in six-axis inertial measurement unit is used to collect three-dimensional rotation Euler angle data and acceleration data of the trainee's head in real time, and the data is converted into head pose matrix in global coordinate system in real time through quaternion algorithm; Extremity spatial coordinate data: Real-time coordinates of the limb extremities in a three-dimensional coordinate system are acquired through an interactive handle or wearable sensor associated with the trainee's limb, at a sampling frequency of no less than 60Hz. ; Step 3, Trajectory Feature Fusion Processing: The system maps the collected heterogeneous multimodal data to the system's main clock axis using a timestamp alignment algorithm. For data with a sampling frequency lower than the preset benchmark, a third-order Bézier curve is used for path completion to ensure consistency of data in all dimensions in terms of time step. Subsequently, a dynamic time warping algorithm is used to perform a nonlinear mapping comparison between the trainee's actual limb movement trajectory and the preset standard movement trajectory. The optimal alignment path is found by calculating the cumulative distance between the two time series, thereby extracting the offset features of the trainee's movement trajectory in the spatial dimension and the hysteresis features in the temporal dimension. Step 4, Attention and Coordination Quantification: Based on the processed multimodal data, the system performs deep quantitative evaluation from two dimensions: visual assignment and dynamic control, including: Attention concentration calculation: using gaze vector The collision detection mechanism with the task target in the virtual scene determines whether the point of gaze is within the geometric bounding box of the target and calculates the ratio of the cumulative coverage time of the gaze on the target to the total task time as the attention concentration. Movement smoothness evaluation: Based on the spatial coordinate point sequence of the limb end, the acceleration value of limb movement is extracted through continuous third-order numerical differentiation, and the standardized average acceleration is calculated to quantify the coordination of the trainee's central nervous system in controlling muscle contraction. Step 5: Automatic generation of the evaluation report: The system integrates the attention concentration, motion smoothness, trajectory deviation, and interference resistance data, and projects the gaze dwell point onto the planar unfolded map of the virtual scene using the Gaussian kernel density estimation algorithm to generate a spatial behavior heatmap representing the blind spot of visual search; at the same time, based on the multi-dimensional index sequence, a weighted scoring model is used to output a comprehensive completion score, generating a digital evaluation report.

2. The immersive sensory training and assessment method for special educational needs according to claim 1, characterized in that, The environmental calibration process described in step one also includes adaptive calibration logic for the trainee's initial height: the system obtains the vertical axial height value of the virtual display device worn by the trainee in a static state and sets it as the reference height offset; subsequently, the system automatically adjusts the height offset vector of all interactive elements in the virtual scene and the task target relative to the global coordinate origin to ensure that the initial relative height of the task target relative to the trainee's visual axis is maintained within a preset physiological comfort range, so as to eliminate the systematic impact of individual height differences of trainees on the accuracy of interactive trajectory.

3. The immersive sensory training and assessment method for special educational needs according to claim 1, characterized in that, In step two, the eye movement vector data acquisition process also integrates real-time automatic classification logic for fixation points and saccade behaviors. Specific steps include: calculating the eye movement angular velocity of the gaze vector within a continuous sampling period; setting a velocity threshold of 30° / s; when the eye movement angular velocity is below this threshold and the duration exceeds 100ms, the system classifies the data segment as a fixation behavior and records its coordinates in virtual space; when the eye movement angular velocity is above 30° / s, the system classifies it as a saccade behavior and analyzes the geometric features of the saccade path, including saccade amplitude, peak velocity, and damped oscillation characteristics after saccade, thereby extracting the trainee's search strategy features in the face of complex visual backgrounds.

4. The immersive sensory training and assessment method for special educational needs according to claim 1, characterized in that, In the trajectory feature fusion processing described in step three, the detailed calculation process of the system executing the dynamic time warping algorithm is as follows: The standard trajectory sequence is defined as... The actual limb movement trajectory sequence is ; Calculate the actual coordinates with standard coordinate points The Euclidean distance between them is used as the node distance. ; Using recursive formulas Calculate the cumulative distance matrix, where Indicates the destination of the path The minimum cumulative cost is determined by backtracking the cumulative distance matrix to find the optimal alignment path that minimizes the total cost. Based on the slope fluctuation of the path in the matrix, the reaction time fluctuation and motion speed consistency of the trainee when performing the trajectory tracking task are quantitatively calculated.

5. The immersive sensory training and assessment method for special educational needs according to claim 1, characterized in that, In step three, the trajectory feature fusion processing also includes the extraction of deep features from the micro-limb tremor signal. Specifically, the system calls a high-order Chebyshev high-pass filter to filter the original motion trajectory of the limb end, setting the cutoff frequency to 4Hz to filter out the low-frequency components generated by the limb's voluntary movement; retains the involuntary tremor components with frequencies between 4Hz and 12Hz, and performs a Fast Fourier Transform (FFT) on these components to convert them from a time series to a frequency domain distribution; calculates the power spectral density of the tremor signal, and extracts the dominant frequency, peak amplitude, and energy distribution density of the tremor as digital auxiliary indicators for evaluating the trainee's peripheral neuromuscular control ability.

6. The immersive sensory training and assessment method for special educational needs according to claim 1, characterized in that, In step four, the level of attention concentration The quantization logic is as follows: the system monitors the gaze vector in real time. The intersection state with the bounding box of the task target; defining the attention concentration. The calculation formula is: , in This represents the time segments in which the line-of-sight vector continuously penetrates the bounding box. The total task duration is denoted by . Simultaneously, the system constructs a multi-dimensional evaluation vector characterizing attention stability by recording the frequency of the gaze leaving the bounding box and the average offset angle after departure.

7. The immersive sensory training and assessment method for special educational needs according to claim 1, characterized in that, In step four, the quantitative evaluation process of the standardized average jerk $NAJ$ follows the following physical model: First, through the spatial coordinate points of the limb end... Regarding time By performing third-order differentiation, the instantaneous acceleration components are obtained; Subsequently, according to the formula Calculate standardized indicators, among which The duration of the action. The total geometric length of the trajectory in three-dimensional space; The smaller the value, the smoother the movement trajectory and the higher the muscle coordination; the system will calculate the... The values ​​are compared with a pre-defined health norm database to identify the degree of ataxia or fine motor developmental delay.

8. The immersive sensory training and assessment method for special educational needs according to claim 1, characterized in that, In the attention quantification step described in step four, an "interference resistance" dimension is also introduced. Specifically, several interference objects with dynamic acoustic and optical characteristics are preset in the non-task activity area of ​​the virtual scene. The system uses the multimodal perception module to monitor and record in real time the response time when the trainee's gaze vector shifts from the task target to any interference object, and the return time when the gaze returns to the task target. If the trainee's gaze lingers on the interference object for more than a preset 2.0-second threshold, the system automatically identifies it as an attention drift event and statistically analyzes the spatial quadrant distribution of the event to evaluate the trainee's perceptual filtering effectiveness in a multi-interference environment.

9. The immersive sensory training and assessment method for special educational needs according to claim 1, characterized in that, In the data acquisition stage described in step two, a robust compensation mechanism for instantaneous signal loss is also included. The specific steps are as follows: The system introduces an adaptive Kalman filter algorithm in the underlying data processing stream, and uses the line-of-sight angular velocity, limb motion acceleration and corresponding state transition matrix of the previous sampling moment to predict the line-of-sight position and limb coordinates at the current moment. When a sensor signal interruption is detected due to involuntary blinking or hardware slippage, if the interruption duration is less than 150ms, the system uses Kalman prediction values ​​for interpolation to complete the data; if the interruption duration exceeds 150ms, the system marks the data segment as invalid and removes the dynamic characteristic calculations affected by the signal interruption from the evaluation report to ensure... The authenticity and continuity of value and trajectory deviation data.

10. The immersive sensory training and assessment method for special educational needs according to claim 1, characterized in that, In step five, the generation process of the assessment report also integrates trend prediction logic based on Long Short-Term Memory (LSTM) networks. Specifically, this includes: the system calling the multi-dimensional indicator sequence stored in the historical training database of the trainee in real time, using the current attention score, movement smoothness score, and trajectory consistency score as the current input feature vector; inputting the feature vector into a pre-trained LSTM neural network model, using the memory units inside the model to capture the evolution pattern of the trainee's behavioral characteristics in long-term rehabilitation training; calculating the slope of ability change in each assessment dimension, and automatically adjusting the difficulty parameters of the next training task based on the prediction results. The difficulty parameters include the running speed step of the task target, the motion amplitude expansion rate, and the visual salience of the interference object.

Images