Cognitive training path scoring system and method based on VR interaction

Abstract

The invention relates to the technical field of intelligent processing of medical information technology and virtual reality man-machine interaction data, in particular to a cognitive training path scoring system and method based on VR interaction, hand motion data and head posture data are synchronously collected through a motion capture handle and a VR head display, and real-time motion data are output after being preprocessed by a computing server; the path generation module generates an ideal path model in combination with a preset task target, the multi-dimensional evaluation module calculates spatial precision, time efficiency, path fluency and energy consumption economy scores, and the weight fusion unit dynamically adjusts weight coefficients in combination with a historical database to output a comprehensive score. The real-time feedback module drives feedback equipment to provide real-time visual and audio guidance according to the comprehensive score, and analyzes the continuous change of the score through the adaptive controller to generate a difficulty parameter adjustment instruction to dynamically optimize a training task, so that comprehensive quantification of cognitive competence, real-time feedback guidance and adaptive difficulty adjustment are realized; and the rehabilitation training accuracy and effect are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical information technology and intelligent processing of virtual reality human-computer interaction data, and particularly relates to a cognitive training path scoring system and method based on VR interaction. BACKGROUND

[0002] The virtual reality rehabilitation training system provides a customized cognitive intervention plan for Alzheimer's patients by creating an immersive three-dimensional simulation environment. The core of the system lies in the use of visual, auditory and tactile multi-sensory stimulation to simulate daily activities such as shopping or social interaction, thereby specifically training memory, attention and executive function. The virtual reality rehabilitation training system can promote brain neuroplasticity because repetitive tasks are performed in a safe and controllable virtual environment, which helps patients gradually rebuild cognitive connections, slow down cognitive decline and improve self-care ability.

[0003] The existing virtual reality rehabilitation training technology has the following technical pain points. Specifically, the existing virtual reality rehabilitation training system relies on single trajectory endpoint coordinate comparison or subjective observation scoring, and fails to establish a multi-modal data quantitative analysis model that integrates spatial trajectory, time series and kinematic characteristics.

[0004] When Alzheimer's patients perform VR navigation training, the existing virtual reality rehabilitation training system can only feedback the deviation of the final arrival position from the target point or the total time consumption, and cannot quantitatively evaluate deep cognitive indicators such as the rationality of the patient's path planning, the fluency of the movement process and the economy of energy consumption. For example, although the patient can reach the end point, the path is tortuous and repetitive, or the movement speed fluctuates sharply, reflecting key information about spatial perception and executive control ability, which is lost due to the lack of multi-dimensional algorithm model. The lack of model further leads to the system only providing lagging and general summary evaluation, which cannot identify specific defect dimensions in real time during the training process and generate targeted guidance signals, such as timely prompting of excessive turning speed or dynamically adjusting obstacle density, so that the rehabilitation training lacks precise adaptive optimization ability based on data-driven, and the effective improvement of training effect is restricted. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a cognitive training path scoring system and method based on VR interaction, which solves the technical problem that the existing VR training system lacks a multi-modal data fusion quantitative analysis model, resulting in single evaluation dimension of user cognitive ability, lagging feedback, and difficulty in achieving precise rehabilitation training effect dynamic optimization.

[0006] To solve the above technical problems, the specific content of the present application is as follows:

[0007] In a first aspect, the present application provides a cognitive training path scoring system based on VR interaction, which includes a physical device and a control device, and the control device is in communication connection with the physical device.

[0008] The physical device includes a VR headset, a motion capture handle, a computing server, and a feedback device, the motion capture handle is used to collect user hand movement data and send the hand movement data to the computing server, the VR headset is used to display a virtual environment and collect head posture data and send the head posture data to the computing server, the computing server is used to receive and process the hand movement data and the head posture data, and the feedback device is used to receive instructions from the computing server and provide visual and audio feedback;

[0009] The control device includes a data acquisition module, a path generation module, a multi-dimensional evaluation module, and a real-time feedback module;

[0010] The data acquisition module receives the hand movement data and the head posture data from the computing server, pre-processes the hand movement data and the head posture data, and outputs real-time action data including a spatial coordinate sequence and a timestamp; the data acquisition module sends the real-time action data to the path generation module and the multi-dimensional evaluation module;

[0011] The path generation module receives the real-time action data from the data acquisition module, takes the real-time action data as input, combines a preset task target, and generates an ideal path model including an ideal spatial coordinate sequence and an ideal time sequence; the path generation module sends the ideal path model to the multi-dimensional evaluation module;

[0012] The multi-dimensional evaluation module receives the real-time action data from the data acquisition module and the ideal path model from the path generation module; the multi-dimensional evaluation module performs parallel quantitative analysis on the real-time action data and the ideal path model in terms of spatial accuracy, time efficiency, path fluency, and energy economy, generates multiple dimension scores, and obtains a comprehensive score by weighted fusion of the multiple dimension scores; the multi-dimensional evaluation module sends the comprehensive score to the real-time feedback module;

[0013] The real-time feedback module receives the comprehensive score from the multi-dimensional evaluation module; the real-time feedback module generates a feedback control signal according to the comprehensive score and sends the feedback control signal to the computing server to drive the feedback device; at the same time, the real-time feedback module generates a difficulty parameter adjustment instruction according to the continuous change of the comprehensive score and sends the difficulty parameter adjustment instruction to the path generation module for dynamic adjustment of the preset task target.

[0014] Further, the data acquisition module of the cognitive training path scoring system based on VR interaction includes a signal verification unit and a coordinate extraction unit;

[0015] The signal checking unit receives hand movement data and head posture data from the computing server as an input data stream; the signal checking unit performs data packet sequence continuity verification on the input data stream, when an interruption in the sequence number is detected, a linear interpolation algorithm is started to complete the missing frame with the front and rear frame data, data points exceeding the preset amplitude range are marked as invalid and replaced with the average value of adjacent valid data points, and the verified data stream is output to the coordinate extraction unit;

[0016] The coordinate extraction unit is connected with a sliding window buffer; the coordinate extraction unit receives the verified data stream from the signal checking unit, stores the verified data stream in the sliding window buffer in the order of reception; when the sliding window buffer is full, a data ready signal is triggered; the coordinate extraction unit converts the buffered data in the sliding window buffer into a time-stamped spatial coordinate sequence in batches according to the sensor calibration parameters, and outputs real-time action data.

[0017] Further, the cognitive training path scoring system based on VR interaction of the present application, the path generation module includes a key point recognition unit and a curve fitting unit;

[0018] The key point recognition unit analyzes the obstacle boundary geometric information in the preset task target, calculates the turning point coordinates that must be avoided in the path, combines the turning point coordinates with the preset starting point coordinates and end point coordinates to form an ordered key point set, and outputs the ordered key point set to the curve fitting unit;

[0019] The curve fitting unit receives the ordered key point set from the key point recognition unit; for every two adjacent key points in the ordered key point set, the curve fitting unit takes the front key point as the starting point of the curve segment, the rear key point as the end point of the curve segment, calculates the normal vector of the line connecting the two points, and generates an intermediate control point along the normal vector by a certain distance; for the starting point and the end point of the path, a virtual control point is generated by mirroring the adjacent key point vector; a cubic Bezier curve segment is defined using the starting point of the curve segment, the intermediate control point, the end point of the curve segment and the next key point; the curve fitting unit uniformly parameterizes and samples all the curve segments to generate an ideal spatial coordinate sequence.

[0020] Further, the cognitive training path scoring system based on VR interaction of the present application, the multi-dimensional evaluation module includes a data alignment unit, a parallel evaluation unit and a weighted fusion unit;

[0021] The data alignment unit receives real-time action data from the data acquisition module and receives an ideal path model from the path generation module; the data alignment unit matches a timestamp sequence of the real-time action data with an ideal time sequence of the ideal path model using a dynamic time warping algorithm, and calculates an optimal time mapping path; the data alignment unit performs time axis alignment on a spatial coordinate sequence in the real-time action data and an ideal spatial coordinate sequence in the ideal path model using the optimal time mapping path, and outputs the aligned real-time action data and the aligned ideal path model to the parallel evaluation unit;

[0022] The parallel evaluation unit receives the aligned real-time action data and the aligned ideal path model from the data alignment unit; the parallel evaluation unit includes a spatial accuracy submodule, a time efficiency submodule, a path fluency submodule, and an energy economy submodule; the spatial accuracy submodule calculates the Euclidean distance based on the spatial coordinate sequence in the aligned real-time action data and the ideal spatial coordinate sequence in the aligned ideal path model at each mapping point of the optimal time mapping path, and calculates the average to output a spatial accuracy score; the time efficiency submodule compares the ratio of the timestamp sequence of the aligned real-time action data and the ideal time sequence of the aligned ideal path model, and outputs a time efficiency score; the path fluency submodule calculates the curvature by second-order difference on the spatial coordinate sequence in the aligned real-time action data and analyzes and calculates the distribution entropy value, and outputs a path fluency score; the energy economy submodule calculates the energy consumption integral of the motion acceleration corresponding to the aligned real-time action data, and outputs an energy economy score; the parallel evaluation unit synchronously outputs the spatial accuracy score, the time efficiency score, the path fluency score, and the energy economy score to the weighted fusion unit;

[0023] The weighted fusion unit receives the spatial accuracy score, the time efficiency score, the path fluency score, and the energy economy score from the parallel evaluation unit; the weighted fusion unit linearly combines the spatial accuracy score, the time efficiency score, the path fluency score, and the energy economy score according to a predefined weight coefficient, and outputs a comprehensive score.

[0024] Further, the VR interaction-based cognitive training path scoring system of the present application is connected with a historical database;

[0025] The historical database stores historical comprehensive scores and historical dimension scores of the user, and the historical dimension scores include historical spatial accuracy scores, historical time efficiency scores, historical path fluency scores, and historical energy economy scores; the weighted fusion unit reads the historical dimension scores from the historical database, and extracts historical fluctuation characteristics of the historical spatial accuracy scores, the historical time efficiency scores, the historical path fluency scores, and the historical energy economy scores;

[0026] The weighting fusion unit is internally provided with a weight mapping table, the weight mapping table stores a corresponding relationship between historical score fluctuation characteristics and weight coefficient adjustment strategies, the weighting fusion unit matches the extracted historical fluctuation characteristics with the weight mapping table to obtain a corresponding weight coefficient adjustment strategy, the weighting fusion unit dynamically adjusts the weight coefficients that are used for weighting and fusing the spatial accuracy score, the time efficiency score, the path fluency score and the energy economy score next time according to the weight coefficient adjustment strategy, and the weighting fusion unit caches the adjusted weight coefficients as new predefined weight coefficients to the historical database.

[0027] Further, the real-time feedback module of the VR interaction-based cognitive training path scoring system comprises an adaptive controller, the real-time feedback module comprises the adaptive controller, and the adaptive controller is connected with a score cache area.

[0028] The real-time feedback module stores the comprehensive scores of the continuous multiple periods received from the multi-dimensional evaluation module to the score cache area.

[0029] The adaptive controller obtains the comprehensive scores of the continuous multiple periods from the score cache area and calculates a moving average change rate of the comprehensive scores.

[0030] The adaptive controller compares the calculated moving average change rate with preset multiple difficulty level thresholds.

[0031] The adaptive controller generates a difficulty adjustment instruction according to the comparison result and sends the difficulty adjustment instruction to the path generation module to modify the path sampling step length parameter or the standard motion speed parameter in the preset task target.

[0032] Further, the adaptive controller of the VR interaction-based cognitive training path scoring system adopts a preset model predictive control algorithm.

[0033] The model predictive control algorithm is internally provided with a prediction model related to system dynamics.

[0034] The adaptive controller inputs the moving average change rate into the prediction model to rollingly optimize the system state in a future time window.

[0035] The adaptive controller takes the rollingly optimized result as an output quantity of the difficulty adjustment instruction.

[0036] The output quantity of the difficulty adjustment instruction is used to determine a specific adjustment value of the path sampling step length parameter or the standard motion speed parameter.

[0037] Further, the VR interaction-based cognitive training path scoring system is internally provided with a data synchronization bus and a shared memory area.

[0038] The data synchronization bus connects the data acquisition module, the path generation module, the multi-dimensional evaluation module and the real-time feedback module to the shared memory area;

[0039] The data acquisition module writes the real-time action data obtained after preprocessing into the shared memory area through the data synchronization bus, and adds a unified timestamp mark to the written real-time action data;

[0040] The path generation module and the multi-dimensional evaluation module query the shared memory area based on the unified timestamp mark through the data synchronization bus, and read the real-time action data with the same timestamp mark.

[0041] Further, the wireless communication between the motion capture handle and the computing server of the cognitive training path scoring system based on VR interaction adopts a time division multiple access protocol;

[0042] The video transmission between the VR head-mounted display and the computing server adopts a frame synchronization mechanism;

[0043] The computing server is provided with a data alignment unit;

[0044] The data alignment unit receives the hand motion data transmitted through the time division multiple access protocol and the head pose data transmitted through the frame synchronization mechanism from the communication interface of the computing server;

[0045] The data alignment unit analyzes the time slot allocation information of the time division multiple access protocol to obtain the wireless transmission delay of the hand motion data, and analyzes the vertical blanking interval signal of the frame synchronization mechanism to obtain the video frame transmission delay of the head pose data;

[0046] The data alignment unit offsets and compensates the timestamps of the hand motion data and the head pose data respectively according to the wireless transmission delay and the video frame transmission delay, and outputs the timestamp-calibrated hand motion data and head pose data to the data acquisition module.

[0047] In a second aspect, the application provides a cognitive training path scoring method based on VR interaction, which is applied to the cognitive training path scoring system based on VR interaction as described, and includes:

[0048] Step S1: collecting user hand motion data and head pose data, and sending the hand motion data and the head pose data to a computing server;

[0049] Step S2: preprocessing the hand motion data and the head pose data from step S1 in the computing server, and outputting real-time action data including a spatial coordinate sequence and a timestamp;

[0050] Step S3: receiving real-time action data from step S2, processing in combination with preset task target to generate ideal path model including ideal spatial coordinate sequence and ideal time sequence;

[0051] Step S4: receiving real-time action data from step S2 and ideal path model from step S3, performing parallel quantitative analysis on real-time action data and ideal path model in terms of spatial accuracy, time efficiency, path fluency and energy economy to generate multiple dimension scores;

[0052] Step S5: weighting and fusing multiple dimension scores from step S4 to obtain comprehensive score;

[0053] Step S6: generating feedback control signal according to comprehensive score from step S5 and driving feedback device to perform visual and audio feedback corresponding to feedback control signal;

[0054] Step S7: generating difficulty parameter adjustment instruction according to continuous change of comprehensive score from step S5 and feeding difficulty parameter adjustment instruction back to step S3 to dynamically adjust preset task target.

[0055] Advantages of the present application;

[0056] The present application integrates multi-modal data acquisition, ideal path generation, multi-dimensional quantitative evaluation and real-time feedback control, constructs a cognitive training path scoring system based on VR interaction, solves the problems of single evaluation dimension and feedback lag caused by lack of multi-modal data fusion quantitative analysis model in existing VR training system, the system synchronously collects hand movement data and head posture data through action capture handle and VR head-mounted display, outputs real-time action data after pre-processing by a computing server, generates ideal path model in combination with preset task target by a path generation module, calculates spatial accuracy, time efficiency, path fluency and energy economy score in parallel after aligning time axis by a multi-dimensional evaluation module using dynamic time warping algorithm, outputs comprehensive score by a weighting fusion unit in combination with historical database to dynamically adjust weight coefficient, drives feedback device to provide instant visual and audio guidance according to comprehensive score by a real-time feedback module, and generates difficulty parameter adjustment instruction to dynamically optimize training task through an adaptive controller to analyze continuous change of score, so as to realize comprehensive quantitative cognitive ability, real-time feedback guidance and adaptive difficulty adjustment, and improve rehabilitation training accuracy and effect. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and obviously, other drawings can be obtained by the drawings without creative labor for those skilled in the art.

[0058] Figure 1A flowchart of a cognitive training path scoring method based on VR interaction of the present application. DETAILED DESCRIPTION

[0059] In order to make the technical solutions of the present application clearer, the present application will be described below in detail with reference to specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. The present application provided by the embodiments of the present application will be described in detail below with reference to the drawings. For the purpose of better understanding the present application, the present application will be further described in detail below.

[0060] In a first aspect, the present application provides a cognitive training path scoring system based on VR interaction, comprising a physical device and a control device, the control device being in communication connection with the physical device;

[0061] The physical device comprises a VR headset, a motion capture handle, a computing server and a feedback device. The motion capture handle is used to collect user hand movement data and send the hand movement data to the computing server. The VR headset is used to display a virtual environment and collect head pose data and send the head pose data to the computing server. The computing server is used to receive and process the hand movement data and the head pose data. The feedback device is used to receive instructions from the computing server and provide visual and audio feedback;

[0062] The control device comprises a data acquisition module, a path generation module, a multi-dimensional evaluation module and a real-time feedback module;

[0063] The data acquisition module receives the hand movement data and the head pose data from the computing server, pre-processes the hand movement data and the head pose data, and outputs real-time action data including spatial coordinate sequences and time stamps. The data acquisition module sends the real-time action data to the path generation module and the multi-dimensional evaluation module;

[0064] The path generation module receives the real-time action data from the data acquisition module, takes the real-time action data as input, combines with the preset task target, and generates an ideal path model including ideal spatial coordinate sequences and ideal time sequences. The path generation module sends the ideal path model to the multi-dimensional evaluation module;

[0065] The multi-dimensional evaluation module receives the real-time action data from the data acquisition module and the ideal path model from the path generation module. The multi-dimensional evaluation module performs parallel quantitative analysis on the real-time action data and the ideal path model in terms of spatial accuracy, time efficiency, path fluency and energy economy, generates multiple dimension scores, and obtains a comprehensive score by weighted fusion of the multiple dimension scores. The multi-dimensional evaluation module sends the comprehensive score to the real-time feedback module;

[0066] The real-time feedback module receives the comprehensive score from the multi-dimensional evaluation module; the real-time feedback module generates a feedback control signal according to the comprehensive score, and sends the feedback control signal to the computing server to drive the feedback device; at the same time, the real-time feedback module generates a difficulty parameter adjustment instruction according to the continuous change of the comprehensive score, and sends the difficulty parameter adjustment instruction to the path generation module for dynamically adjusting the preset task target.

[0067] The cognitive training path scoring system based on VR interaction includes a physical device and a control device, and the control device is in communication connection with the physical device. The physical device includes a VR headset, a motion capture handle, a computing server and a feedback device. The motion capture handle collects user hand movement data and sends it to the computing server, the VR headset displays a virtual environment and collects head posture data and sends it to the computing server, the computing server processes the hand movement data and the head posture data, and the feedback device receives instructions from the computing server and provides visual and audio feedback. The control device is composed of a data acquisition module, a path generation module, a multi-dimensional evaluation module and a real-time feedback module, and each module cooperates in sequence to realize real-time scoring and adaptive adjustment of the cognitive training path.

[0068] Specifically, through the combination of the VR headset and the motion capture handle in the physical device in this embodiment, an immersive simulation scene can be built for the user, and the user's effective motion data can be collected in this scene. The motion data includes the following dimensions: first, the VR headset is equipped with an inertial measurement unit, an optical sensor and other components to collect the user's head posture in real time; second, the motion capture handle collects data such as the user's position, rotation and key state. The above data is fused in the computing server, and through data processing of the control device, the feedback device controls the VR headset to adjust to a visual and audio environment suitable for the user's condition. At the same time, the real-time feedback module also sends difficulty parameter adjustment instructions to the path generation module according to the time sequence change trend of the comprehensive score, for dynamically modifying the complexity of subsequent tasks (such as increasing the number of interfering objects, shortening the task time limit, introducing multi-target selection, etc.), thereby realizing a closed-loop adaptive rehabilitation mechanism of “evaluation-feedback-parameter adjustment-retraining”, and realizing process-based, refined and real-time evaluation of the cognitive ability of Alzheimer's disease patients. Based on the evaluation results, individualized feedback and dynamic difficulty adjustment are driven, which significantly improves the accuracy, adaptability and clinical effectiveness of rehabilitation training.

[0069] The control principle of the control module is as follows:

[0070] The data acquisition module receives hand motion data and head posture data from the computing server, pre-processes and outputs real-time motion data including spatial coordinate sequences and time stamps. The data acquisition module internally includes a signal verification unit and a coordinate extraction unit. The signal verification unit verifies the continuity of the data packet sequence of the input data stream. When a sequence number interruption is detected, a linear interpolation algorithm is started to complete the missing frame, and the data points exceeding the preset amplitude range are marked as invalid and replaced with the average value of the adjacent valid data points. The verified data stream is output. The coordinate extraction unit is connected with a sliding window buffer, which stores the verified data stream in the order of reception. When the sliding window buffer is full, a data ready signal is triggered. The coordinate extraction unit converts the buffered data into spatial coordinate sequences with time stamps based on the sensor calibration parameters, generating real-time motion data. The data acquisition module sends the real-time motion data to the path generation module and the multi-dimensional evaluation module.

[0071] Specifically, the real-time motion data acquisition process of spatial coordinate sequences and time stamps is as follows:

[0072] Step one: define the state vector, input the 3D coordinate sequence collected by the VR handle:

[0073]

[0074] Where P is the position coordinate parameter of the user, x, y, z are the coordinate information of the user's real-time action, and t is the time information of the user's real-time action. user i i i i

[0075] The Kalman filter noise reduction method is used to remove noise from the coordinate sequence. First, define the state vector x (State Vector-x), which uses a 6-dimensional state identifier x to identify the value of x. In order to achieve better smoothing effect and tracking dynamic, the state identifier x includes the position and velocity of the target.

[0076] Define the system state vector as:

[0077]

[0078] : Position of the object on the axis.

[0079] : Velocity of the object on the axis.

[0080] Step two: build the state transition matrix:

[0081] ​​​​​The state transition matrix describes how the state evolves from time step k to k+1 based on the user's motion model, here using a constant velocity model for the six-dimensional state of x, assuming that the velocity remains constant for a short time. dt is the time interval between consecutive measurements (state updates): its formula is:

[0082]

[0083] where: dt is the time interval between two time instants

[0084] Step three, define the process noise covariance matrix, which represents the noise introduced in the state transition process, specifically:

[0085]

[0086] q_px, q_py, q_pz: Position process noise variance. Usually set to a small value.

[0087] q_vx, q_vy, q_vz: Velocity process noise variance.

[0088] Through the above model establishment, the degree of expected speed change is reflected. The larger the value, the faster the filter reacts to changes in observations (but may introduce more noise); the smaller the value, the more the filter "believes" the model prediction.

[0089] Step four: Kalman filter iteration:

[0090] This step first defines the observation matrix and observation noise covariance matrix, defines the initial state (x0) and the initial state estimation error covariance matrix (P0), and the Kalman filter iteration process. For each time step k = 1,2, 3,..., N in the sequence:

[0091] This process is divided into two steps: prediction and update, where:

[0092] Prediction step:

[0093] Use the optimal estimate state x_{k-1|k-1} and covariance P_{k-1|k-1} at the last time (k-1) to predict the prior state and covariance at the current time (k):

[0094]

[0095]

[0096] x_{k|k-1} is the current state predicted based on the model.

[0097] P_{k|k-1} is the covariance of the predicted state (uncertainty).

[0098] Update step:

[0099] Obtain the actual observation at time k, z_k = [px_k, py_k, pz_k]^T.

[0100] Compute the Kalman gain K_k:

[0101]

[0102] The Kalman gain determines the weight of the predicted value and the observation in the final estimate.

[0103] R large -> K_k small -> more trust the prediction;

[0104] R small -> K_k large -> more trust the observation.

[0105] Correct the predicted state with the observation z_k to get the optimal posterior estimate x_{k|k}:

[0106]

[0107] is the observation residual / innovation.

[0108] Update the state estimation error covariance:

[0109] I is the identity matrix.

[0110] Step five: output the denoised result:

[0111] At each time step k, the optimal posterior state estimate x_{k|k} obtained after the update step is the smoothed state after Kalman filtering. We extract the position components from this state vector, for the 6-dimensional state:

[0112] [px_k_smoothed, py_k_smoothed, pz_k_smoothed] = [x_{k|k}(1), x_{k|k}(2), x_{k|k}(3)]

[0113] Collect all the [px_k_smoothed, py_k_smoothed, pz_k_smoothed] at time k to get the denoised 3D coordinate sequence.

[0114] By introducing a six-dimensional constant velocity Kalman filter, the high-frequency noise introduced by the VR sensor due to jitter, delay or occlusion is effectively suppressed, while the dynamic characteristics of the user's real motion are preserved. Compared with simple moving average or low-pass filtering, this method achieves a good balance between position smoothness and motion followability, significantly improving the accuracy of subsequent path matching and cognitive assessment.

[0115] The path generation module receives real-time motion data output by the data acquisition module, and generates an ideal path model including an ideal spatial coordinate sequence and an ideal time sequence in combination with a preset task target. The path generation module includes a key point identification unit and a curve fitting unit. The key point identification unit analyzes the obstacle boundary geometric information in the preset task target, calculates the turning point coordinates that must be avoided in the path, and combines the turning point coordinates with the preset start point coordinates and end point coordinates to form an ordered key point set. The curve fitting unit receives the ordered key point set, takes the front key point as the start point of the curve segment and the rear key point as the end point of the curve segment for each two adjacent key points, calculates the normal vector of the direction of the connecting line of the two points and generates an intermediate control point by offsetting a specific distance along the normal vector, generates virtual control points by mirroring the adjacent key point vectors for the path start point and the path end point, defines a cubic Bezier curve segment using the curve segment start point, the intermediate control point, the curve segment end point and the next key point, and uniformly parameterizes and samples all curve segments to generate an ideal spatial coordinate sequence. The path generation module sends the ideal path model to the multi-dimensional evaluation module.

[0116] The key point identification unit is used to analyze the environmental semantic information contained in the preset task target, especially the boundary geometric data of obstacles (such as polygon vertex coordinates, bounding boxes or mesh models). Based on this geometric information, in combination with the position constraints of the start point and the end point, the unit calculates the path turning point coordinates that must be avoided by the user during task execution through path planning algorithms, including but not limited to the visibility graph method, A* or the fast marching method. These turning points represent key navigation nodes for bypassing obstacles or crossing passages. Subsequently, the key point identification unit combines the turning point coordinates with the preset task start point coordinates and the task end point coordinates in spatial order to form an ordered key point set, and outputs the set to the curve fitting unit.

[0117] The curve fitting unit receives the ordered key point set from the key point identification unit and constructs a smooth and executable ideal path model based on the set. Specifically, for any two adjacent key points P i and P i +1 in the set, the curve fitting unit takes P i as the start point of the current curve segment and P i +1 as the end point, and performs the following operations:

[0118] (1) Calculate the vector unit directional vector of the stable normal vector;

[0119] (2) In the plane perpendicular to v, select a stable normal vector direction;

[0120] (3) Offset a preset distance d along the normal vector direction to generate an intermediate control point C i , which is used to guide the curve to bypass the potential collision area;

[0121] (4) For the first segment (start point) and the last segment (end point) of the path, in order to ensure the continuity of the tangent at the end points of the curve, the mirror vector method is used: that is, the displacement vector between adjacent key points is copied and extended in the opposite direction to generate virtual control points C0 and Cn;

[0122] (5) Use the cubic Bezier curve formula

[0123]

[0124] where the control point configuration is dynamically determined according to the segment position (the first segment uses P0, C0, C1, P1, the middle segment uses P i ,C i ,C i +1, P i +1, etc.), and the continuous spline path is constructed segment by segment;

[0125] (6) Uniformly parameterize the entire composite Bezier path to generate a high-density, time-stamped ideal spatial coordinate sequence as the reference trajectory for the subsequent multi-dimensional evaluation module.

[0126] The multi-dimensional evaluation module receives the real-time action data of the data acquisition module and the ideal path model of the path generation module, performs parallel quantitative analysis on spatial accuracy, time efficiency, path fluency and energy economy, and generates a comprehensive score after weighted fusion of multiple dimension scores. The multi-dimensional evaluation module includes a data alignment unit, a parallel evaluation unit and a weighted fusion unit. The data alignment unit matches the timestamp sequence of the real-time action data and the ideal time sequence of the ideal path model using the dynamic time warping algorithm, calculates the optimal time mapping path, aligns the spatial coordinate sequence on the time axis using the optimal time mapping path, and outputs the aligned real-time action data and the aligned ideal path model to the parallel evaluation unit. The parallel evaluation unit includes a spatial accuracy submodule, a time efficiency submodule, a path fluency submodule and an energy economy submodule. The spatial accuracy submodule calculates the average Euclidean distance based on the aligned spatial coordinate sequence to output a spatial accuracy score. The time efficiency submodule compares the ratio of the timestamp sequence and the ideal time sequence to output a time efficiency score. The path fluency submodule calculates the curvature by second-order difference of the spatial coordinate sequence and analyzes and calculates the distribution entropy value to output a path fluency score. The energy economy submodule calculates the energy consumption integral of the motion acceleration to output an energy economy score. The weighted fusion unit linearly combines the four dimension scores according to the predefined weight coefficients to output a comprehensive score. The weighted fusion unit is connected to a historical database, which stores user historical score data. The weighted fusion unit reads the historical dimension scores to extract historical fluctuation characteristics, matches the weight coefficient adjustment strategy through the built-in weight mapping table, and dynamically adjusts the weight coefficients and caches them to the historical database. The multi-dimensional evaluation module sends the comprehensive score to the real-time feedback module.

[0127] Specifically,

[0128] The spatial accuracy submodule calculates the average Euclidean distance based on the aligned spatial coordinate sequence to output a spatial accuracy score, and its formula is:

[0129] Wherein:

[0130] P ideal : spatial coordinates of the ideal path at arc length parameter s;

[0131] P user(s) : spatial coordinates of the user path at the corresponding arc length position s after time alignment and resampling;

[0132] L total : total length of the ideal path;

[0133] S s is the spatial accuracy score to be obtained;

[0134] ​The time efficiency submodule outputs a time efficiency score by comparing the ratio of the timestamp sequence and the ideal time sequence, whose formula is:

[0135]

[0136] Wherein:

[0137] T user : the actual total time consumption of the user to complete the current task;

[0138] T ideal : the preset standard task time consumption in the ideal path model;

[0139] T ideal / T user -1: represents the relative time deviation rate, that is, the proportion of the actual time consumption deviating from the ideal time consumption;

[0140] S T is the time precision score to be obtained;

[0141] The path fluency submodule is to calculate the curvature by second-order difference of the spatial coordinate sequence and analyze and calculate the distribution entropy value to output the path fluency score, whose formula is:

[0142] ;

[0143] Wherein: θ i is the curvature change entropy;

[0144] N is the number of effective trajectories;

[0145] S f is the path fluency score to be obtained;

[0146] The energy consumption economy submodule calculates the energy consumption integral of the motion acceleration to output the energy consumption economy score, whose formula is:

[0147]

[0148] Wherein: is the energy consumption integral

[0149] E min is the minimum energy consumption corresponding to the ideal path

[0150] S E is the energy consumption economy score to be obtained.

[0151] The comprehensive score is: Score =

[0152] Wherein: the weights α, β, γ, δ are dynamically adjusted based on the cognitive training target

[0153] After obtaining the corresponding integral in the above manner, the historical database is used to store the user's historical score data, the weighting fusion unit reads the historical dimension score to extract the historical fluctuation characteristics, the weight coefficient adjustment strategy is matched through the built-in weight mapping table, the weight coefficient is dynamically adjusted and cached to the historical database for subsequent task evaluation benchmark update, thereby forming a personalized weight self-adaptive closed loop of "evaluation-analysis-adjustment-re-evaluation".

[0154] The real-time feedback module receives the comprehensive score of the multi-dimensional evaluation module, generates a feedback control signal and sends it to the computing server to drive the feedback device, and simultaneously generates a difficulty parameter adjustment instruction according to the continuous change of the comprehensive score and sends it to the path generation module to dynamically adjust the preset task target. The real-time feedback module includes an adaptive controller, which is connected with a score cache area, which stores the comprehensive score of a plurality of continuous periods, and the adaptive controller calculates the moving average change rate of the comprehensive score, compares the moving average change rate with a plurality of difficulty level thresholds preset, and generates a difficulty adjustment instruction. The adaptive controller adopts a model predictive control algorithm, and the model predictive control algorithm has a built-in system dynamic prediction model. The adaptive controller inputs the moving average change rate into the prediction model to optimize the system state in the future time window, and uses the optimization result as the output of the difficulty adjustment instruction to modify the path sampling step length parameter or the standard motion speed parameter in the preset task target.

[0155] The computing server is internally deployed with a data synchronization bus and a shared memory area, and the data synchronization bus connects the data acquisition module, the path generation module, the multi-dimensional evaluation module and the real-time feedback module to the shared memory area. The data acquisition module writes the real-time action data into the shared memory area through the data synchronization bus and attaches a unified time stamp mark, and the path generation module and the multi-dimensional evaluation module query the shared memory area based on the unified time stamp mark to read the real-time action data with the same time stamp mark, thereby realizing data synchronization between modules. The wireless communication between the motion capture handle and the computing server adopts a time division multiple access protocol, and the video transmission between the VR head-mounted display and the computing server adopts a frame synchronization mechanism. The computing server is provided with a data alignment unit, which analyzes the time slot allocation information of the time division multiple access protocol to obtain the wireless transmission delay of the hand motion data, analyzes the vertical blanking interval signal of the frame synchronization mechanism to obtain the video frame transmission delay of the head posture data, offsets the time stamp, and outputs the time stamp calibrated data to the data acquisition module.

[0156] Through the cooperative work of the above-mentioned modules, the system realizes the closed loop processing from data acquisition, path generation, multi-dimensional evaluation to real-time feedback, and provides precise path score and self-adaptive adjustment capability for cognitive training.

[0157] The signal checking unit receives hand motion data and head posture data from the computing server as input data streams, which are transmitted in the form of a data packet sequence. The signal checking unit performs continuity verification on the data packet sequence, checking whether the data packet sequence numbers are continuous. When a sequence number interruption is detected, the signal checking unit starts a linear interpolation algorithm to calculate the approximate value of the missing frame using the data values of the frames before and after the interruption to complete the data stream. For data point amplitude values, the signal checking unit presets an amplitude range, marks data points exceeding the range as invalid, and replaces the invalid points with the average value of adjacent valid data points, thereby outputting the checked data stream. The coordinate extraction unit is connected with a sliding window buffer. The coordinate extraction unit receives the checked data stream from the signal checking unit, stores the data stream in the sliding window buffer in the order of reception; the sliding window buffer has a fixed capacity, and triggers a data ready signal when it is full; the coordinate extraction unit converts the buffered data in the sliding window buffer into a sequence of spatial coordinates with timestamps in batches according to the sensor calibration parameters, and outputs real-time action data. The data preprocessing of the signal checking unit is used to improve data integrity, and the sliding window mechanism of the coordinate extraction unit realizes batch data processing, thereby improving system efficiency.

[0158] The path generation module includes a key point identification unit and a curve fitting unit. The key point identification unit analyzes the obstacle boundary geometric information in the preset task target, calculates the coordinates of the turning points that must be avoided in the path; the turning point coordinates are combined with the preset starting point coordinates and end point coordinates to form an ordered key point set. The curve fitting unit receives the ordered key point set from the key point identification unit; for every two adjacent key points in the ordered key point set, the curve fitting unit takes the former key point as the starting point of the curve segment and the latter key point as the end point of the curve segment; calculates the normal vector of the line connecting the two points, and generates an intermediate control point by offsetting a specific distance along the normal vector; for the starting point and the end point of the path, a virtual control point is generated by mirroring the adjacent key point vector; a cubic Bezier curve segment is defined using the starting point of the curve segment, the intermediate control point, the end point of the curve segment, and the next key point; the curve fitting unit uniformly parameterizes and samples all curve segments to generate an ideal spatial coordinate sequence. The key point identification unit determines the key positions of the path, and the curve fitting unit realizes smooth connection of the path by Bezier curve to generate an ideal path model.

[0159] The multi-dimensional evaluation module includes a data alignment unit, a parallel evaluation unit and a weighted fusion unit. The data alignment unit receives real-time action data from the data acquisition module and receives an ideal path model from the path generation module; the data alignment unit matches a timestamp sequence of the real-time action data with an ideal time sequence of the ideal path model using a dynamic time warping algorithm, calculates an optimal time mapping path; uses the optimal time mapping path to align the spatial coordinate sequence in the real-time action data and the ideal spatial coordinate sequence in the ideal path model on the time axis, and outputs the aligned real-time action data and the aligned ideal path model. The parallel evaluation unit includes a spatial accuracy sub-module, a time efficiency sub-module, a path fluency sub-module and an energy economy sub-module; the spatial accuracy sub-module calculates the Euclidean distance based on the aligned spatial coordinate sequence at each mapping point of the optimal time mapping path and takes the average, and outputs the spatial accuracy score; the time efficiency sub-module compares the ratio of the aligned timestamp sequence and the ideal time sequence, and outputs the time efficiency score; the path fluency sub-module calculates the curvature by second-order difference on the aligned spatial coordinate sequence and analyzes and calculates the distribution entropy value, and outputs the path fluency score; the energy economy sub-module calculates the energy consumption integral of the motion acceleration corresponding to the aligned real-time action data, and outputs the energy economy score. The weighted fusion unit linearly combines the four-dimensional scores according to the predefined weight coefficients, and outputs the comprehensive score. The data alignment unit realizes time synchronization, the parallel evaluation unit multi-dimensionally quantitatively analyzes, and the weighted fusion unit integrates the scores.

[0160] The weighted fusion unit is connected with a historical database, and the historical database stores the historical comprehensive score and the historical dimension score of the user, and the historical dimension score includes the historical spatial accuracy score, the historical time efficiency score, the historical path fluency score and the historical energy economy score. The weighted fusion unit reads the historical dimension score from the historical database, extracts the historical fluctuation characteristics, such as the change trend or stability of the score. The weighted fusion unit has a weight mapping table built-in, and the weight mapping table stores the correspondence between the historical score fluctuation characteristics and the weight coefficient adjustment strategy; the weighted fusion unit matches the extracted historical fluctuation characteristics with the weight mapping table to obtain the corresponding weight coefficient adjustment strategy. According to the weight coefficient adjustment strategy, the weighted fusion unit dynamically adjusts the weight coefficient to be used for the next weighted fusion, such as increasing the weight of the fluctuation dimension or reducing the weight of the stable dimension; the adjusted weight coefficient is cached as a new predefined weight coefficient to the historical database. The historical database provides historical data support, the weight mapping table realizes strategy mapping, and dynamically adjusting the weight coefficient makes the scoring system adapt to individual user changes.

[0161] The application further sets an adaptive controller and a score buffer area in the real-time feedback module, the score buffer area is used for storing the comprehensive scores of continuous multiple periods received from the multi-dimensional evaluation module, the adaptive controller reads the comprehensive scores from the score buffer area periodically and calculates a moving average change rate, the moving average change rate is obtained by weighted average processing of the continuous score sequence through a sliding window algorithm to smooth short-term fluctuations, the adaptive controller compares the moving average change rate with a plurality of difficulty level thresholds preset according to historical training data, and generates a difficulty adjustment instruction according to the comparison result, the difficulty adjustment instruction includes the modification amount of the path sampling step length parameter or the standard movement speed parameter, and the difficulty adjustment instruction is sent to the path generation module, the path generation module adjusts the path generation parameter in the preset task target according to the difficulty adjustment instruction, so as to realize adaptive adjustment of the training difficulty.

[0162] In the application, the adaptive controller adopts a model predictive control algorithm, the model predictive control algorithm has a prediction model related to system dynamics built-in, the prediction model is constructed based on user historical score data and behavior patterns, the adaptive controller inputs the moving average change rate into the prediction model, the prediction model optimizes the system state in a future time window in a rolling manner, the rolling optimization process minimizes the score deviation by iteratively solving an optimal control problem, and the adaptive controller takes the result of the rolling optimization as the output of the difficulty adjustment instruction, the output is specifically a reduction amount of the path sampling step length parameter or an increment of the standard movement speed parameter, the output of the difficulty adjustment instruction is converted into a parameter adjustment value through a linear mapping relationship, and the path generation module updates the task target configuration after receiving the parameter adjustment value to adapt to the real-time cognitive ability change of the user.

[0163] In the application, a data synchronization bus and a shared memory area are deployed inside the computing server, the data synchronization bus connects the data acquisition module, the path generation module, the multi-dimensional evaluation module and the real-time feedback module to the shared memory area by using a high-speed serial communication protocol, the shared memory area is divided into a plurality of data blocks for storing real-time action data, the data acquisition module writes the preprocessed real-time action data into the shared memory area through the data synchronization bus, and a unified timestamp mark is attached to each data block, the timestamp mark is generated synchronously by a system clock, the path generation module and the multi-dimensional evaluation module query the shared memory area through the data synchronization bus, and search for real-time action data with the same timestamp based on the unified timestamp mark, and the data synchronization bus notifies the module of a data ready state through an interrupt mechanism, so that the synchronization and consistency of data access between the modules are realized.

[0164] The wireless communication between the motion capture handle and the computing server in the application adopts a time division multiple access protocol, the time division multiple access protocol divides the communication time into periodic time slots, the motion capture handle sends hand movement data in a designated time slot, and the video transmission between the VR head-mounted display and the computing server adopts a frame synchronization mechanism, the frame synchronization mechanism aligns the video frame transmission based on a vertical blanking interval signal, and the data alignment unit is arranged in the computing server, the data alignment unit receives the hand movement data and the head posture data from the communication interface of the computing server, the data alignment unit analyzes the time slot allocation information of the time division multiple access protocol to obtain the wireless transmission delay of the hand movement data, analyzes the vertical blanking interval signal of the frame synchronization mechanism to obtain the video frame transmission delay of the head posture data, the data alignment unit calculates the timestamp offset according to the wireless transmission delay and the video frame transmission delay, compensates and corrects the timestamps of the hand movement data and the head posture data, and outputs the data after timestamp calibration to the data acquisition module, so as to ensure the time alignment accuracy of the multi-source data.

[0165] In a second aspect, referring to Figure 1 The application provides a cognitive training path scoring method based on VR interaction, which is applied to the cognitive training path scoring system based on VR interaction.

[0166] Step S1: collecting hand movement data and head posture data of a user and sending the hand movement data and the head posture data to a computing server;

[0167] Step S2: preprocessing the hand movement data and the head posture data from step S1 in the computing server, and outputting real-time action data including a spatial coordinate sequence and a timestamp;

[0168] Step S3: receiving the real-time action data from step S2, processing in combination with a preset task target, and generating an ideal path model including an ideal spatial coordinate sequence and an ideal time sequence;

[0169] Step S4: receiving the real-time action data from step S2 and the ideal path model from step S3, and performing parallel quantitative analysis on the real-time action data and the ideal path model in terms of spatial accuracy, time efficiency, path fluency, and energy economy, to generate multiple dimension scores;

[0170] Step S5: weighting and fusing the multiple dimension scores from step S4 to obtain a comprehensive score;

[0171] Step S6: generating a feedback control signal according to the comprehensive score from step S5, and driving a feedback device to perform visual and audio feedback corresponding to the feedback control signal;

[0172] Step S7: According to the continuous change of the comprehensive score from step S5, generate difficulty parameter adjustment instructions, and feed back the difficulty parameter adjustment instructions to step S3 to dynamically adjust the preset task target.

[0173] The technical scheme of the present application constructs a cognitive training path scoring system based on VR interaction. The system integrates multi-modal data acquisition, ideal path generation, multi-dimensional quantitative evaluation and real-time feedback control to solve the problems of single evaluation dimension, feedback lag and difficulty in dynamic optimization caused by the lack of multi-modal data fusion quantitative analysis model in existing VR training systems. The system first synchronously acquires hand movement data and head posture data of the user through the VR headset and motion capture handle in the physical device. The multi-modal data is preprocessed by the computing server, including data packet sequence continuity verification, invalid data point replacement and coordinate conversion, and the real-time action data including spatial coordinate sequence and timestamp are output. The real-time action data is transmitted to the data acquisition module of the control device, and then the path generation module generates an ideal path model by combining the preset task target, key point recognition and curve fitting. The model includes ideal spatial coordinate sequence and ideal time sequence. The multi-dimensional evaluation module receives real-time action data and ideal path model, performs time axis alignment using dynamic time warping algorithm, and performs quantitative analysis of spatial accuracy, time efficiency, path fluency and energy economy in parallel, generates multiple dimension scores, and then dynamically adjusts the weight coefficient by combining the historical database of user historical score fluctuation characteristics through the weighted fusion unit, and outputs the comprehensive score. The real-time feedback module immediately generates a feedback control signal according to the comprehensive score to drive the feedback device to provide visual and audio feedback, and at the same time analyzes the continuous change of the comprehensive score through the adaptive controller, generates difficulty parameter adjustment instructions using model predictive control algorithm, and dynamically adjusts the preset task target parameters in the path generation module, such as path sampling step or standard movement speed. This multi-modal data fusion quantitative analysis model realizes the comprehensiveness of the evaluation dimension, the real-time feedback mechanism realizes the immediate guidance, and the closed-loop adaptive adjustment optimizes the dynamic accuracy of the rehabilitation training effect.

[0174] In view of the technical problems of single evaluation dimension, feedback lag and lack of multi-modal data quantitative analysis model in the background technology of virtual reality rehabilitation training system, the present application provides a specific embodiment of a cognitive training path scoring system and method based on VR interaction. The system is applied to the cognitive rehabilitation training scene of Alzheimer's patients, and realizes real-time scoring and adaptive adjustment through multi-dimensional data fusion analysis.

[0175] The system comprises a physical device and a control device, and the physical device and the control device are connected through communication to realize data interaction. The physical device comprises a VR head-mounted display, a motion capture handle, a computing server and a feedback device. The motion capture handle is used to collect user hand movement data and send the hand movement data to the computing server. The VR head-mounted display is used to display a virtual environment and collect head posture data and send the head posture data to the computing server. The computing server is used to receive and process the hand movement data and the head posture data. The feedback device is used to receive instructions from the computing server and provide visual and audio feedback.

[0176] The control device comprises a data acquisition module, a path generation module, a multi-dimensional evaluation module and a real-time feedback module. The data acquisition module receives the hand movement data and the head posture data from the computing server, pre-processes the hand movement data and the head posture data, and outputs real-time action data comprising a spatial coordinate sequence and a time stamp. The data acquisition module sends the real-time action data to the path generation module and the multi-dimensional evaluation module. The path generation module receives the real-time action data from the data acquisition module, takes the real-time action data as input, combines a preset task target, and generates an ideal path model comprising an ideal spatial coordinate sequence and an ideal time sequence. The path generation module sends the ideal path model to the multi-dimensional evaluation module. The multi-dimensional evaluation module receives the real-time action data from the data acquisition module and receives the ideal path model from the path generation module. The multi-dimensional evaluation module performs parallel quantitative analysis on the real-time action data and the ideal path model in terms of spatial accuracy, time efficiency, path fluency and energy economy, generates multiple dimension scores, performs weighted fusion on the multiple dimension scores, and obtains a comprehensive score. The multi-dimensional evaluation module sends the comprehensive score to the real-time feedback module. The real-time feedback module receives the comprehensive score from the multi-dimensional evaluation module. The real-time feedback module generates a feedback control signal according to the comprehensive score, and sends the feedback control signal to the computing server to drive the feedback device. At the same time, the real-time feedback module generates a difficulty parameter adjustment instruction according to the continuous change of the comprehensive score, and sends the difficulty parameter adjustment instruction to the path generation module for dynamically adjusting the preset task target.

[0177] In the data acquisition module, the signal verification unit receives hand motion data and head posture data from the computing server as input data streams. The signal verification unit performs packet sequence continuity verification on the input data streams. When a sequence number interruption is detected, a linear interpolation algorithm is started to complete the missing frame with the previous and next frame data. Data points exceeding the preset amplitude range are marked as invalid and replaced with the average value of adjacent valid data points. The verified data stream is output to the coordinate extraction unit. The coordinate extraction unit is connected with a sliding window buffer. The coordinate extraction unit receives the verified data stream from the signal verification unit and stores it in the sliding window buffer in the order of reception. When the sliding window buffer is full, a data ready signal is triggered. The coordinate extraction unit converts the buffered data in the sliding window buffer into a time-stamped spatial coordinate sequence in batches according to the sensor calibration parameters, and outputs real-time motion data.

[0178] The path generation module includes a key point identification unit and a curve fitting unit. The key point identification unit analyzes the obstacle boundary geometric information in the preset task target, calculates the turning point coordinates that must be avoided in the path, combines the turning point coordinates with the preset starting point coordinates and end point coordinates to form an ordered key point set, and outputs the ordered key point set to the curve fitting unit. The curve fitting unit receives the ordered key point set from the key point identification unit. For every two adjacent key points in the ordered key point set, the curve fitting unit takes the front key point as the starting point of the curve segment and the rear key point as the end point of the curve segment, calculates the normal vector of the line connecting the two points, and generates an intermediate control point along the normal vector by a certain distance. For the starting point and the end point of the path, virtual control points are generated by mirroring the adjacent key point vectors. A cubic Bezier curve segment is defined using the starting point of the curve segment, the intermediate control point, the end point of the curve segment, and the next key point. The curve fitting unit uniformly parameterizes all curve segments to generate an ideal spatial coordinate sequence.

[0179] The multi-dimensional evaluation module comprises a data alignment unit, a parallel evaluation unit and a weighted fusion unit. The data alignment unit receives real-time action data from the data acquisition module and receives an ideal path model from the path generation module. The data alignment unit matches a timestamp sequence of the real-time action data with an ideal time sequence of the ideal path model using a dynamic time warping algorithm to calculate an optimal time mapping path. The data alignment unit uses the optimal time mapping path to perform time axis alignment on a spatial coordinate sequence in the real-time action data and an ideal spatial coordinate sequence in the ideal path model, and outputs the aligned real-time action data and the aligned ideal path model to the parallel evaluation unit. The parallel evaluation unit receives the aligned real-time action data and the aligned ideal path model from the data alignment unit. The parallel evaluation unit comprises a spatial accuracy sub-module, a time efficiency sub-module, a path fluency sub-module and an energy economy sub-module. The spatial accuracy sub-module calculates the Euclidean distance based on the spatial coordinate sequence in the aligned real-time action data and the ideal spatial coordinate sequence in the aligned ideal path model at each mapping point of the optimal time mapping path, and takes the average to output a spatial accuracy score. The time efficiency sub-module compares the ratio of the timestamp sequence of the aligned real-time action data and the ideal time sequence of the aligned ideal path model to output a time efficiency score. The path fluency sub-module calculates the curvature by second-order difference on the spatial coordinate sequence in the aligned real-time action data and analyzes the statistical distribution entropy value to output a path fluency score. The energy economy sub-module calculates the energy consumption integral of the motion acceleration corresponding to the aligned real-time action data to output an energy economy score. The parallel evaluation unit synchronously outputs the spatial accuracy score, the time efficiency score, the path fluency score and the energy economy score to the weighted fusion unit. The weighted fusion unit receives the spatial accuracy score, the time efficiency score, the path fluency score and the energy economy score from the parallel evaluation unit. The weighted fusion unit linearly combines the spatial accuracy score, the time efficiency score, the path fluency score and the energy economy score according to the pre-defined weight coefficients to output a comprehensive score.

[0180] The weighted fusion unit is connected with a history database. The history database stores a history comprehensive score and history dimension scores of the user, the history dimension scores including a history spatial accuracy score, a history time efficiency score, a history path fluency score and a history energy economy score. The weighted fusion unit reads the history dimension scores from the history database, and extracts history fluctuation characteristics of the history spatial accuracy score, the history time efficiency score, the history path fluency score and the history energy economy score. The weighted fusion unit is internally provided with a weight mapping table, the weight mapping table storing a corresponding relationship between the history score fluctuation characteristics and a weight coefficient adjustment strategy. The weighted fusion unit matches the extracted history fluctuation characteristics with the weight mapping table to obtain the corresponding weight coefficient adjustment strategy. The weighted fusion unit dynamically adjusts the weight coefficients to be used for next time weighting fusion of the spatial accuracy score, the time efficiency score, the path fluency score and the energy economy score according to the weight coefficient adjustment strategy. The weighted fusion unit caches the adjusted weight coefficients as new predefined weight coefficients to the history database.

[0181] The real-time feedback module includes an adaptive controller connected with a score cache area. The real-time feedback module stores the continuous multiple-period comprehensive scores received from the multi-dimensional evaluation module to the score cache area. The adaptive controller obtains the continuous multiple-period comprehensive scores from the score cache area, and calculates a moving average change rate of the comprehensive scores. The adaptive controller compares the calculated moving average change rate with preset multiple difficulty level thresholds. The adaptive controller generates a difficulty adjustment instruction according to the comparison result, and sends the difficulty adjustment instruction to the path generation module to modify the path sampling step length parameter or the standard motion speed parameter in the preset task target.

[0182] The adaptive controller adopts a preset model predictive control algorithm. The model predictive control algorithm is internally provided with a prediction model related to system dynamics. The adaptive controller inputs the moving average change rate into the prediction model, and rolls to optimize the system state in a future time window. The adaptive controller takes the roll-optimized result as an output quantity of the difficulty adjustment instruction. The output quantity of the difficulty adjustment instruction is used to determine a specific adjustment value of the path sampling step length parameter or the standard motion speed parameter.

[0183] The computing server is internally provided with a data synchronization bus and a shared memory area. The data synchronization bus connects the data acquisition module, the path generation module, the multi-dimensional evaluation module and the real-time feedback module to the shared memory area. The data acquisition module writes the preprocessed real-time action data into the shared memory area through the data synchronization bus, and attaches a unified time stamp mark to the written real-time action data. The path generation module and the multi-dimensional evaluation module query the shared memory area based on the unified time stamp mark through the data synchronization bus, and read the real-time action data with the same time stamp mark.

[0184] The wireless communication between the motion capture handle and the computing server adopts a time division multiple access protocol. The video transmission between the VR head-mounted display and the computing server adopts a frame synchronization mechanism. A data alignment unit is arranged in the computing server. The data alignment unit receives hand motion data transmitted through the time division multiple access protocol and head pose data transmitted through the frame synchronization mechanism from a communication interface of the computing server. The data alignment unit parses time slot allocation information of the time division multiple access protocol to obtain a wireless transmission delay of the hand motion data. The data alignment unit parses a vertical blanking interval signal of the frame synchronization mechanism to obtain a video frame transmission delay of the head pose data. The data alignment unit offsets and compensates time stamps of the hand motion data and the head pose data, respectively, according to the wireless transmission delay and the video frame transmission delay, and outputs the hand motion data and the head pose data after timestamp calibration to a data acquisition module.

[0185] The cognitive training path scoring method based on VR interaction includes the following steps. Step S1: collect user hand motion data and head pose data, and send the hand motion data and the head pose data to a computing server. Step S2: pre-process the hand motion data and the head pose data from step S1 in the computing server, and output real-time action data including spatial coordinate sequences and time stamps. Step S3: receive the real-time action data from step S2, process in combination with a preset task target, and generate an ideal path model including ideal spatial coordinate sequences and ideal time sequences. Step S4: receive the real-time action data from step S2 and the ideal path model from step S3, and perform parallel quantitative analysis on the real-time action data and the ideal path model in terms of spatial accuracy, time efficiency, path fluency, and energy economy, to generate multiple dimension scores. Step S5: weight and fuse the multiple dimension scores from step S4 to obtain a comprehensive score. Step S6: generate a feedback control signal according to the comprehensive score from step S5, and drive a feedback device to perform visual and audio feedback corresponding to the feedback control signal. Step S7: generate a difficulty parameter adjustment instruction according to the continuous change of the comprehensive score from step S5, and feed the difficulty parameter adjustment instruction back to step S3 to dynamically adjust the preset task target.

[0186] In the implementation process, the method quantifies the user's cognitive ability in real time through a multi-dimension scoring model, dynamically adjusts the weight in combination with historical data, and realizes personalized evaluation. The feedback system provides instant visual and audio guidance according to the scoring results, and optimizes the training difficulty through an adaptive mechanism to improve the rehabilitation training effect. The overall system runs on a computing server platform to realize the real-time and accuracy of data processing.

Claims

1. A VR interaction based cognitive training path scoring system, characterized in that, The physical device and a control device connected with the physical device; The physical device includes a VR headset, a motion capture handle, a computing server and a feedback device, the motion capture handle is used to collect user hand movement data and send the hand movement data to the computing server, the VR headset is used to display a virtual environment and collect head posture data and send the head posture data to the computing server, the computing server is used to receive and process the hand movement data and the head posture data, and the feedback device is used to receive instructions from the computing server and provide visual and audio feedback; The control device includes a data acquisition module, a path generation module, a multi-dimensional evaluation module and a real-time feedback module; The data acquisition module receives the hand movement data and the head posture data from the computing server, pre-processes the hand movement data and the head posture data, and outputs real-time action data including spatial coordinate sequences and time stamps; the data acquisition module sends the real-time action data to the path generation module and the multi-dimensional evaluation module; The path generation module receives the real-time action data from the data acquisition module, takes the real-time action data as input, combines a preset task target, and generates an ideal path model including ideal spatial coordinate sequences and ideal time sequences; the path generation module sends the ideal path model to the multi-dimensional evaluation module; The multi-dimensional evaluation module receives the real-time action data from the data acquisition module and the ideal path model from the path generation module; The multi-dimensional evaluation module performs parallel quantitative analysis on the real-time action data and the ideal path model in terms of spatial accuracy, time efficiency, path fluency and energy economy, generates multiple dimension scores, and obtains a comprehensive score by weighted fusion of the multiple dimension scores; The multi-dimensional evaluation module sends the comprehensive score to the real-time feedback module; The real-time feedback module receives the comprehensive score from the multi-dimensional evaluation module; the real-time feedback module generates the feedback control signal according to the comprehensive score and sends the feedback control signal to the computing server to drive the feedback device; at the same time, the real-time feedback module generates a difficulty parameter adjustment instruction according to the continuous change of the comprehensive score, and sends the difficulty parameter adjustment instruction to the path generation module for dynamically adjusting the preset task target.

2. The VR interaction based cognitive training path scoring system of claim 1, wherein, The data acquisition module includes a signal verification unit and a coordinate extraction unit; The signal verification unit receives the hand movement data and the head posture data from the computing server as input data stream; the signal verification unit performs data packet sequence continuity verification on the input data stream, when detecting sequence number interruption, starts linear interpolation algorithm to complete missing frames with previous and next frame data, marks data points exceeding the preset amplitude range as invalid and replaces them with the average value of adjacent valid data points, and outputs the verified data stream to the coordinate extraction unit; The coordinate extraction unit is connected with a sliding window buffer; the coordinate extraction unit receives the checked data stream from the signal checking unit, and stores the checked data stream into the sliding window buffer in the receiving order; when the sliding window buffer is full, a data ready signal is triggered; the coordinate extraction unit converts the buffered data in the sliding window buffer into a time-stamped spatial coordinate sequence in batches according to sensor calibration parameters, and outputs the real-time action data.

3. The VR interaction based cognitive training path scoring system of claim 1, wherein, The path generation module comprises a key point identification unit and a curve fitting unit; The key point identification unit analyzes the obstacle boundary geometric information in the preset task target, calculates the turning point coordinates that must be avoided in the path, combines the turning point coordinates with preset starting point coordinates and end point coordinates to form an ordered key point set, and outputs the ordered key point set to the curve fitting unit; The curve fitting unit receives the ordered key point set from the key point identification unit; for each two adjacent key points in the ordered key point set, the curve fitting unit takes the front key point as the starting point of a curve segment, takes the rear key point as the end point of the curve segment, calculates the normal vector of the line connecting the two points, and generates an intermediate control point by offsetting a specific distance along the normal vector; for the starting point and the end point of the path, a virtual control point is generated by mirroring the adjacent key point vector; a cubic Bezier curve segment is defined by using the starting point of the curve segment, the intermediate control point, the end point of the curve segment and the next key point; the curve fitting unit uniformly parameterizes and samples all curve segments to generate the ideal spatial coordinate sequence.

4. The VR interaction based cognitive training path scoring system of claim 1, wherein, The multi-dimensional evaluation module comprises a data alignment unit, a parallel evaluation unit and a weighted fusion unit; The data alignment unit receives the real-time action data from the data acquisition module and receives the ideal path model from the path generation module; the data alignment unit matches the timestamp sequence of the real-time action data with the ideal time sequence of the ideal path model by using a dynamic time warping algorithm, calculates an optimal time mapping path, and uses the optimal time mapping path to perform time axis alignment on the spatial coordinate sequence in the real-time action data and the ideal spatial coordinate sequence in the ideal path model, and outputs the aligned real-time action data and the aligned ideal path model to the parallel evaluation unit; The parallel evaluation unit receives the aligned real-time action data and the aligned ideal path model from the data alignment unit; the parallel evaluation unit comprises a spatial accuracy sub-module, a time efficiency sub-module, a path fluency sub-module and an energy economy sub-module; The spatial accuracy submodule calculates the Euclidean distance based on the spatial coordinate sequence in the aligned real-time action data and the ideal spatial coordinate sequence in the aligned ideal path model at each mapping point of the optimal time mapping path and takes the average to output a spatial accuracy score; the time efficiency submodule compares the ratio of the timestamp sequence of the aligned real-time action data and the ideal time sequence of the aligned ideal path model to output a time efficiency score; The path fluency submodule calculates the curvature by second-order difference of the spatial coordinate sequence in the aligned real-time action data and analyzes and calculates the distribution entropy value to output a path fluency score; the energy consumption economy submodule calculates the energy consumption integral of the motion acceleration corresponding to the aligned real-time action data to output an energy consumption economy score; the parallel evaluation unit synchronously outputs the spatial accuracy score, the time efficiency score, the path fluency score and the energy consumption economy score to the weighted fusion unit; The weighted fusion unit receives the spatial accuracy score, the time efficiency score, the path fluency score and the energy consumption economy score from the parallel evaluation unit; the weighted fusion unit linearly combines the spatial accuracy score, the time efficiency score, the path fluency score and the energy consumption economy score according to the predefined weight coefficients to output the comprehensive score.

5. The VR interaction based cognitive training path scoring system of claim 4, wherein, The weighted fusion unit is connected with a historical database; The historical database stores the historical comprehensive score and the historical dimension score of the user, and the historical dimension score includes the historical spatial accuracy score, the historical time efficiency score, the historical path fluency score and the historical energy consumption economy score; the weighted fusion unit reads the historical dimension score from the historical database and extracts the historical fluctuation characteristics of the historical spatial accuracy score, the historical time efficiency score, the historical path fluency score and the historical energy consumption economy score; The weighted fusion unit is built-in with a weight mapping table, and the weight mapping table stores the corresponding relationship between the historical score fluctuation characteristics and the weight coefficient adjustment strategy; the weighted fusion unit matches the extracted historical fluctuation characteristics with the weight mapping table to obtain the corresponding weight coefficient adjustment strategy; the weighted fusion unit dynamically adjusts the weight coefficient to be used for the next time of weighted fusion of the spatial accuracy score, the time efficiency score, the path fluency score and the energy consumption economy score according to the weight coefficient adjustment strategy; the weighted fusion unit caches the adjusted weight coefficient as a new predefined weight coefficient to the historical database.

6. The VR interaction based cognitive training path scoring system of claim 1, wherein, The real-time feedback module includes an adaptive controller, and the real-time feedback module includes an adaptive controller, and the adaptive controller is connected with a score cache area; The real-time feedback module stores the comprehensive score of the continuous multiple periods received from the multi-dimensional evaluation module to the score cache area; The adaptive controller obtains the comprehensive score of the continuous multiple periods from the score cache area and calculates the moving average change rate of the comprehensive score; The adaptive controller compares the calculated moving average change rate with preset difficulty level thresholds; The adaptive controller generates a difficulty adjustment instruction according to the comparison result, and sends the difficulty adjustment instruction to the path generation module to modify the path sampling step length parameter or the standard motion speed parameter in the preset task target.

7. The VR interaction based cognitive training path scoring system of claim 6, wherein, The adaptive controller adopts a preset model predictive control algorithm; The model predictive control algorithm has a built-in prediction model of system dynamics; The adaptive controller inputs the moving average change rate into the prediction model to rollingly optimize the system state in a future time window; The adaptive controller takes the rollingly optimized result as the output of the difficulty adjustment instruction; The output of the difficulty adjustment instruction is used to determine the specific adjustment value of the path sampling step length parameter or the standard motion speed parameter.

8. The VR interaction based cognitive training path scoring system of claim 1, wherein, The computing server internally deploys a data synchronization bus and a shared memory area; The data synchronization bus connects the data acquisition module, the path generation module, the multi-dimensional evaluation module, and the real-time feedback module to the shared memory area; The data acquisition module writes the preprocessed real-time action data into the shared memory area through the data synchronization bus, and attaches a uniform timestamp label to the written real-time action data; The path generation module and the multi-dimensional evaluation module query the shared memory area based on the uniform timestamp label through the data synchronization bus, and read the real-time action data with the same timestamp label.

9. The VR interaction based cognitive training path scoring system of claim 1, wherein, The wireless communication between the motion capture handle and the computing server adopts a time division multiple access protocol; The video transmission between the VR head-mounted display and the computing server adopts a frame synchronization mechanism; The computing server is internally provided with a data alignment unit; The data alignment unit receives the hand motion data transmitted through the time division multiple access protocol and the head pose data transmitted through the frame synchronization mechanism from the communication interface of the computing server; The data alignment unit analyzes the time slot allocation information of the time division multiple access protocol to obtain the wireless transmission delay of the hand motion data; the data alignment unit analyzes the vertical blanking interval signal of the frame synchronization mechanism to obtain the video frame transmission delay of the head pose data; The data alignment unit offsets and compensates the timestamps of the hand motion data and the head pose data respectively according to the wireless transmission delay and the video frame transmission delay, and outputs the hand motion data and the head pose data with timestamp calibrated to the data acquisition module.

10. The method of scoring a cognitive training path based on VR interaction, applied to the system of scoring a cognitive training path based on VR interaction according to any one of claims 1-9, characterized in that, Comprising: Step S1: Collecting user hand motion data and head pose data, and sending the hand motion data and the head pose data to a computing server; Step S2: Preprocessing the hand motion data and the head pose data from step S1 in the computing server, and outputting real-time action data including spatial coordinate sequences and timestamps; Step S3: receiving the real-time action data from step S2, processing in combination with a preset task target to generate an ideal path model including an ideal spatial coordinate sequence and an ideal time sequence; Step S4: receiving the real-time action data from step S2 and the ideal path model from step S3, performing parallel quantitative analysis on the real-time action data and the ideal path model in terms of spatial accuracy, time efficiency, path fluency and energy economy to generate multiple dimension scores; Step S5: weighting and fusing the multiple dimension scores from step S4 to obtain a comprehensive score; Step S6: generating a feedback control signal according to the comprehensive score from step S5, and driving a feedback device to perform visual and audio feedback corresponding to the feedback control signal; Step S7: generating a difficulty parameter adjustment instruction according to the continuous change of the comprehensive score from step S5, and feeding back the difficulty parameter adjustment instruction to step S3 to dynamically adjust the preset task target.

Images