Personnel cognitive ability assessment method based on virtual reality scene, edge computing device and medium

Abstract

The invention provides a person cognitive ability assessment method based on a virtual reality scene, an edge computing device and a medium, and relates to the technical field of cognitive assessment. After a to-be-evaluated user wears a virtual reality VR device, the pose change of the head of the user is obtained in real time, and the picture of the evaluation task displayed in the VR device is adjusted based on the pose change, so that the switching of the picture is synchronized with the pose change of the head. And obtaining multi-modal physiological data and an execution result of the evaluation task in the process of executing the evaluation task by the user, and determining a cognitive ability evaluation result of the user based on the multi-modal physiological data and the execution result. Manual participation is not needed, and the evaluation efficiency and accuracy are improved; and adjusting the picture according to the pose change so as to synchronize the picture switching and the head pose change, namely the VR equipment provided by the invention has a picture self-adaptive adjustment capability aiming at the user, accords with the real motion condition of the head, and reduces the shaking of the picture, so that the discomfort of the user in the evaluation process is relieved, and the evaluation experience is improved.

Description

Technical Field

[0001] This disclosure relates to the field of cognitive assessment technology, and in particular to a method for assessing human cognitive abilities based on virtual reality scenarios, an edge computing device, and a medium. Background Technology

[0002] As older adults age, their cognitive abilities gradually decline. Therefore, early cognitive assessment of older adults can help detect signs of cognitive decline in a timely manner and prevent the development of dementia.

[0003] Currently, the main assessment method is for elderly people to go to the hospital, complete a cognitive ability test provided by the hospital, and then have a doctor assess the health of their cognitive abilities based on the test results.

[0004] However, cognitive ability tests provided by hospitals typically need to cater to the assessment needs of various population groups. Even for elderly people with normal cognitive abilities, their ability to process information and their on-the-spot reaction time are weaker than those of middle-aged and younger people. In particular, when some tests involve the use of VR devices, they lack the ability to adaptively adjust the image. Summary of the Invention

[0005] One of the technical problems this disclosure aims to solve is: how to improve the efficiency and accuracy of cognitive ability assessment.

[0006] In a first aspect, to address the aforementioned technical problems, embodiments of this disclosure provide a method for assessing human cognitive abilities based on virtual reality scenarios, the method comprising: After the user to be evaluated wears the virtual reality (VR) device, the changes in the user's head pose are acquired in real time. Based on the pose change, the screen of the evaluation task displayed in the VR device is adjusted so that the screen switching is synchronized with the pose change; Acquire multimodal physiological data of the user during the execution of the assessment task, as well as the execution results of the assessment task; Based on the multimodal physiological data and the execution results, the cognitive ability assessment results of the user are determined.

[0007] In some embodiments, when it is determined that the pose change is greater than the change threshold, a transition screen is generated based on the first screen of the evaluation task and the user's head pose before the pose change. The first screen is determined based on the user's head pose before the pose change. The transition screen is displayed to adjust the screen of the evaluation task displayed in the VR device.

[0008] In some embodiments, the pose change is characterized by the angular velocity and movement velocity of the user's head; when the pose change is determined to be greater than a change threshold, a transitional frame is generated based on the first frame of the evaluation task and the user's head pose before the pose change, including: When it is determined that the angular velocity is greater than a first threshold and / or the movement speed is greater than a second threshold, the first pose of the virtual camera of the VR device corresponding to the first screen is obtained; Interpolation is performed between the first pose and the second pose of the virtual camera to obtain an intermediate position, wherein the second pose is determined based on the user's head pose after the pose change; Based on the intermediate position, a transitional screen from the first screen to the second screen is obtained, wherein the second screen is determined based on the second pose.

[0009] In some embodiments, the first pose and the second pose are subjected to quaternion spherical interpolation or smooth interpolation with damping parameters to obtain the intermediate position.

[0010] In some embodiments, the user's body posture is obtained, wherein the body posture is a sitting posture or a standing posture; Based on the user's body posture, the display position of the assessment task in the VR device is adjusted.

[0011] In some embodiments, the execution results include: average reaction time and pass rate for performing the assessment task; determining the user's cognitive ability assessment result based on the multimodal physiological data and the execution results includes: Based on the multimodal physiological data, the user's fatigue level is obtained; Based on the average reaction time, the pass rate, and the degree of fatigue, the cognitive ability assessment results of the user are obtained; The user's cognitive ability assessment result is directly proportional to the pass rate and inversely proportional to the average reaction time and the degree of fatigue.

[0012] In some embodiments, the cognitive ability assessment result characterizes the health level of the user's cognitive ability; the method further includes: If the health status is outside the target range, the difficulty of the assessment task is adjusted based on the cognitive ability assessment result, and the cognitive ability assessment result is updated based on the multimodal physiological data and corresponding execution results of the user during the execution of the assessment task with adjusted difficulty.

[0013] In some embodiments, if the health level is greater than the upper limit of the target range, the difficulty of the assessment task is increased; If the health status is below the lower limit of the target range, the difficulty of the assessment task is reduced.

[0014] In a second aspect, embodiments of this disclosure provide an edge computing device, including: a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the human cognitive ability assessment method based on a virtual reality scene as described in any of the first aspects above.

[0015] Thirdly, embodiments of this disclosure provide a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the human cognitive ability assessment method based on a virtual reality scene as described in any of the first aspects above.

[0016] Fourthly, embodiments of this disclosure provide an assessment system, comprising: a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the human cognitive ability assessment method based on a virtual reality scene as described in any of the first aspects above.

[0017] As can be seen from the above technical solution, this disclosure provides a method, edge computing device, and medium for assessing human cognitive abilities based on virtual reality scenarios. This method involves acquiring the user's head pose changes in real time after the user wears the virtual reality device, and adjusting the screen displaying the assessment task in the VR device based on these pose changes, ensuring that screen switching is synchronized with the user's head pose changes. It also acquires multimodal physiological data and the execution results of the assessment task during the user's execution, and determines the user's cognitive ability assessment result based on the multimodal physiological data and execution results. The assessment method in this disclosure requires no human intervention, improving the efficiency and accuracy of the assessment; and it can adjust the screen according to pose changes, ensuring that screen switching is synchronized with the user's head pose changes. In other words, the VR device provided in this embodiment has the ability to adaptively adjust the screen for the user, conforming to the user's actual head movements and reducing the possibility of violent screen shaking, thereby alleviating user discomfort during the assessment process and improving the assessment experience. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a flowchart illustrating a method for assessing human cognitive abilities based on a virtual reality scenario, as disclosed in an embodiment of this disclosure. Figure 2 This is a flowchart illustrating another method for assessing human cognitive abilities based on a virtual reality scenario, as disclosed in this embodiment. Figure 3 This is a schematic diagram of the structure of an evaluation system disclosed in an embodiment of this disclosure. Detailed Implementation

[0020] The embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings and examples. The detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of this disclosure by way of example, but should not be used to limit the scope of this disclosure. This disclosure can be implemented in many different forms and is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

[0021] These embodiments are provided in this disclosure to make the disclosure thorough and complete, and to fully express the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specifically stated, the relative arrangement of components and steps, the composition of materials, numerical expressions and values ​​set forth in these embodiments should be interpreted as exemplary only and not as limiting.

[0022] All terms used in this disclosure have the same meaning as understood by one of ordinary skill in the art to which this disclosure pertains, unless otherwise specifically defined. It should also be understood that terms defined in general dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant art, and not as idealized or highly formalized, unless expressly defined herein.

[0023] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0024] The physiological data involved in this disclosure are various measurable data signals in the human body, including but not limited to electrocardiogram (ECG) signals, skin temperature (SKT) signals, photoplethysmogram (PPG) signals, electrodermal activity (EDA) signals, heart rate (HR) signals, electromyogram (EMG) signals, electroencephalogram (EEG) signals, and peripheral capillary oxygen saturation (SPO2) signals.

[0025] As older adults age, their cognitive abilities gradually decline. Therefore, early cognitive assessment of older adults can help detect signs of cognitive decline in a timely manner and prevent the development of dementia.

[0026] Currently, the main method for assessing cognitive abilities involves elderly individuals visiting hospitals and completing cognitive tests provided by the hospitals. Doctors then manually assess the cognitive health of the elderly based on the test results. However, this method is relatively inefficient.

[0027] In view of this, embodiments of this disclosure provide a method for assessing human cognitive abilities based on virtual reality scenarios. This method can acquire real-time head pose changes of the user wearing a virtual reality (VR) device, and adjust the screen displaying the assessment task in the VR device based on these pose changes to synchronize screen transitions with the user's head pose changes. It then acquires multimodal physiological data and the results of the assessment task execution, and finally determines the user's cognitive ability assessment result based on the multimodal physiological data and the execution results. Therefore, on the one hand, it eliminates the need for manual cognitive ability assessment, thus improving the efficiency of cognitive ability assessment; on the other hand, acquiring cognitive ability assessment results based on multimodal physiological data improves the accuracy of the results; furthermore, adjusting the screen according to pose changes to synchronize screen transitions with the user's head pose changes allows the screen changes to conform to the head's movement patterns, reducing visual abruptness and thus alleviating user discomfort during the assessment process, improving the assessment experience.

[0028] The method for assessing human cognitive abilities based on virtual reality scenarios, as provided in the embodiments of this disclosure, will now be described in detail with reference to the accompanying drawings. Figure 1This disclosure provides a method for assessing human cognitive abilities based on a virtual reality scenario. The method is applied to a VR device and executed by a processor within the VR device. Optionally, the VR device can be an all-in-one (AIO) device, a PC-tethered virtual reality (PCVR) device, or a SteamVR device. For details, see [link to relevant documentation]. Figure 1 The method includes: Step 101: After the user to be evaluated wears the VR device, the position and pose changes of the user's head are acquired in real time.

[0029] The VR device's processor acquires real-time changes in the user's head pose after the user wears the VR device. These changes can be represented by the user's head angular velocity and movement speed.

[0030] Optionally, the VR device in this embodiment may include an inertial measurement unit (IMU). The VR device can acquire the angular velocity and movement velocity of the user's head through the IMU. For example, the IMU includes a gyroscope and an accelerometer. The VR device can acquire the angular velocity of the user's head through the gyroscope and the movement velocity of the user's head through the accelerometer, thereby obtaining the head pose change.

[0031] Step 102: Based on the head pose change, adjust the screen of the assessment task displayed in the VR device so that the screen switching is synchronized with the pose change.

[0032] The VR device's processor adjusts the displayed assessment task image based on changes in head posture, so that the image switching is synchronized with the posture change.

[0033] Optionally, the processor of the VR device can obtain the display screen of the evaluation task displayed in the VR device based on the head pose change of the external device, adjust the display screen, and control the VR device to display the display screen.

[0034] When a user's head shakes violently due to some reason (such as accidental operation), the processor can determine that the change in head posture exceeds a threshold. At this time, the VR device's processor can generate and display a transitional frame based on the previous frame to adjust the assessment task displayed to the user in the VR device. This ensures that the frame switching (transition) is synchronized with the user's head posture change, avoiding drastic changes in the assessment task's display that could cause dizziness. The number of these transitional frames can be one frame or multiple frames.

[0035] In this embodiment of the disclosure, the transition screen is used to smoothly transition the evaluation task screen displayed on the VR device from the first screen to the second screen. The first screen is determined based on the user's head pose before the pose change (i.e., the screen displayed in the previous evaluation task), and the second screen is determined based on the user's head pose after the pose change.

[0036] Step 103: Obtain multimodal physiological data of the user during the execution of the assessment task, as well as the execution results of the assessment task.

[0037] The processor of a VR device can acquire multimodal physiological data of the user during the performance of an assessment task, as well as the results of the task. This multimodal physiological data can include at least two of the following: electromyography (EMG) data, heart rate, skin conductivity, and eye movement data. By extracting features from the EMG, heart rate, skin conductivity, and eye movement data, the user's level of fatigue during the assessment task can be determined.

[0038] Optionally, VR devices may include physiological sensors and eye trackers. Physiological sensors include electromyography (EMG) sensors, heart rate sensors, and skin resistance sensors. The EMG sensor is used to collect the user's electromyographic data; the heart rate sensor is used to collect the user's heart rate; and the skin resistance sensor is used to collect the user's skin conductivity. The eye tracker is used to collect the user's eye movement data.

[0039] The results of the assessment task, i.e., behavioral data, can include at least: average reaction time and pass rate. For example, this could include: average reaction time, pass rate, and reaction time stability.

[0040] Optionally, the VR device may include a timer. Each assessment task includes multiple sub-tasks. The type and number of sub-tasks can be set according to actual conditions; for example, a sub-task could be selecting a corresponding ball based on prompts, grabbing an object, etc. For each sub-task, the VR device's processor can control the timer to start timing from when the user begins performing the sub-task and end timing after the user completes the sub-task, and determine the timing duration as the user's reaction time for performing the sub-task. Subsequently, the VR device's processor can determine the average reaction time of the user when performing the assessment task by averaging the reaction times of multiple sub-tasks.

[0041] The VR device's processor can obtain the number of sub-tasks successfully completed by the user and determine the pass rate of the user when performing the assessment task by dividing the number by the total number of sub-tasks. The VR device can obtain the standard deviation of the reaction time of multiple sub-tasks and obtain the stability of the user's reaction time when performing the assessment task based on the standard deviation. This stability of reaction time is inversely proportional to the standard deviation.

[0042] Step 104: Based on multimodal physiological data and execution results, determine the user's cognitive ability assessment results.

[0043] The processor in a VR device can determine the user's cognitive ability assessment results based on multimodal physiological data and execution results.

[0044] Since unimodal physiological data is easily affected by environmental noise or individual baseline differences, while multimodal physiological data can eliminate the noise interference of unimodal physiological data through the complementarity between physiological data of different modalities, multimodal physiological data and execution results can be used to determine the user's cognitive ability assessment results, which can improve the accuracy of cognitive ability assessment results.

[0045] In summary, the cognitive ability assessment method based on virtual reality scenarios provided in this disclosure can acquire the user's head pose changes in real time after the user wears a virtual reality device. Based on these pose changes, the method adjusts the screen displaying the assessment task in the VR device to synchronize screen transitions with the user's head pose changes. It then acquires multimodal physiological data and the results of the assessment task execution, and finally determines the user's cognitive ability assessment result based on the multimodal physiological data and the execution results. Therefore, on the one hand, it eliminates the need for manual cognitive ability assessment, thus improving the efficiency of cognitive ability assessment; on the other hand, acquiring cognitive ability assessment results based on multimodal physiological data improves the accuracy of the assessment results; and furthermore, the method can adjust the screen according to pose changes to synchronize screen transitions with the user's head pose changes. In other words, the VR device provided in this disclosure has adaptive screen adjustment capabilities for the user, allowing screen changes to conform to the user's head movement patterns, reducing visual abruptness, and thus alleviating user discomfort during the assessment process and improving the assessment experience.

[0046] Figure 2 This disclosure provides another method for assessing human cognitive abilities based on virtual reality scenarios. This method can be applied to VR devices. For details, see... Figure 2 The method may include: Step 201: After the user to be evaluated wears the VR device, the user's head pose is acquired in real time.

[0047] The processor of the VR device can acquire the user's head pose in real time after the user is wearing the VR device. This pose change can be represented by the angular velocity and movement velocity of the user's head, based on the inertial measurement unit (IMU).

[0048] Step 202: Determine whether the pose change is greater than the change threshold.

[0049] The VR device's processor can determine whether the pose change exceeds a threshold. If the processor determines the pose change exceeds the threshold, it can identify that the user's head movement is large / abnormal, and then proceed to step 203. If the processor determines the pose change is less than or equal to the threshold, it can identify that the user's head movement is normal, and then proceed to step 201 to obtain the user's head pose.

[0050] In this embodiment of the disclosure, the processor of the VR device can determine that the pose change is greater than a change threshold when the angular velocity and movement speed representing the user's head pose change meet preset conditions.

[0051] The preset condition is at least one of the following: the angular velocity of the user's head is greater than a first threshold; and the movement speed is greater than a second threshold.

[0052] In other words, when the VR device's processor determines that the user's head angular velocity is greater than a first threshold and / or the movement speed is greater than a second threshold, it can determine that the pose change is greater than a change threshold. When the VR device's processor determines that the user's head angular velocity is less than or equal to the first threshold and the movement speed is less than or equal to the second threshold, it can determine that the pose change is less than a change threshold. The first and second thresholds are pre-stored by the VR device's processor; for example, the first threshold could be 20 degrees per second (° / s), and the second threshold could be 0.3 meters per second (m / s).

[0053] Step 203: When the change in pose is determined to be greater than the change threshold, a transitional screen is generated based on the first screen of the evaluation task and the user's head pose before the change in pose.

[0054] When the VR device's processor determines that the pose change exceeds a threshold, it can generate a transitional frame based on the first frame of the evaluation task and the user's head pose before the pose change. The first frame is determined based on the user's head pose before the pose change. For example, the VR device's processor can determine the first pose of the VR device's virtual camera based on the user's head pose before the pose change, and then obtain the first frame based on that first pose.

[0055] The following is an example illustrating how the processor of a VR device generates transitional frames based on the first frame of the evaluation task and the user's head pose before the pose change: Step A1: Obtain the first pose of the virtual camera of the VR device corresponding to the first screen.

[0056] The processor of a VR device can obtain the first pose of the virtual camera of the VR device corresponding to the first screen, so as to represent the rotation state / rotation posture of the virtual camera relative to the fixed coordinate system when the VR device presents the first screen. This can be represented by Euler angles or quaternions. The fixed coordinate system can be the world coordinate system or a coordinate system defined by the actual situation after the object to be evaluated wears the VR device.

[0057] In this embodiment, a "visual stabilization mechanism" is introduced to generate transitional images when the user's head pose changes significantly. During this process, the orientation of the user's head when wearing the VR device / the user's field of vision direction is used as the field of vision orientation of the virtual camera.

[0058] The VR device's processor can record several frames of the displayed image, along with the user's head pose corresponding to each frame. Based on the user's head pose at two adjacent moments, it determines the corresponding pose change. When the VR device's processor determines that the pose change exceeds a threshold, it acquires the user's head pose from the previous moment (i.e., before the pose change) and determines the virtual camera's first pose (i.e., the field of view orientation) based on this head pose. This head pose includes both position and orientation, with the orientation represented using quaternions.

[0059] The processor of a VR device can perform matrix transformations on the quaternion based on coordinate transformation theory to calculate the rotation and translation matrices corresponding to the virtual camera, and then obtain the product of these rotation and translation matrices to get the first pose of the virtual camera. q t-1 The translation matrix is ​​determined based on the head position when the first screen is displayed.

[0060] Optionally, both the rotation matrix and the translation matrix can be 4×4 matrices.

[0061] Step A2: Interpolate the first pose and the second pose of the virtual camera to obtain the intermediate position.

[0062] The processor of a VR device can interpolate the first and second poses of a virtual camera to obtain an intermediate position. This second pose is determined based on the user's head pose after the pose change and can be represented using quaternions.

[0063] In this embodiment of the disclosure, the first pose and the second pose satisfy the following relationship: (1) In formula (1), θ For rotation angle, It is the instantaneous angular velocity. For rotation change, w is the w component of the relative rotation quaternion calculated from the pose of the virtual camera in the current frame (second pose) and the pose of the virtual camera in the previous frame (first pose), which is the real part of the quaternion. The first posture q t-1 The quaternion inverse. Δt is the actual sampling time between the first pose and the second pose.

[0064] In this embodiment of the disclosure, the processor of the VR device can perform quaternion spherical interpolation or smooth interpolation with damping parameters on the first and second poses of the virtual camera to obtain the intermediate position.

[0065] Step A3: Based on the intermediate position, generate a transition screen from the first screen to the second screen.

[0066] The VR device's processor can generate a transitional frame from the first frame to the second frame based on an intermediate position. The second frame is determined based on the second pose of the virtual camera. The intermediate position is the location of the virtual camera.

[0067] The processor of a VR device can perform rendering based on the central position, with the virtual camera as the origin of the three-dimensional scene's viewpoint. This involves calculating the distance, angle, and occlusion relationship of each test task's objects and background objects in the three-dimensional scene relative to the virtual camera, and then projecting the three-dimensional shapes of each object onto the two-dimensional imaging plane of the virtual camera to obtain a transitional image.

[0068] Step 204: Display a transition screen to adjust the screen of the assessment task displayed in the VR device so that the screen switching is synchronized with the head pose change.

[0069] The processor in a VR device can control the display of transitional frames to adjust the visuals of the assessment task displayed on the VR device, ensuring that the frame transitions are synchronized with changes in head posture. Specifically, the processor can control the VR device to display a transitional frame after the first frame, and then a second frame after the transitional frame, to adjust the visuals of the assessment task displayed on the VR device and achieve a smooth transition from the first frame to the second frame.

[0070] Adding transitional frames during the display process ensures that even if the user's head rotates rapidly (i.e., significant changes in head posture), the frame switching rate remains relatively low (i.e., a transitional frame is displayed between the first and second frames), avoiding the visual stimulation of directly displaying the second frame after the first. This maintains a stable virtual camera perspective, aligning with the visual processing characteristics of users (such as the elderly), reducing discomfort (such as dizziness) during the assessment process. This enhances the cognitive ability assessment experience, increasing the duration and engagement of user tasks.

[0071] Step 205: Obtain the user's body posture.

[0072] The processor in a VR device can acquire the user's body posture, which can be either sitting or standing.

[0073] In some optional implementations, the VR device also includes a distance sensor. Since the VR device is worn on the user's head, the VR device's processor can control the distance sensor to obtain the height of the VR device above the user's ground. The VR device's processor can determine the user's posture as standing when the height is greater than a height threshold. The VR device's processor can determine the user's posture as sitting when the height is less than or equal to the height threshold. The height threshold is pre-stored by the VR device's processor.

[0074] Step 206: Adjust the display position of the assessment task in the VR device based on the user's body posture.

[0075] The VR device's processor can adjust the display position of the assessment task based on the user's body posture. The VR device's display screen can show the user interface (UI). This UI moves with the user's movement, and the assessment task's screen can be displayed within the UI; that is, the assessment task's screen is the UI's content. The VR device can adjust the UI's position based on the user's body posture to adjust the display position of the assessment task within the VR device.

[0076] Specifically, the processor of the VR device can obtain the position of the user's gaze point on the display screen when looking straight ahead based on the body posture, and determine the display area of ​​the UI based on the position of the gaze point on the display screen, and then adjust the UI to the display area to adjust the display position of the assessment task in the VR device.

[0077] The VR device's processor can adjust the size of the UI displaying the assessment task so that the content displayed in the UI (i.e., all task objects) is distributed within a preset range of the user's vertical field of view. This preset range can be pre-stored by the VR device's processor, for example, it can be 20°-60°.

[0078] This allows the UI (i.e. the display screen of the assessment task) to be located in the user's comfortable viewing area, meaning that the user can view the complete assessment task display screen without looking up or down.

[0079] Optionally, the user performs sub-tasks of the assessment task that are target objects in the selected display screen. The VR device's processor determines the user's selection position on the display screen; if the selection is within the target area, the processor determines that the user has selected the target object. This target area refers to a circular region with the target object as its center and a preset radius. This preset length is pre-stored by the VR device's processor; for example, the preset length is the radius of the target object. times.

[0080] In other words, when the processor of a VR device obtains the results of the user's evaluation task, it can enlarge the size of the effective selected area (i.e., the target area) to a preset multiple of the size of the target object, so as to improve the error tolerance of the user when performing the evaluation task.

[0081] Step 207: Obtain multimodal physiological data of the user during the execution of the assessment task, as well as the execution results of the assessment task.

[0082] The processor of a VR device can acquire multimodal physiological data during the user's performance of an assessment task, as well as the results of that task. The multimodal physiological data can include at least two of the following: electromyography (EMG) data, heart rate, skin conductivity, and eye movement data. For example, the multimodal physiological data may include EMG data, heart rate, skin conductivity, and eye movement data. Each of these data can at least be used to reflect the user's level of fatigue.

[0083] The results of the assessment task are behavioral data, which should include at least the average reaction time and pass rate. For example, they may include the average reaction time, pass rate, and reaction time stability.

[0084] The implementation method of this step is similar to that of step 103, and will not be repeated here.

[0085] Step 208: Based on multimodal physiological data, obtain the user's fatigue level.

[0086] The processor of a VR device can determine the user's fatigue level based on multimodal physiological data. It can also obtain fixation duration based on eye-tracking data. Since greater fatigue is associated with a larger root mean square amplitude of electromyography (EMG) signals, lower skin resistance, higher heart rate, and shorter fixation duration, and because skin conductivity and skin resistance are inversely related, the VR device's processor can determine the user's fatigue level based on EMG data, skin conductivity, heart rate, and eye-tracking data.

[0087] The processor of a VR device can acquire the root mean square (RMS) of electromyography (EMG) signals, heart rate, skin resistance, and fixation duration, and then weight and sum these parameters to obtain the user's fatigue level.

[0088] Optionally, the VR device's processor can determine the user's fatigue level based on multimodal physiological data and the results of the assessment task. Specifically, the VR device's processor can obtain the number of failures in subtasks during the user's assessment task, as well as the reaction time stability. The user's fatigue level is directly proportional to the number of failures and inversely proportional to the reaction time stability. Based on this, the VR device's processor can obtain the root mean square (RMS) of electromyography (EMG) signals, heart rate, skin resistance, fixation duration, number of failures, and the reciprocal of reaction time stability, and then perform a weighted sum of these parameters to determine the user's fatigue level. Step 209: Based on average reaction time, pass rate, and fatigue level, obtain the user's cognitive ability assessment results.

[0089] The VR device's processor can obtain a user's cognitive ability assessment result based on average reaction time, pass rate, and fatigue level. The user's cognitive ability assessment result is directly proportional to the pass rate and inversely proportional to the average reaction time and fatigue level.

[0090] Optional, cognitive ability assessment results Score The pass rate, average reaction time, and fatigue level can satisfy the following formula (2): (2) In formula (2), W1 As the weight of the pass rate, W2 The weights for average reaction time, and W3 This represents the weighting of fatigue levels. Among them, W1 , W2 and W3 All of these are pre-stored by VR devices.

[0091] VR devices include: controller devices. Optionally, the processor of the VR device can also acquire the number of times the user's head rotates during the performance of the assessment task, as well as the number of times the user's hands rotate through the controller devices.

[0092] If the VR device's processor determines that the head rotation angle is greater than a first angle threshold, it can increase the head rotation count by one. If the VR device's processor determines that the hand rotation angle is greater than a second angle threshold, it can increase the hand rotation count by one. Both the first and second angle thresholds are pre-stored by the VR device's processor. The VR device's processor can determine the user's sensitivity based on the number of head and hand rotations. This sensitivity is proportional to the number of head and hand rotations. In this way, user sensitivity can be obtained during the cognitive ability assessment process, leading to a more comprehensive understanding of the user's physical function and providing a basis for clinical diagnosis and rehabilitation training.

[0093] Step 210: If the health status is outside the target range, adjust the difficulty of the assessment task based on the cognitive ability assessment results, and update the cognitive ability assessment results based on the user's multimodal physiological data during the execution of the assessment task with adjusted difficulty.

[0094] If the VR device's processor determines that the user's cognitive health level is outside the target range, it can adjust the difficulty of the assessment task based on the cognitive ability assessment results and update the results based on the user's multimodal physiological data during the execution of the adjusted task. The cognitive ability assessment results characterize the user's cognitive health level. Specifically, the VR device's processor can increase the difficulty of the assessment task if the user's health level is above the upper limit of the target range; conversely, it can decrease the difficulty if the user's health level is below the lower limit of the target range. The target range is pre-stored by the VR device's processor.

[0095] In this way, the difficulty of the assessment task can be dynamically adjusted so that the adjusted assessment task matches the user's cognitive ability, thereby improving the effectiveness of training the user's cognitive ability.

[0096] In this embodiment of the disclosure, if the updated cognitive ability assessment result is still less than the lower limit of the target range after adjusting the difficulty of the assessment task multiple times (more than a preset number of times), the processor of the VR device can increase the target completion time of the assessment task to prevent user frustration and improve the user's experience in performing the assessment task.

[0097] The evaluation system of this disclosure includes at least one of the following assessment tasks: multiple object tracking (MOT) task, spatial memory span task, stroopcolor word test (SCWT) task, and three-dimensional Schulte grid task. For example, the assessment tasks in the evaluation system include: MOT task, spatial memory span task, SCWT task, and three-dimensional Schulte grid task.

[0098] The MOT task involves randomly generating N movable spheres, selecting T as target spheres, displaying flashing prompts, and generating movement paths for the N spheres. The VR device's processor pre-stores a multi-frequency combined sine wave model, which is used to generate the movement paths to ensure smoothness and controllability. NT movable spheres act as interference spheres, creating a dynamic attentional load. Each target sphere and each interference sphere moves according to its respective path, with the target spheres and interference spheres having different colors. After the movable spheres have completed their movement, the user selects a target sphere using a controller ray. Difficulty parameters include the number of targets, movement speed, number of interferences, and path complexity. The difficulty of this assessment task is positively correlated with the number of targets, movement speed, number of interferences, and path complexity.

[0099] The spatial memory span task involves randomly generating K points, which are then sequentially illuminated. The user selects the corresponding point based on the illumination order. The VR device's processor uses a randomized scattering algorithm to generate the positions of each point, ensuring uniqueness in each instance. The illumination time of each point and the range of K are pre-stored by the VR device's processor, such as K ∈ [3,9]. Difficulty parameters can include the number of points, illumination duration, and minimum distance between point positions. The difficulty of this assessment task is positively correlated with the number of points and negatively correlated with illumination duration and minimum distance between point positions.

[0100] The SCWT task involves randomly presenting text content (such as "red" and "blue"), where the text color may or may not match the semantic meaning. Users must select a target based solely on the text color, ignoring the semantics of the text itself. For example, if the text color matches the semantic meaning, it might be "red" and displayed as red; if the text color doesn't match the semantic meaning, it might be "red" and displayed as blue. When a user sees a red "blue" text, they must select "red" instead of "blue." The difficulty of this assessment task is positively correlated with the frequency of text switching, the number of texts, the degree of color difference, the presentation duration, and the number of distractors.

[0101] The 3D Schulte Grid task involves randomly generating multiple numbers, each in a different position, and the user must select the numbers in ascending order. The VR device's processor has a pre-stored 3D scatter algorithm that can be used to generate the positions of the numbers, thus avoiding the learning effect. The learning effect refers to the phenomenon where, after repeatedly performing the same type of task or encountering the same type of stimuli, a user's performance on subsequent test tasks changes due to experience accumulation and increased familiarity, such as a shorter reaction time and higher accuracy. The difficulty of this assessment task is positively correlated with both the density of the numbers and the radius of the range of their positions.

[0102] When configuring various evaluation tasks, the processor of a VR device can include the type of evaluation task, the number of interfering objects, the movement speed of the target and the interfering objects, and the number of times the evaluation task is executed.

[0103] This disclosure provides a method for automatically acquiring cognitive assessment results of users using a processor employed in a VR device. This improves the efficiency of acquiring cognitive assessment results, and acquiring these results based on multimodal data improves their accuracy. Furthermore, combining multiple assessment tasks and adaptively adjusting the difficulty of these tasks enhances the ecological validity, interactivity, and training effectiveness of the cognitive assessment. It also allows for personalized, systematic, and highly precise training of users' (e.g., the elderly) cognitive abilities.

[0104] In summary, the present disclosure provides a method for assessing human cognitive abilities based on virtual reality scenarios. This method can acquire real-time head pose changes of the user after the user wears a virtual reality device, and adjust the screen displaying the assessment task in the VR device based on these pose changes to synchronize screen transitions with the user's head pose changes. It then acquires multimodal physiological data and the results of the assessment task execution, and finally determines the user's cognitive ability assessment result based on the multimodal physiological data and the execution results. Therefore, on the one hand, it eliminates the need for manual cognitive ability assessment, thus improving the efficiency of cognitive ability assessment; on the other hand, acquiring cognitive ability assessment results based on multimodal physiological data improves the accuracy of the assessment results; furthermore, it can adjust the screen according to pose changes to synchronize screen transitions with the user's head pose changes. In other words, the VR device provided in this disclosure has adaptive screen adjustment capabilities for the user, allowing screen changes to conform to the user's head movement patterns, reducing visual abruptness, and thus alleviating user discomfort during the assessment process and improving the assessment experience.

[0105] Figure 3 An evaluation system is provided for a disclosed embodiment. See also: Figure 3The evaluation system can be integrated into VR devices or installed on electronic devices with data processing capabilities (such as mobile phones, tablets, laptops, personal computers, servers, etc.). The evaluation system includes at least the following: The data acquisition module 10 is used to acquire the changes in the user's head posture in real time after the user to be evaluated wears the virtual reality (VR) device.

[0106] The screen adjustment module 20 adjusts the screen of the evaluation task displayed in the VR device based on the change of pose, so that the screen switching is synchronized with the change of pose.

[0107] The data acquisition module 10 is also used to acquire multimodal physiological data during the user's performance of the assessment task, as well as the results of the assessment task.

[0108] The assessment and adjustment module 30 is used to determine the user's cognitive ability assessment results based on multimodal physiological data and execution results.

[0109] The data acquisition module 10 is also used to acquire and update the difficulty of the assessment task. This difficulty is represented by the movement speed of the targets in the task, the number of targets, and the completion time of the task.

[0110] The assessment and adjustment module 30 is also used to adaptively adjust the difficulty of the assessment task based on the user's cognitive ability assessment results, and to update the cognitive ability assessment results based on the user's multimodal physiological data during the execution of the assessment task with adjusted difficulty. The data acquisition module 10, the screen adjustment module 20, and the evaluation and adjustment module 30 can achieve instruction scheduling, task routing, and data synchronization through the system's controller, and interact with each other using a unified data structure. Task routing refers to the process of allocating event tasks (such as user requests, business processes, and data processing tasks) to corresponding processing modules according to preset rules or real-time conditions.

[0111] Optionally, the evaluation system may also include a front-end interaction module. This module responds to the user's actions, such as initiating or selecting a task, to launch the assessment task and load its configuration. The front-end interaction module then displays stimuli based on the task configuration, allowing the user to perform the assessment task accordingly. The module also receives the end signal of the assessment task and controls the data acquisition module to update the collected data. Here, displayed stimuli refer to visual signals conveyed to the user through specific visual design and technical presentation, thereby eliciting corresponding perceptual, emotional, or behavioral responses from the user.

[0112] Optionally, the evaluation system may also include a data processing module. This module is used to preprocess the multimodal physiological data and generate data files based on the preprocessed multimodal physiological data and the execution results of the assessment task (i.e., behavioral data). Preprocessing includes data cleaning and timestamp alignment. Data cleaning includes noise reduction and data completion. Optionally, the data file format can be comma-separated values ​​(CSV) or JavaScript object notation (JSON).

[0113] Optionally, the assessment system may also include a data storage module. This data storage module is used to store the assessment task performed by the user, records of the difficulty changes of the assessment task, the multimodal physiological data of the user in performing the assessment task and the corresponding execution results, as well as the cognitive ability assessment results corresponding to the performed assessment task.

[0114] Each assessment task can be performed multiple times, and the corresponding cognitive ability assessment results can be obtained after each performance. These results can be used for clinical evaluation, research analysis, or long-term cognitive tracking.

[0115] The evaluation system provided in this embodiment includes a processor and a memory. The data acquisition module 10, the image adjustment module 20, and the evaluation and adjustment module 30 are all deployed in the processor. The memory stores programs or instructions that can run on the processor. When executed by the processor, these programs or instructions implement the human cognitive ability evaluation method based on a virtual reality scene provided in the above method embodiments. For example, implementing... Figure 1 or Figure 2 The method shown.

[0116] This disclosure also provides an edge computing device, comprising a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the human cognitive ability assessment method based on a virtual reality scene provided in the above method embodiments. For example, implementing... Figure 1 or Figure 2 The method shown.

[0117] This disclosure also provides a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the human cognitive ability assessment method based on a virtual reality scene provided in the above-described method embodiments. For example, implementing... Figure 1 or Figure 2 The method shown.

[0118] This disclosure also provides a computer program product that, when executed by a processor of a vehicle or cloud server, implements the human cognitive ability assessment method based on a virtual reality scene provided in the above-described method embodiments. For example, it implements the method described above. Figure 1 or Figure 2 The method shown.

[0119] The embodiments of this disclosure have now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

[0120] While specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of this disclosure. In particular, as long as there is no structural conflict, the technical features mentioned in the various embodiments can be combined in any manner.

Claims

1. A method for assessing human cognitive abilities based on virtual reality scenarios, characterized in that, The method includes: After the user to be evaluated wears the virtual reality (VR) device, the changes in the user's head pose are acquired in real time. Based on the pose change, the screen of the evaluation task displayed in the VR device is adjusted so that the screen switching is synchronized with the pose change; Acquire multimodal physiological data of the user during the execution of the assessment task, as well as the execution results of the assessment task; Based on the multimodal physiological data and the execution results, the cognitive ability assessment results of the user are determined.

2. The method according to claim 1, characterized in that, The adjustment of the evaluation task screen displayed in the VR device based on the pose change includes: When it is determined that the pose change is greater than the change threshold, a transition screen is generated based on the first screen of the evaluation task and the head pose of the user before the pose change. The first screen is determined based on the head pose of the user before the pose change. The transition screen is displayed to adjust the screen of the evaluation task displayed in the VR device.

3. The method according to claim 2, characterized in that, The pose change is characterized by the angular velocity and movement speed of the user's head; When the change in pose is determined to be greater than a threshold, a transitional frame is generated based on the first frame of the assessment task and the head pose before the change in user pose, including: When it is determined that the angular velocity is greater than a first threshold and / or the movement speed is greater than a second threshold, the first pose of the virtual camera of the VR device corresponding to the first screen is obtained; Interpolation is performed between the first pose and the second pose of the virtual camera to obtain the intermediate position, wherein the second pose is determined based on the user's head pose after the user's pose change; Based on the intermediate position, a transition screen is generated from the first screen to the second screen, wherein the second screen is determined based on the second pose.

4. The method according to claim 3, characterized in that, The step of interpolating the first pose and the second pose of the virtual camera to obtain the intermediate position includes: The intermediate position is obtained by performing quaternion spherical interpolation or smooth interpolation with damping parameters on the first pose and the second pose of the virtual camera.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The user's body posture is obtained, wherein the body posture is a sitting posture or a standing posture; Based on the user's body posture, the display position of the assessment task in the VR device is adjusted.

6. The method according to any one of claims 1 to 4, characterized in that, The execution results include at least: average reaction time and pass rate for completing the assessment task; The determination of the user's cognitive ability assessment result based on the multimodal physiological data and the execution result includes: Based on the multimodal physiological data, the user's fatigue level is obtained; Based on the average reaction time, the pass rate, and the degree of fatigue, the cognitive ability assessment results of the user are obtained; The user's cognitive ability assessment result is directly proportional to the pass rate and inversely proportional to the average reaction time and the degree of fatigue.

7. The method according to any one of claims 1 to 4, characterized in that, The cognitive ability assessment results characterize the health level of the user's cognitive abilities; The method further includes: If the health status is outside the target range, the difficulty of the assessment task is adjusted based on the cognitive ability assessment result, and the cognitive ability assessment result is updated based on the multimodal physiological data and corresponding execution results of the user during the execution of the assessment task with adjusted difficulty.

8. The method according to claim 7, characterized in that, Adjusting the difficulty of the assessment task based on the cognitive ability assessment results includes: If the health status exceeds the upper limit of the target range, increase the difficulty of the assessment task; If the health status is below the lower limit of the target range, the difficulty of the assessment task is reduced.

9. An edge computing device, characterized in that, include: A processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions which, when executed by the processor, implement the method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 8.

Images