Virtual reality techniques for characterizing visual abilities

Abstract

A virtual reality system for quantifying a user's functional visual ability under various evaluation conditions (e.g., various light, contrast, and color conditions) using a head-mounted display. In addition to high relevance to the user's eye condition, this embodiment can derive rapid and simple measurements within the controlled and reproducible testing conditions that virtual reality can provide. The virtual environment system can obtain a selection of tasks to be performed. During the execution of the task, the virtual environment optical settings (e.g., modified light settings within the virtual environment display) can be dynamically modified. The user can interact with objects during the execution of the task, which can provide insights into the user's functional visual ability. After the completion of the task, an output quantifying the user's functional visual ability during the execution of the task can be generated.

Description

Background Technology

[0001] Cross-reference regarding related applications

[0002] This application claims the benefit and priority of U.S. Provisional Application No. 63 / 211,930 filed June 17, 2021, the entirety of which is incorporated herein by reference for all purposes.

[0003] Background Technology

[0004] Various eye conditions (e.g., eye diseases) can limit a portion of an individual's vision. For instance, retinitis pigmentosa, a hereditary retinal disease, can primarily affect night vision and peripheral vision and may lead to central vision loss and legal blindness. As another example, geographic atrophy, or Stargardt disease, can reduce an individual's central vision before causing the loss of other visual abilities.

[0005] In many cases, only central vision loss is routinely assessed at the clinic by performing tests such as the best corrected visual acuity test. While this test is well-established, it may fail to detect eye conditions that affect the subject in daily life, such as low-light vision. To address these limitations, one or more other assessments, such as electroretinography, dark adaptation measurements, or visual field measurements, may be supplemented with the best corrected visual acuity test. However, combining these assessments is time-consuming, can be cumbersome for both the subject and the healthcare provider, and may require specialized resources. Consequently, such assessments are generally performed at most once, for instance, at the time of diagnosis. Furthermore, even when combining these assessments, the extent of functional impairment in the subject's vision may not be determined. Meanwhile, the technology forming the background of the present invention is described in 'US 2017 / 273552 A1 (published September 28, 2017)' and 'US 2019 / 391638 A1 (published December 26, 2019)'. means of solving the problem

[0006] Some embodiments of the present disclosure relate to providing a virtual reality environment that displays a visual scene for implementing a task and facilitates tracking the interaction between the environment and the user (e.g., via sensor data). The interaction is converted into an output that evaluates the degree to which the subject's vision is functionally impaired.

[0007] More specifically, a technique for implementing a task to be performed in a virtual reality environment, deriving performance metrics during the execution of the task, and generating an output that quantifies the user's functional visual ability based on performance during the execution of the task is disclosed herein. Optical characteristics may be dynamically modified during the implementation of the task to modify the optical features of the virtual reality environment, which may further identify the user's functional visual ability. Various embodiments are described herein, including devices, systems, modules, methods, and non-transient computer-readable storage media storing programs, code, or instructions executable by one or more processors.

[0008] According to specific embodiments, a method for measuring a user’s functional visual ability within a virtual reality environment, such as an object selection task, an object interaction task, or a reading task, is provided. The method may include the step of identifying a task to be implemented in the virtual reality environment. The virtual reality environment may be displayed by a head-mounted display. The display of the virtual reality environment may include at least one optical setting that is dynamically modified during the implementation of the task. The method may further include the step of facilitating the implementation of the task. The implementation of the task may include displaying a plurality of virtual objects on the display of the VR environment by the head-mounted display.

[0009] The method may further include the step of acquiring a sensor data set from a sensor set during the implementation of the task. The method may further include the step of processing the sensor data set to map a first set of coordinates representing movement within the virtual reality environment directed by the user to a second set of coordinates specifying the location of a dynamic virtual object within the virtual reality environment. The method may further include the step of deriving a first performance metric based on the mapped coordinates. The method may further include the step of generating an output based on the first performance metric. The output may quantify the user's functional visual ability.

[0010] According to a specific embodiment, a virtual environment system is provided. The virtual environment system may include a head-mounted display configured to display a virtual reality environment. The virtual environment system may further include one or more data processors and a non-transient computer-readable storage medium. The non-transient computer-readable storage medium may include instructions that, when executed on one or more data processors, cause said one or more data processors to perform a method. The method may include the step of identifying an operation to be implemented in the virtual reality environment by the head-mounted display. The method may further include the step of facilitating the implementation of the operation. The implementation of the operation may include displaying a plurality of virtual objects on a display of the VR environment by the head-mounted display.

[0011] The method may further include the step of acquiring a sensor data set from a sensor set during the implementation of the task. The method may further include the step of processing the set of sensor data to identify a subset of virtual objects with which the user interacts using a virtual reality system, and the time the user interacted with each of the subsets of virtual objects. The method may further include the step of deriving a first performance metric based on mapped coordinates. The method may further include the step of generating an output based on the first performance metric. The output may quantify the user's functional visual ability during the implementation of the task.

[0012] According to specific embodiments, a computer-implemented method is provided. The computer-implemented method may include the step of identifying a task to be implemented in a virtual reality environment. The virtual reality environment may be configured to be displayed on a head-mounted display. The display of the virtual reality environment may include at least one optical setting that is dynamically modified during the implementation of the task. The computer-implemented method may further include the step of facilitating the implementation of the task. Facilitating the implementation of the task may include displaying a plurality of virtual objects on the display of the VR environment by the head-mounted display. The computer-implemented method may further include the step of acquiring a set of sensor data from a set of sensors during the implementation of the task.

[0013] The computer-implemented method may further include the step of processing the set of sensor data to map a first set of coordinates representing movement within the virtual reality environment directed by the user to a second set of coordinates specifying the location of a dynamic virtual object within the virtual reality environment. The computer-implemented method may further include the step of deriving a first performance metric based on the mapped coordinates. The computer-implemented method may further include the step of processing the set of sensor data to derive spatial movement of a head-mounted display during the implementation of a task. Spatial movement may represent the user's head movement for interacting with a virtual object during the task. The computer-implemented method may further include the step of deriving a second performance metric based on the derived spatial movement. The computer-implemented method may further include the step of generating an output based on the first performance metric and the second performance metric. The output may quantify the user's functional visual ability and the user's spatial movement for interacting with a virtual object during the implementation of a task.

[0014] Some embodiments of the present disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transient computer-readable storage medium comprising instructions that, when executed on one or more data processors, cause one or more data processors to perform all or part of one or more methods disclosed herein and / or all or part of one or more processes. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transient machine-readable storage medium comprising instructions configured to cause one or more data processors to perform all or part of one or more methods disclosed herein and / or all or part of one or more processors.

[0015] The terms and expressions used are for descriptive rather than limiting purposes, and there is no intention to exclude any equivalents or parts thereof of the features illustrated and described in the use of such terms and expressions. However, it is acknowledged that various modifications are possible within the scope of the claimed system and method. Accordingly, although the system and method have been specifically disclosed by way of examples and optional features, modifications and variations of the concepts disclosed herein should be recognized by a person skilled in the art, and it should be understood that such modifications and variations will be considered to be within the scope of the system and method defined by the appended claims.

[0016] This brief description is not intended to identify the principal or essential features of the claimed subject matter, nor is it intended to be used separately to determine the scope of the claimed subject matter. These examples are not mentioned to limit or define the disclosure, but to provide examples to aid in understanding. Additional embodiments and examples are discussed in the detailed description, and further explanation is provided therein. The subject matter should be understood by referring to the entire specification of the disclosure, some or all of the drawings, and the appropriate parts of each claim.

[0017] The foregoing will become more apparent with reference to the following specification, claims, and accompanying drawings, together with other features and embodiments. Brief explanation of the drawing

[0018] The features, embodiments, and advantages of the present disclosure are better understood when the following detailed description is read with reference to the following drawings. Figure 1 is a block diagram illustrating the components of a virtual environment system. Figure 2 is a flow process illustrating an exemplary method for executing a selected task in a virtual environment. Figure 3 illustrates an exemplary virtual environment display of an object selection operation. Figure 4 illustrates an exemplary virtual environment display of an object interaction operation. Figure 5 illustrates an exemplary output representing the user's performance during the task. FIG. 6 illustrates an example of a computer system for implementing some of the embodiments disclosed in this specification. Specific details for implementing the invention

[0019] The technology disclosed herein relates to systems and processes for configuring and using one or more virtual-reality devices to present one or more work environments having a task and various evaluation conditions (e.g., variable light, contrast, color conditions) and to capture user interaction with said work environments. The interaction described herein may include selecting a virtual object in a virtual reality environment as an interaction type corresponding to an interaction type for a task. For example, in an object selection task, the interaction type may include moving the user's position over the location of the virtual object in the virtual reality environment to interact with the virtual object (and optionally providing a trigger event). The user may interact with multiple virtual objects while performing a task.

[0020] Systems and processes may generate metrics that characterize a user's functional visual ability or visual function based on interactions that may be presented to the user and / or transmitted to another device (e.g.). Metrics may be determined based on how well and / or how quickly the user performs each of one or more tasks and / or the user's movement or position during the task (e.g., the degree to which the user leans forward) and how the task performance, movement, and / or position change according to evaluation conditions.

[0021] Therefore, systems and processes can support the rapid collection of large volumes of multidimensional measurements under controlled and reproducible test conditions, so that comparisons over time can provide controlled and quantifiable information on how a user's functional visual ability changes. In some cases, such systems can be used as the primary endpoint in clinical studies to test investigational medicinal products in the field of ophthalmology.

[0022] In an exemplary embodiment, the present embodiment may provide a system and a method performed by a virtual environment system. The virtual environment system may include various components, e.g., a head-mounted display, a base station, and / or a hand controller that tracks the user's hand movements. The virtual environment system may further include a computing device capable of performing some or all of the computational tasks described herein. The head-mounted display may include a display configured to present visual stimuli, one or more speakers configured to present auditory stimuli, one or more sensors configured to measure device movements (corresponding to head movements) (e.g., one or more accelerometers), one or more cameras configured to collect image or video data of the user's eyes (to facilitate eye movement tracking), and one or more speakers configured to capture sound or haptic feedback and / or audio signals. One or more virtual reality devices may include one or more sensors that may be worn or attached to the user's hand or arm, or external sensors (e.g., sensors placed on a chair) that may be used to track hand and / or arm movements.

[0023] A virtual environment system can execute one or more tasks. A task may include a set of commands executed by the virtual environment system. For example, a task may include an object selection task that displays one or more virtual objects in a visual scene and allows interaction with the virtual objects for a certain period of time. A task may be selected from a number of task types based on various parameters, such as a specified eye state related to the user. A task can be executed to display one or more virtual objects in the visual scene.

[0024] A virtual reality system may include one or more sensors (e.g., one or more accelerometers and / or cameras) to detect whether, when, and / or how a user moves their head, hands, and / or arms. Measurements from the sensor(s) may be used to infer the position, location, and / or tilt of the user's head, hands, and / or arms, respectively. The virtual environment system may convert real-world movements, positions, locations, and / or tilts into movements, positions, locations, and / or tilts, respectively, within the virtual environment. In some cases, the coordinate system may be the same for the real world and the virtual environment data, so moving a given amount in a given direction is the same in either space. However, the virtual environment space may be configured so that any movement, position, location, and / or tilt associated with the user conveys information about one or more other objects within the visual scene. For example, in the virtual environment space, data conveying how a user moves their arm may indicate how the movement changes the relative position of the user's arm between the virtual environment space and a specific virtual object. This relative information can be used to determine whether and / or how a user interacts with an object in virtual space (e.g., whether the user touches, grasps, and / or moves the virtual object). Action execution can be determined based on whether and / or when a given type of interaction has occurred.

[0025] During the implementation of the task, the optical settings of the virtual environment can be dynamically modified (e.g., modified optical settings within the visual scene). The display of the virtual reality environment can be modified during the implementation of the task. Through the modified optical setting(s) within the virtual reality environment, the user can interact with virtual objects in a modified eye state provided within the virtual reality environment, which can provide insight into the user's functional visual capabilities.

[0026] After the completion of one or more tasks, the system may process sensor data acquired during the implementation of the tasks to generate an output that characterizes and / or quantifies the user's functional visual abilities. The output may quantify performance metrics related to virtual objects interacted with by the user and the user's spatial movements during the tasks.

[0027] The present embodiment may provide a virtual reality system capable of performing tasks and capturing sensor data from a series of sensors included in the virtual reality system. The virtual reality system includes a head-mounted display that displays a visual scene with one or more modified optical settings. The virtual reality environment displayed on the head-mounted display may simulate a real-world environment and may provide an approximation for evaluating a user's functional visual ability. The virtual reality environment may represent the scene in a closed and restricted manner by blocking external ambient light so that a test can be performed under defined light conditions (e.g., light intensity, color, contrast, and scene composition settings may be controlled in the visual scene).

[0028] The virtual environment system can be used anywhere without special facilities or resources. The system can simultaneously measure body posture, changes in posture, and hand movements, which can provide insights into the user's hand-eye coordination and user compensation strategies due to visual impairment. The system can also measure the user's performance of daily living activities provided in a VR environment, which may include the measurement of functional vision performance.

[0029] Light and scene conditions include any of the following: luminance (e.g., various light levels from bright to dark or vice versa), and dynamic changes in luminance (e.g., flickering light, abrupt changes, gradual changes, fade in / out). The scene may be of low complexity, and the real-world scene may be represented as a 360-degree panoramic image of an object (e.g., restaurant, landscape, night / day scene, busy road). Simulation of the real-world scene may be provided by rendering and computer 3D modeling. The virtual environment system may integrate any of eye tracking, hand tracking, or body / motion capture to evaluate and track changes in posture as indicators of user coping / compensation behaviors. The system may further include object selection and human-system interaction, such as foot switches, audio processing, voice commands, gestures, etc.

[0030] As used herein, the terms “virtual reality environment” or “VR environment” relate to electronically generated displays on a VR-enabled device, such as a head-mounted display (HMD) described herein. A VR environment may display one or more virtual objects that may be static or dynamic (e.g., moving) within the VR environment. In some cases, the environment may include both virtual objects and depictions of real-world features, such as augmented reality (AR) or extended reality (XR) displays. A user may interact with objects within the VR environment using the VR environment system described herein.

[0031] The following examples are provided to introduce specific embodiments. In the following description, specific details are provided for illustrative purposes to provide a thorough understanding of the examples of the disclosure. However, it will be apparent that various examples may be implemented without these specific details. For example, devices, systems, structures, assemblies, methods, and other components may be shown as components in the form of block diagrams so as not to obscure the examples due to unnecessary details. In other cases, well-known devices, processes, systems, structures, and technologies may be shown without specific details to avoid obscuring the examples. The drawings and descriptions are not intended to be limiting. The terms and expressions used in this disclosure are for descriptive purposes only and not for limitation, and the use of such terms and expressions is not intended to exclude any equivalents of the features or parts thereof that are illustrated and described. In this specification, the word "example" is used to mean "an example, a case, or serving as an example." Any embodiment or design described as an "example" in this specification must not be interpreted as being more desirable or advantageous than other embodiments or designs.

[0032] I. Hardware Overview

[0033] FIG. 1 is a block diagram illustrating the components of a virtual environment system (100). The system may include any of a head-mounted display (102), eye-tracking sensors (104a-b), a base station (106a-b), a hand controller (108a-b), and a computing device (110).

[0034] A head-mounted display (HMD) (102) can provide a controlled and independent environment (i.e., controlled lighting conditions, contrast, and scene settings during execution) while the user is wearing it. The HMD (102) can prevent ambient light from the real environment from interfering with the VR scene and can subsequently enable these conditions to be changed in a defined and controlled manner. In some embodiments, the function of projecting the scene through the HMD display may be integrated into the head-mounted display device. In some embodiments, the system may use an external device (e.g., a portable device / smartphone) mounted in a special case (e.g., cardboard) that forms the head-mounted display.

[0035] The HMD (102) may be equipped with a set of electronic sensors, such as a rotational speed sensor, an eye tracking sensor (104a-b), and a camera. The sensors of the HMD (102) can record spatiotemporal dynamics of head movement, such as rotational speed and translational motion, and can reconstruct 3D positional information (in time and space) together with a base station. The eye tracking sensor (104a-b) can enable eye tracking and can capture sensor data about the eyes, such as blinking, pupil size, gaze direction, saccadic motion, and corresponding timestamps. Subsequent analysis of these recorded sensor data may enable evaluation of the user's spatiotemporal reactions and behaviors (head and eyes) in relation to changes in luminance, contrast, scene and object attributes.

[0036] The base station (106a-b) may be equipped with a photoelectric sensor for detecting and reconstructing position information of the HMD (102) and the hand controller (108a-b). The base station (106a-b) may reconstruct position information from the HMD (102) and the hand controller (108a-b) in time and space. This may make it possible to analyze the influence of light and scene conditions provided to the HMD (102) on the subject's performance, movement, and the spatial and temporal trajectories of the hand and head trajectories. In some cases, the HMD (102) may perform the functions described with respect to the base station (106a-b).

[0037] The hand controller (108a-b) may be equipped with electronic sensors, such as accelerometers, and tags that enable the reconstruction of 3D position information. The sensors can record the spatiotemporal dynamics of the movement of both hands and, together with the base station, can derive 3D position information at any time during the use of the system. Subsequent analysis of this recorded sensor data can evaluate the user's spatiotemporal reactions and behaviors in relation to changes in the VR environment projected onto the HMD (102). A system that tracks both hand movements (e.g., via the hand controller (108a-b)) and eye movements (e.g., via the HMD (102)) can track functional visual ability, light conditions, and behavior (and subsequent performance) when performing one or more tasks.

[0038] A motion tracking system can track multiple degrees of motion freedom (position and orientation) of a display as well as user body landmarks of interest (e.g., hand, torso position). The motion tracking system may be an inside-out type (e.g., fusion of optical and depth sensors, LiDAR (Light Detection and Ranging), inertial measurement unit, magnetometer, etc.). The motion tracking system can track position and / or orientation and the location of points of interest on other body parts (e.g., hand, arm). The motion tracking system may include additional components that allow a single device or a set of external devices (optical-based, infrared-based, depth-based, ultra-wideband systems) to track landmark locations on the user and the usage environment directly (e.g., via visual functions) or through additional body-mounted, portable, or environment-placed active (e.g., photodiode, IMU, magnetic sensor, UWB receiver) or passive (e.g., reflective marker) tracking devices.

[0039] A computing device (110) (e.g., a personal computer (PC), a laptop) can execute at least part of the control software of the VR system (scene management and projection, communication with the HMD, controller, base station). The computing device (110) can provide the ability to register and manage user data (e.g.), and to edit and / or select configuration parameters. Additionally or alternatively, the computing device (110) can manage recorded data (e.g., data indicating head movements, eye movements, pupil dilation) and user data, and manage secure data transmission to other data infrastructures. The computing device (110) may include a processing unit that executes commands defined in a software program and handles communication with a display and available optional feedback systems (e.g., audio, haptics).

[0040] In some embodiments, the system may include any of a set of portable controllers, a motion tracking system, an eye tracking system, an auditory system, a haptic feedback system, and an auditory input system.

[0041] The computing device (110) may include software for controlling the characteristics of a visual scene projected onto an HMD (102) by providing functions for capturing and recording user actions / interactions and managing recorded sensor data and user data (e.g., user height, arm length, user identifier). The software and its configuration may control the VR scene and display the environment described herein. The software may control the projection of the visual scene in relation to the user's field of view. The software may convert user interaction with the visual scene into the progression of actions and tasks (e.g., progressing to the next step). The software program may also handle data logging of available tracking data and meta-information.

[0042] The software may include a module for a supervised user to select one or more tasks for the implementation of tasks for the user. The software may include a module for the supervised user to register the user and input demographic data (e.g., age), physical parameters (e.g., arm length, height), eye parameters (e.g., interpupillary distance), etc. The software may further include a module for the supervised user to configure the system according to individual physical parameters (e.g., eye tracking correction, arm reach, seated body height).

[0043] The software may include functions for editing and / or selecting predefined system configurations. Configurations may define evaluation parameters, such as light conditions, timing and duration of tasks and subtasks, contrast of scene objects, size of scene objects, object velocity, target object distribution, and / or composition of scene objects. The software may configure visual scene (panoramic scene) settings, such as a basic scene without decoration, a city night scene, a hotel lobby, and / or a forest. The software may further include functions for storing data (configuration, user, sensor records) for further processing and analysis and transmitting it to other infrastructure in a secure manner.

[0044] The software may execute tasks for evaluating visual functions. Tasks may include displaying a virtual reality environment with actions requested to be performed by the user interacting with the virtual reality system. For example, the task may involve an object selection task. An object selection task may include displaying a scene of multiple objects and requesting the user to identify objects within the scene of object types (e.g., by visually distinguishing visual scenes or reading text prompts). For example, this may include identifying a food item (e.g., an apple) in a scene containing a table and multiple virtual objects of various types.

[0045] Tasks may include objectives for measuring user performance when selecting abstract objects or rendering real-world objects (e.g., cups, plates, keys). For example, tasks may include object / obstacle interaction tasks aimed at measuring user performance when recognizing moving abstract objects (e.g., moving spheres), reacting to them (e.g., selecting the moving object), and avoiding them. Various types of tasks can be implemented in a virtual reality environment. For example, tasks may be selected for implementation based on the user's visual state, selected as part of a predefined method, or selected randomly.

[0046] II. Execution of selected tasks in a visual scene

[0047] FIG. 2 illustrates a flowchart (200) illustrating an exemplary method for executing a selected task in a visual scene. As described herein, various types of tasks may be implemented using a virtual reality system. For example, tasks may include object selection tasks, object interaction tasks, and / or reading tasks as described herein. In some cases, a series of tasks may be implemented in order (e.g., random order, or order defined by user input).

[0048] As previously described, the task may include displaying a scene on a virtual reality display and requesting an action to be performed in a virtual reality environment. In some cases, a series of tasks may be configured to be implemented in sequence, wherein a second task may begin after a first task in the series is completed (e.g., a new scene may be displayed on a virtual reality display and a new requested action to be performed may be provided).

[0049] In block (210), the system may obtain a selection of tasks. This may include identifying a selection to initiate a task or a series of tasks in sequence. For example, a task may be selected based on input provided by the user (or supervised user) based on a visual state associated with the user.

[0050] In block (220), the visual scene may be displayed with specific visual characteristics. Specific optical characteristics may include any feature of the virtual reality display. Examples of specific optical characteristics may include light levels, contrast of virtual objects within the display, addition of text labels, number / size / position of virtual objects within the display, trajectory of movement of virtual objects within the display, etc.

[0051] A visual scene may display one or more virtual objects to allow the user to interact (e.g., by identifying virtual objects and colliding with virtual objects moving toward the user). The visual scene displayed may be specific to the selected task.

[0052] One or more specific optical properties may be dynamically modified during the implementation of the task. For example, specific optical properties that are dynamically modified may include the modified size of a virtual object during the implementation of the task (e.g., to make an object smaller, to modify a text label). As another example, specific optical properties that are dynamically modified may include lowering the light level of a virtual reality display during the implementation of the task.

[0053] Optical properties modified during the operation can enable interaction with virtual objects in a virtual reality environment to test a user's functional visual abilities under modified optical conditions. For example, as light levels decrease during the implementation of the operation, the user's performance during the operation (e.g., performance when identifying virtual objects) may change, which can further assess the user's functional visual abilities.

[0054] In block (230), sensor data may be obtained from a series of sensors included in a virtual reality system. Sensor data may be obtained from sensors within the system (e.g., a set of eye sensors, a set of base stations, a set of hand controllers). Sensor data obtained from the series of sensors may be sorted by data type for subsequent processing. For example, data from eye sensors and data from hand controllers may be sorted separately based on the timestamps of the sensor data for subsequent processing.

[0055] The acquired sensor data can be processed to derive characteristics of the user's behavior in a virtual reality environment. For example, data from an eye-tracking sensor captures pupil position over time, which can be mapped to real-world coordinates.

[0056] Changes in identified coordinates in a real-world coordinate space over time can be identified by specifying the movement of an object (e.g., pupil) over time. For example, changes in the identified coordinates of the pupil over a certain duration can specify the movement of the pupil during said duration. As another example, changes in the identified coordinates of the head within the real-world coordinate space (provided by base station sensor data) can provide the movement of the user's head.

[0057] As described in more detail below, spatial movement can be tracked during operation using sensor data. Spatial movement may include detected physical movements of a head-mounted display captured by a base station sensor. For example, a user may move their head to perform a requested action associated with the task to compensate for various visual limitations of the user. The set of spatial movements may also identify changes in the user's posture, head movements, sudden movements, pupil size, eye movements, etc. Spatial movement may be identified as a second performance metric that may be provided in the output as described below.

[0058] In some cases, the spatial motion set may specify user movements, blinking, user pupil size, etc. These movements may include anomalous behaviors that deviate from the expected range of spatial movement during the implementation of the task. Anomalous movements or behaviors detected by the virtual reality system may be provided as part of the output.

[0059] The identified coordinates of an object in real-world coordinate space can be mapped to coordinates within the virtual reality environment. For example, coordinates specifying the position of the user's pupil in real-world coordinate space at the initial instance can specify the orientation of the pupil in the virtual reality environment coordinate space. The mapped coordinates of an object in the virtual reality environment can be used to identify whether the user is interacting with a virtual object, as described below.

[0060] In block (240), the coordinates of the user's movement can be mapped to the coordinates of a virtual object in the visual scene. The measured coordinates of the user (e.g., the user's pupil, hand, head) within the virtual reality environment can be compared with the coordinates of a virtual object within the virtual reality environment to determine whether the user has interacted with the virtual object in the visual scene in a specific manner. For example, if the coordinates of an object within the visual scene are within a threshold proximity to the coordinates of the user's hand within the visual scene at a given time, it can be determined that the user has interacted with the virtual object in a specific manner.

[0061] In some examples, determining that a user has interacted with a virtual object in a specific manner may include detecting that the mapped virtual space location of the user's hand corresponds to the virtual space location of the virtual object and trigger detection. Trigger events may include interaction with a trigger button on a hand controller, an audible trigger word detected by a virtual reality system, or detection of gaze toward the virtual object for a specified period of time. For example, a criterion may be configured to be satisfied if the virtual space location of the user's hand is within a threshold proximity to the location of the virtual object, and if a trigger is detected within a threshold time while the virtual space location of the hand is within the threshold proximity to the location of the virtual object.

[0062] In block (250), a performance metric may be derived from mapped coordinates. The performance metric may quantify the user's performance for a designated task. For example, if the selected task is an object selection task, the performance metric may quantify the number of virtual objects correctly identified by the user and the time taken to identify / select each virtual object. In some embodiments, the performance metric may indicate or be based on the number of virtual objects that the user interacted with in a specific manner (e.g., selecting a virtual object of the correct object type or trajectory), the number of virtual objects that the user interacted with in another specific manner (e.g., selecting a virtual object of the wrong object type or trajectory), and the virtual space location of each of one or more virtual objects that the user interacted with in a specific manner (e.g., regarding the virtual space location of the user and / or target type of object). The performance metric may indicate the user's functional visual ability. In some cases, block (250) includes deriving a performance metric for each of a plurality of optical settings.

[0063] When deriving performance metrics, the number of virtual objects interacted with by the user can be identified depending on the task. Performance metrics may include a value or a series of values ​​that specify the number of virtual objects interacted with by the user during the task. For example, performance metrics may include a value based on the number of virtual objects interacted with by the user, and the value of the performance metric increases as the number of virtual objects interacted with by the user increases.

[0064] In some embodiments, performance metrics may use dynamically modified optical property(s) in the visual scene to provide insights into various performance levels of the user upon task completion. For example, as the light level of the visual scene decreases during the performance of a task, the user's measured performance when identifying virtual objects may decrease. As another example, it may be determined whether the user's performance in selecting or interacting with virtual objects according to the task decreases as the light level of the visual scene decreases. Performance metrics may identify that the user's performance (e.g., the number of virtual objects correctly identified by the user) has decreased as the light level of the virtual reality environment decreases.

[0065] In some embodiments, a second performance metric may be derived based on the user's spatial movement during the task. The second performance metric may be used together with the first performance metric to generate multiple datasets as shown in the output.

[0066] A set of sensor data (e.g., data acquired from eye tracking sensors (104a-b), hand controllers (108a-b), and base station sensors (106a-b)) can be processed to derive spatial movement of the head-mounted display during the implementation of the task. Spatial movement can represent the user's head movements when interacting with virtual objects. Such spatial movement can further quantify the user's functional visual ability, as more spatial movement generally indicates an increased level of effort required to correctly identify virtual objects. For example, if a user has limited peripheral vision, the user may move their head to identify objects in the visual scene to compensate for the limited peripheral vision. The detected spatial movement can quantify such limitations, which can be expressed in the output as described below.

[0067] A second performance metric can be generated based on derived spatial movements. The second performance metric may include values ​​that quantify the number and magnitude of spatial movements during a task by quantifying the user's head movements while performing the task. The output may be updated to represent both the first performance metric and the second performance metric. The output may quantify the user's functional visual ability and the user's spatial movements while interacting with virtual objects in a virtual reality environment.

[0068] In block (260), an output may be generated. The output may provide a representation of the user's performance during the implementation of the task and / or the user's spatial movement during the implementation of the task. For example, the performance metric may include a series of values ​​representing virtual objects that the user interacted with during the implementation of the task. The system may graphically represent the performance metric as an output to provide a visual representation of the virtual objects that the user interacted with during the implementation of the task. The output may be analyzed to help identify the user's various optical conditions. The output is discussed in more detail in relation to FIG. 5.

[0069] FIG. 3 illustrates an example of a virtual environment display (300) for an object selection task. For example, in FIG. 3, a plurality of virtual objects may be depicted in the virtual environment display. For example, the visual scene may include a first type of virtual object (302), a second type of virtual object (304a-b), and a third type of virtual object (306a-c). A user may interact with virtual objects in the visual scene by identifying virtual objects in the scene of objects. For example, the location of the user (308) may be provided in the virtual environment display (300), and the display (300) may be modified based on movements detected by the user. Additionally, the user may select virtual objects by orienting the user's location over the virtual objects and providing a trigger (e.g., pressing a button on a hand controller). In some embodiments, haptic stimuli may be displayed as feedback in response to the selection of objects. For example, auditory stimuli may be provided based on the correct or incorrect selection of objects.

[0070] As shown in FIG. 3, the operation may include an object selection operation. The purpose (or requested action) of the object selection operation may be to select an abstract target object from a defined number of selectable element objects spread out in a defined manner on a virtual table. In some cases, the object selection operation may specify various object types to find and select within the scene of the object. For example, in a scene containing a table with various object types (e.g., food items, personal items, random objects), the virtual reality system may prompt the selection of a first object type (e.g., a prompt to identify an apple located on the table). Thus, in this example, multiple objects that the user interacts with may include objects of the object type for which selection was prompted during the object selection operation.

[0071] The number of objects, object characteristics, object contrast, light conditions, timing, duration of each task and each attempt, object diffusivity, object shape, object content (e.g., whether the object is filled with text), and the geometric shape of the table may be configured. Various measurements, such as object selection performance (e.g., both correct and incorrect selections), selection time for each object, gaze direction, head position and movement, hand position, movement and speed, upper body posture, and eye parameters (pupil size, fixation, and saccadic movement over time) may be obtained. The result of the task may be performance as a function of light and contrast conditions, serving as an indicator of visual function suitable for distinguishing users with or without functional visual limitations, evaluating disease status and progression, and assessing treatment outcomes.

[0072] In some embodiments, the characteristics of the task may be modified based on the user's performance during the execution of the task. For example, the difficulty of the task (e.g., the number of objects in the environment, the speed of moving objects in the environment, light settings) may increase or decrease depending on the user's performance during the task.

[0073] FIG. 4 illustrates an example of a virtual environment display (400) for an object interaction task. The purpose of the object interaction task may be to recognize moving virtual objects (404, 406) and avoid collisions with such objects. Avoiding collisions with virtual objects moving toward the user's location may include selecting the virtual objects (e.g., "collide" with the virtual objects using hand controllers). In some cases, the user's movement may be intended to avoid incoming virtual objects.

[0074] During the implementation of the task, a user interacting with the head-mounted display can move their eyes / head to move the virtual position of the user (402) in the visual scene to select an incoming object (e.g., select a virtual object (404) to collide with). The number of virtual objects, the characteristics of the objects, the contrast of the objects, the lighting conditions, the speed of the objects, the location where the objects were created, the direction and location in which the objects are delivered to the user (e.g., via the HMD), the timing and duration of the task, and any of the attempts at each task.

[0075] Measurements that may be captured during this task may include the number of selected (touched, missed, ignored) objects, the time taken to selection, the scene hemisphere in which the object was selected / missed, the direction of gaze, head position and movement, hand position, movement and speed, upper body posture, and performance reflected by eye parameters (pupil size over time, fixation, saccadic movement). The output of the task may be performance as a function of light, contrast, and / or object conditions as an indicator of visual function suitable for distinguishing users with or without limitations in functional visual ability, evaluating disease status and progression, and assessing treatment outcomes.

[0076] In some embodiments, the operation may include a read-based operation. The read-based operation may include a request to perform a corresponding action based on text displayed in a virtual reality environment. For example, the read-based operation may include displaying a scene containing text elements (e.g., a bus stop sign indicating a bus schedule). In this operation, a user interacting with the virtual reality system is asked to identify, for example, a bus displayed on the sign. A mode of the read-based operation may be incorporated into any other operation described herein.

[0077] In some embodiments, the operation may include a calibration task. The calibration task may include displaying a visual scene and modifying aspects of the scene to improve the quality of acquired data, system control, etc. For example, virtual objects may be modified during the calibration task to calibrate the mode of operation. The calibration may consist of the implementation of a standard evaluation of visual functions. The calibration may further include identifying user characteristics, e.g., the user's height, where the operation may be adapted based on the user characteristics.

[0078] III. Output generation

[0079] FIG. 5 illustrates an exemplary output (500) representing the user's performance during a task. As illustrated in FIG. 5, the output can quantify the user's performance during the implementation of the task. The output may be based on performance metrics derived as described herein.

[0080] In the example illustrated in FIG. 5, the output (500) can quantify the number of virtual objects that the user interacts with during each part of the task's execution. The output can be generated based on a performance metric that specifies the user's performance during each part of the task. Each point (e.g., 502a-d) exemplifying the first trend line (e.g., solid line) may include the number of objects that the user interacts with during the part of the task, as represented by the first performance metric. The output can quantify the user's performance during the implementation of the task, which can be analyzed to identify the user's various functional visual abilities.

[0081] The output may also represent various spatial movements of the user during the implementation of the task. The second performance metric described herein may specify the number / magnitude of spatial movements in the performance of the task during parts of the task. Each point (504a-d) representing the second trend line (e.g., a dotted line) may quantify the spatial movements of the user during the performance of parts of the task as specified in the second performance metric.

[0082] For example, increased spatial movement may indicate that the user has increased their effort to identify virtual objects in a visual scene under low-light conditions. The output containing the second performance metric may provide examples of spatial movement during task implementation that may indicate user tension in performing the task.

[0083] In some cases, the set of sensor data can capture various movements or actions performed by the user, such as sudden movements, blinking, or changes in pupil size. Many of these actions may be inherently anomalous (e.g., of a type or magnitude that deviates from an expected series of actions) and may indicate various visual limitations of the user. The output may specify the type of action and the time of occurrence of the abnormal event detected during the implementation of the task.

[0084] In some embodiments, the output may provide a graphical representation of the area within the visual scene where interaction with the virtual object occurred during the execution of the task. For example, the graphical representation may provide a heatmap identifying the area of ​​the visual scene (e.g., a quadrant) containing the location of the virtual object. The heatmap may provide insight into the area where the user identified the virtual object and the corresponding area where the user's vision possesses various functional visual capabilities.

[0085] IV. computing environment

[0086] FIG. 6 illustrates an example of a computer system (600) for implementing some of the embodiments disclosed herein. The computer system (600) may have a distributed architecture, wherein some of the components (e.g., memory and processor) are part of an end-user device and some other similar components (e.g., memory and processor) are part of a computer server. The computer system (600) includes at least a processor (602), memory (604), a storage device (606), an input / output (I / O) peripheral (608), a communication peripheral (610), and an interface bus (612). The interface bus (612) is configured to communicate, transmit, and transfer data, control, and commands between various components of the computer system (600). The processor (602) may include one or more processing units, such as a CPU, GPU, TPU, systolic array, or SIMD processor. Memory (604) and storage device (606) include computer-readable storage media, such as RAM, ROM, electrically erasable programmable read-only memory (EEPROM), hard drive, CD-ROM, optical storage device, magnetic storage device, electronic non-volatile computer storage device, such as flash memory, and other types of storage media. Any of these computer-readable storage media may be configured to store instructions or program code that implement the aspects of the present disclosure. Memory (604) and storage device (606) also include computer-readable signal media. Computer-readable signal media include a propagating data signal containing computer-readable program code. Such propagating signal takes various forms, including but not limited to electromagnetic, optical, or a combination thereof.The computer-readable signal medium is not a computer-readable storage medium but includes any computer-readable medium capable of communicating, propagating, or moving a program to be used in connection with a computer system (600).

[0087] Additionally, memory (604) includes an operating system, programs, and applications. The processor (602) is configured to execute stored instructions and includes, for example, a logic processing unit, a microprocessor, a digital signal processor, and other processors. Memory (604) and / or processor (602) may be virtualized and may be hosted, for example, within a cloud network or another computing system in a data center. I / O peripherals (608) include user interfaces, such as a keyboard, a screen (such as a touch screen), a microphone, a speaker, and other input / output devices, and computing components, such as a graphics processing unit, a serial port, a parallel port, a global serial bus, and other input / output peripherals. I / O peripherals (608) are connected to the processor (602) through any of the ports coupled to the interface bus (612). The communication peripheral device (610) is configured to facilitate communication between the computer system (600) and other computing devices through a communication network and includes, for example, a network interface controller, a modem, wireless and wired interface cards, an antenna, and other communication peripheral devices.

[0088] Although the subject matter has been described in detail with respect to specific embodiments, a person skilled in the art will understand that, upon understanding the foregoing, modifications, variations, and equivalents of such embodiments can be easily made. Accordingly, it should be understood that the present disclosure is presented for illustrative purposes rather than for limitation, and does not exclude such modifications, changes, and / or additions to the subject matter that would be readily apparent to a person skilled in the art. In practice, the methods and systems described herein may be implemented in various other forms, and various omissions, substitutions, and modifications to the forms of the methods and systems described herein may be made without departing from the spirit of the present disclosure. The appended claims and their equivalents are intended to encompass forms or modifications that fall within the scope and spirit of the present disclosure.

[0089] Unless otherwise specifically stated, any discussion throughout this specification using terms such as “processing,” “computing,” “calculation,” “decision,” and “identification” is understood to refer to the operation or process of a computing device, such as one or more computers or similar electronic computing devices, that manipulates or converts data expressed as electronic or magnetic physical quantities within the memory, registers, or other information storage devices, transmission devices, or display devices of a computing platform.

[0090] The systems or systems discussed herein are not limited to any specific hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results adjusted according to one or more inputs. Suitable computing devices include general-purpose microprocessor-based computing systems that access stored software for programming or configuring computing systems, ranging from general-purpose computing devices to specialized computing devices that implement one or more embodiments of the subject matter. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software used to program or configure computing devices.

[0091] Embodiments of the method disclosed herein may be performed by the operation of such a computing device. The order of the blocks presented in the examples may vary; for example, blocks may be rearranged, combined, and / or divided into sub-blocks. Specific blocks or processes may be performed in parallel.

[0092] Conditional language used in this specification, e.g., "can," "could," "might," "may," "e.g. e Unless specifically stated otherwise, or otherwise understood in the context in which they are used, such as ”, etc., are generally intended to convey that a particular example includes a particular feature, element, and / or step, but other examples do not. Therefore, such conditional language generally does not imply that a feature, element, and / or step is required in one or more examples in some way, nor does it imply that one or more examples necessarily include logic to determine whether these features, elements, and / or steps will be included or performed in any particular example, regardless of the presence of author input or prompts.

[0093] Terms such as "comprising," "including," and "having" are synonyms and are used in an open-ended, inclusive manner, without excluding additional elements, features, actions, or behaviors. Furthermore, the term "or" is used in an inclusive sense (rather than an exclusive sense); for example, when used to link a list of elements, the term "or" refers to one, some, or all of the elements within said list. In this specification, the use of "adapted to" or "configured to" implies an open and inclusive language that does not exclude devices adapted or configured to perform additional tasks or steps. Additionally, the use of "based on" implies an open and inclusive nature in that a process, step, calculation, or other action "based on" one or more mentioned conditions or values ​​may, in fact, be based on additional conditions or values ​​not mentioned. Likewise, the use of "at least partially based on" implies an open and inclusive nature in that a process, step, calculation, or other action "based at least partially on" one or more mentioned conditions or values ​​may, in fact, be based on additional conditions or values ​​not mentioned. Titles, lists, and numbering included in this specification are for convenience of explanation only and are not intended to be limiting.

[0094] The various features and processes described above may be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, in some implementations, specific method or process blocks may be omitted. The methods and processes described herein are also not limited to any specific order, and the associated blocks or states may be performed in other appropriate orders. For example, the described blocks or states may be performed in an order other than the specifically disclosed order, or multiple blocks or states may be combined into a single block or state. Exemplary blocks or states may be performed serially, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed examples. Likewise, the exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed examples.

Claims

Claim 1 A method for measuring a user’s functional visual ability through a virtual reality environment, the method comprises: identifying a task to be executed in a virtual reality environment; displaying a visual scene to the user that includes one or more virtual objects under at least one optical setting within the virtual reality environment - wherein the head-mounted display is equipped with a first sensor set configured to record the spatiotemporal dynamics of the user’s head movements, and the at least one optical setting is dynamically modified during the execution of the task -; obtaining a set of sensor data from the first sensor set and a second sensor set - wherein the second sensor set is configured to detect the user’s positional movements, said positional movements include hand movements, and said set of sensor data is generated during the execution of the task -; processing the set of sensor data to map a first set of coordinates representing a movement within the virtual reality environment directed by the user to a second set of coordinates specifying the location of the one or more virtual objects within the virtual reality environment; processing the set of sensor data to derive a spatial movement of the head-mounted display during the execution of the task - said spatial movement represents the user’s head movements when interacting with the one or more virtual objects -; and the mapped A method comprising the steps of: deriving a first performance metric based on coordinates and deriving a second performance metric based on the derived spatial movement; and generating an output based on the first performance metric and the second performance metric, wherein the output quantifies the functional visual ability of the user for at least one optical setting within the virtual reality environment. Claim 2 A method according to claim 1, wherein at least one optical setting is dynamically modified from a first setting to a second setting during the execution of the operation, and the optical setting includes a luminance setting, a virtual object contrast setting, and / or a dynamically modified luminance setting. Claim 3 A method according to claim 1, wherein the first performance metric specifies the user's performance during each part of the execution of the task, the second performance metric specifies the number and / or magnitude of the spatial movement during each part of the execution of the task, and the at least one optical setting is dynamically modified to generate different sub-settings, and each part of the execution of the task corresponds to a different sub-setting. Claim 4 A method according to claim 1, wherein the first sensor set comprises an eye-tracking sensor configured to track the user's eye movements, and the second sensor set comprises a hand controller sensor configured to track the user's hand movements and / or trigger events. Claim 5 A method according to claim 1, wherein the task is identified from a set of tasks, and the set of tasks includes an object selection task, an object interaction task, or a read task. Claim 6 A method according to claim 1, wherein the operation includes an object selection operation, and the object selection operation identifies each of the one or more virtual objects by mapping the movement of a user within the virtual reality environment to a location including a virtual object. Claim 7 A method according to claim 1, wherein the operation includes an object interaction operation, and the object interaction operation maps the movement of a user within the virtual reality environment to select the location of one or more virtual objects moving toward the location of the user within the virtual reality environment. Claim 8 As a virtual environment system, a head-mounted display configured to display a virtual reality environment—the head-mounted display comprises a first set of sensors configured to record spatiotemporal dynamics of a user's head movements—and a computing device, wherein the computing device comprises one or more data processors and a non-transient computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform a method, and the method comprises the steps of: identifying a task to be executed in the virtual reality environment; triggering the display of a visual scene by the head-mounted display—the visual scene comprises one or more virtual objects under at least one optical setting within the virtual reality environment, and the at least one optical setting is dynamically modified during the execution of the task—; obtaining a set of sensor data from the first set of sensors and a second set of sensors—the second set of sensors is configured to detect a user's positional movement, and the set of sensor data is generated during the execution of the task—and processing the set of sensor data to identify a subset of a plurality of virtual objects interacted with by the user and the time of interaction with each of the subsets of the plurality of virtual objects. A step of deriving spatial movement of the head-mounted display during the execution of the above task by processing a set of sensor data - wherein the spatial movement represents the head movement of the user when interacting with one or more virtual objects -, a step of deriving a first performance metric based on a subset of the plurality of virtual objects interacting with the user and the time of interaction with each of the subsets of the plurality of virtual objects, and a step of deriving a second performance metric based on the derived spatial movement.A virtual environment system comprising the step of generating an output based on the first performance metric and the second performance metric—the output quantifying the user's functional visual ability for the at least one optical setting within the virtual reality environment. Claim 9 In claim 8, the step of processing the set of sensor data to identify a subset of virtual objects interacting with by a user comprises the step of mapping a first set of coordinates representing a movement within the virtual reality environment directed by the user to a second set of coordinates specifying the location of a subset of the plurality of virtual objects within the visual scene, a virtual environment system. Claim 10 A virtual environment system according to claim 9, wherein the method further comprises the step of detecting a trigger event in a hand controller sensor configured to track a user's hand movement—the trigger event represents the identification of one of the plurality of virtual objects, and processing a set of sensor data to identify a subset of the plurality of virtual objects interacting with the user includes both mapping a first set of coordinates to a second set of coordinates and detecting the trigger event. Claim 11 A virtual environment system according to claim 10, wherein the first sensor set comprises an eye-tracking sensor configured to track the user's eye movements, and the second sensor set comprises a base station sensor configured to track the spatial movements of the head-mounted display, and the eye-tracking sensor and the base station sensor are configured to capture a set of sensor data. Claim 12 A virtual environment system according to claim 8, wherein the above operation includes an object selection operation, and the object selection operation maps the movement by a user within the virtual reality environment to a location containing a virtual object of a specified virtual object type to identify each virtual object of the specified virtual object type. Claim 13 In claim 8, the above operation includes an object interaction operation, and the object interaction operation maps the movement by a user within the virtual reality environment to a position relative to the position of one or more virtual objects within the visual scene, a virtual environment system. Claim 14 A virtual environment system according to claim 8, wherein the method further comprises the steps of: processing a set of sensor data to derive spatial movement of the head-mounted display during the execution of the task—wherein the spatial movement represents the head movement of the user to interact with the one or more virtual objects—; generating a second performance metric based on the derived spatial movement; and updating the output to represent both the first performance metric and the second performance metric—wherein the output quantifies the user's functional visual ability and the spatial movement of the user interacting with the one or more virtual objects. Claim 15 A virtual environment system according to claim 8, wherein the above operation includes an object interaction operation, and the object interaction operation maps the movement of a user within the virtual reality environment to avoid the location of one or more virtual objects within the visual scene. Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images