Virtual reality technology for characterizing visual ability

Abstract

A virtual reality system for quantifying a user's functional visual ability in various evaluation conditions (e.g., varying light, contrast, color conditions) using a head-mounted display. In addition to high relevance to users with optical symptoms, this embodiment can provide quick and easy measurements within the controlled and reproducible test conditions that virtual reality can provide. The virtual environment system can obtain a selection of a task to be performed. During the performance of the task, the virtual environment optical settings (e.g., changed lighting settings within the virtual environment display) can be dynamically changed. The user can interact with objects during the performance of the task, which can provide insight into the user's functional visual ability. After completion of the task, an output is generated that quantifies the user's functional visual ability during the performance of the task.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 211,930, filed on June 17, 2021, which is hereby incorporated by reference in its entirety for all purposes.

Background Art

[0002] Background Various optical conditions (e.g., eye diseases) can potentially limit an individual's visual function. For example, retinitis pigmentosa, a hereditary retinal disease, can primarily affect night vision and peripheral vision and can lead to loss of central vision and legal blindness. As another example, geographic atrophy or Stargardt's disease can first reduce an individual's central vision before the individual loses other visual capabilities.

[0003] Often, by performing tests such as the best - corrected visual acuity test, only the loss of central vision is routinely evaluated in a clinic. While this test is well - established, it may not be able to detect optical conditions that can also affect a subject in daily life, such as vision in low light. To address such limitations, tests such as the best - corrected visual acuity test can be supplemented with one or more other evaluations, such as electroretinogram testing, dark adaptation testing, or visual field measurement evaluations. However, the combination of evaluations can be time - consuming, burdensome for the subject and caregiver, and may require special resources. As a result, such evaluations are typically performed at most once, such as at the time of diagnosis. Furthermore, even a combination of evaluations may not be able to capture the extent to which a subject's vision is functionally impaired.

Summary of the Invention

[0004] Summary Some embodiments of the present disclosure relate to providing a virtual reality environment that presents a visual scene for performing a task and facilitates tracking of a user's interaction with the environment (e.g., via sensor data). The interaction is converted into an output that evaluates the extent to which the subject's vision is functionally impaired.

[0005] More specifically, this document discloses techniques for performing a task executed in a virtual reality environment, deriving performance metrics during the execution of the task, and generating an output that quantifies the user's functional visual ability based on the performance during the execution of the task. The optical characteristics can be dynamically changed during the execution of the task to change the optical characteristics of the virtual reality environment, which can further identify the user's functional visual ability. This document describes various embodiments including devices, systems, modules, methods, programs, codes, or instructions executable by one or more processors, and non-transitory computer-readable storage media storing the same.

[0006] According to certain embodiments, a method for measuring a user's functional visual ability in a virtual reality environment, such as an object selection task, an object interaction task, or a reading task, is provided. The method can include identifying a task to be performed in the virtual reality environment. The virtual reality environment can be displayed by a head-mounted display. The display of the virtual reality environment can include at least one optical setting that is dynamically changed during the execution of the task. The method can also include facilitating the execution of the task. The execution of the task can include displaying a plurality of virtual objects on the display of the VR environment by the head-mounted display.

[0007] The method can also include obtaining a set of sensor data from the sensor set during the performance of the task. The method can also include processing the set of sensor data to map a first set of coordinates representing movement in the virtual reality environment as indicated by the user to a second set of coordinates that specify the position of the dynamic virtual object in the virtual reality environment. The method can also include deriving a first performance metric based on the mapped coordinates. The method can also include generating an output based on the first performance metric. The output can quantify the user's functional visual ability.

[0008] According to certain embodiments, a virtual environment system is provided. The virtual environment system can include a head-mounted display configured to display a virtual reality environment. The virtual environment system can also include one or more data processors and a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium can include instructions that, when executed on the one or more data processors, cause the one or more data processors to execute the method. The method can include identifying a task to be performed in the virtual reality environment by the head-mounted display. The method can also include facilitating the performance of the task. The performance of the task can include displaying a plurality of virtual objects in the VR environment by the head-mounted display.

[0009] The method can also include obtaining a set of sensor data from a sensor set during the performance of the task. The method can also include processing the set of sensor data to identify a subset of virtual objects with which a user interacts using a virtual reality system, and the time at which each of the subset of virtual objects was interacted with by the user. The method can also include deriving a first performance metric based on the mapped coordinates. The method can also include generating an output based on the first performance metric. The output can quantify the user's functional visual ability during the performance of the task.

[0010] In certain embodiments, a computer-implemented method is provided. The computer-implemented method can include identifying a task to be performed in a virtual reality environment. The virtual reality environment can be configured to be displayed on a head-mounted display. The display of the virtual reality environment can include at least one optical setting that is dynamically changed during the performance of the task. The computer-implemented method can also include facilitating the performance of the task. Facilitating the performance of the task can include displaying a plurality of virtual objects in the display of the VR environment by the head-mounted display. The computer-implemented method can also include obtaining a set of sensor data from a sensor set during the performance of the task.

[0011] The computer-implemented method can also include processing a set of sensor data to map a first set of coordinates representing movement in a virtual reality environment as indicated by a user, to a second set of coordinates that specify the position of a dynamic virtual object in the virtual reality environment. The computer-implemented method can also include deriving a first performance metric based on the mapped coordinates. The computer-implemented method can also include processing a set of sensor data to derive the spatial movement of a head-mounted display during the performance of a task. The spatial movement can indicate the movement of a user's head for interacting with virtual objects during the execution of the task. The computer-implemented method can also include deriving a second performance metric based on the derived spatial movement. The computer-implemented method can also include generating an output based on the first performance metric and the second performance metric. The output can quantify the user's functional visual ability and the user's spatial movement for interacting with virtual objects during the performance of the task.

[0012] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods and / or some or all of one or more of the processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform some or all of one or more of the methods and / or some or all of one or more of the processes disclosed herein.

[0013] The terms and expressions used are used as terms of explanation and not of limitation, and in the use of such terms and expressions there is no intention of excluding equivalents of the features shown and described or portions thereof. However, it is recognized that various modifications are possible within the scope of the claimed systems and methods. Accordingly, although the systems and methods are specifically disclosed by way of example and any features, modifications and variations of the concepts disclosed herein should be apparent to those skilled in the art, and it is to be understood that such modifications and variations are considered to be within the scope of the systems and methods defined by the appended claims.

[0014] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. These illustrative examples are not intended to limit or define the disclosure, but rather are referred to in order to provide examples to assist in understanding the disclosure. Further embodiments and examples are discussed in the detailed description, where additional explanation is provided. The subject matter should be understood by reference to the appropriate portions of the entire specification of this disclosure, any or all of the drawings, and each claim.

[0015] The above will become more apparent when considered in conjunction with other features and embodiments, with reference to the following specification, claims, and appended drawings.

[0016] The features, embodiments, and advantages of the present disclosure will be better understood when the following detailed description is read with reference to the following drawings.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Best Mode for Carrying Out the Invention

[0018] Detailed Description The technology disclosed herein generally relates to systems and processes for presenting tasks and one or more task environments having various evaluation states (e.g., various lights, contrasts, color states), and for configuring and using one or more virtual reality devices to capture the user's interaction with the task environment. The interactions described herein can include the selection of virtual objects in a virtual reality environment having interaction types corresponding to the interaction types for the task. For example, in an object selection task, the interaction type can include moving the user's position over the position of a virtual object within the virtual reality environment (and optionally providing a trigger action) for interacting with the virtual object. The user can interact with several virtual objects during the execution of the task.

[0019] The system and process can generate a metric characterizing the user's functional visual ability or visual function based on interactions that can be presented to the user and / or transmitted to another device. The metric can be determined based on how well and / or how quickly the user performs each of one or more tasks and / or the user's movement or position during task execution (e.g., the degree to which the user bends forward), as well as how the task execution, movement, and / or position change over the evaluation states.

[0020] Accordingly, the systems and processes can support the rapid collection of large amounts of multi-dimensional measurement values within controlled and reproducible inspection conditions, such that comparisons over time can provide controlled and quantifiable information regarding how the user's functional visual ability is changing. In some cases, such systems can be used as a primary endpoint in clinical trials for testing investigational pharmaceuticals in ophthalmology.

[0021] In an exemplary embodiment, the present embodiment can provide a system and method executed by a virtual environment system. The virtual environment system can include various components such as a head-mounted display, a base station, and / or a hand controller that tracks the movement of the user's hand. The virtual environment system can also include a computing device capable of performing some or all of the computing operations described herein. The head-mounted display can include a display configured to present visual stimuli, one or more speakers configured to present audio stimuli, one or more sensors (e.g., one or more accelerometers) configured to measure the movement of the device (corresponding to the movement of the head), one or more cameras configured to collect images or video data of the user's eyes (to facilitate tracking of eye movement), sound or tactile feedback, and / or one or more speakers configured to capture audio signals. One or more virtual reality devices can include one or more sensors that can be worn or attached to the user's hand or arm, or can include external sensors of the system (e.g., sensors disposed on a chair) that can be used to track the movement of the hand and / or arm.

[0022] The virtual environment system can execute one or more tasks. A task can include a set of instructions to be executed by the virtual environment system. For example, a task can include an object selection task that displays one or more virtual objects in a visual scene and enables interaction with the virtual objects over a period of time. The task can be selected from a plurality of task types based on various parameters such as specified optical states related to the user. The task can be executed to display one or more virtual objects in the visual scene.

[0023] A virtual reality system can include one or more sensors (e.g., one or more accelerometers and / or cameras) for detecting whether, when, and / or how a user is moving their head, hands, and / or arms. Measurements from the sensors can be used to infer the position, location, and / or orientation of the user's head, hands, and / or arms, respectively. The virtual environment system can convert real-world movement, position, location, and / or orientation into virtual environment movement, position, location, and / or orientation, respectively. In some cases, the coordinate system can be made the same for real-world and virtual environment data such that a given amount of movement in a given direction is the same in either space. However, the virtual environment space may be configured such that any movement, position, location, and / or orientation associated with the user conveys information about one or more other objects within the visual scene. For example, in the virtual environment space, data indicating how the user is moving their arm can indicate how the movement changes the relative position between the user's arm and a particular virtual object within the virtual environment space. This relative information can be used to determine whether and / or how the user is interacting with an object within the virtual space (e.g., whether the user has touched, grasped, and / or moved a virtual object) and / or how they are interacting. Task performance can be determined based on whether and / or when a given type of interaction has occurred.

[0024] During the performance of a task, the optical settings of the virtual environment (e.g., a changed lighting setting in the visual scene) can be dynamically changed. The display of the virtual reality environment can be changed during the performance of the task. The changed optical settings of the virtual reality environment, when provided in the virtual reality environment, can enable the user to interact with virtual objects in the changed optical state and can provide insight into the user's functional visual capabilities.

[0025] After completion of one or more tasks, the system can process the sensor data acquired during the task execution to generate an output that characterizes and / or quantifies the user's functional vision ability. The output can quantify performance metrics regarding virtual objects with which the user interacted during task execution and the user's spatial movements.

[0026] This embodiment can provide a virtual reality system that can execute a task and capture sensor data from a series of sensors included in the virtual reality system. The virtual reality system includes a head-mounted display that displays a visual scene with one or more modified optical settings. The virtual reality environment displayed on the head-mounted display can simulate the real-world environment and provide an approximation for evaluating the user's functional vision ability. The virtual reality environment can represent the scene in a sealed and limited manner that shields from external ambient light so that an inspection can be performed in a defined optical state (e.g., settings of brightness, color, contrast, and scene composition can be controlled in the visual scene).

[0027] The virtual environment system can be used anywhere without the need for special facilities / resources. The system can simultaneously measure body posture and posture changes as well as hand movements, and can provide insights into the user's hand-eye coordination and user compensation strategies as a result of visual impairments. The system can also measure the user performance of activities of daily life as provided in a VR environment that can include measurement of functional vision performance.

[0028] The optical and scene states include any of luminance (e.g., different light levels from bright to dark and vice versa), dynamic changes in luminance (e.g., blinking light, sudden changes, gradual changes, fade-in / fade-out), etc. The scene can be of low complexity and the real scene can be represented by a 360-degree panoramic image of an object (e.g., a restaurant, a landscape, a night / day scene, a congested road). The simulation of the real scene can be provided by rendering and computer 3D modeling. The virtual environment system can incorporate any of eye tracking, hand tracking, body / motion capture to evaluate and track changes in posture as an indicator of the user's coping / compensatory behavior. The system can also include object selection and human-system interaction such as foot switches, audio processing, voice commands, gestures, etc.

[0029] As used herein, the terms "virtual reality environment" or "VR environment" relate to electronically generated displays in VR-capable devices such as the head-mounted display (HMD) described herein. The VR environment can display one or more virtual objects that can be static or dynamic (e.g., moving) within the VR environment. In some cases, the environment can incorporate both virtual objects and depictions of real-world features such as augmented reality (AR) displays or extended reality (XR) displays. The user can interact with the objects within the VR environment using the VR environment system described herein.

[0030] The following examples are provided to introduce specific embodiments. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the examples of the present disclosure. However, it will be apparent that various examples may be practiced without these specific details. For example, devices, systems, structures, assemblies, methods, and other components may be shown as components in block diagram form in order not to obscure the examples with unnecessary detail. In other instances, well-known devices, processes, systems, structures, and techniques may be shown without specific detail in order to avoid obscuring the examples. The drawings and description are not intended to be limiting. The terms and expressions used in this disclosure are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of the features shown and described or portions thereof. The term "example" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as an "example" should not necessarily be construed as preferred or advantageous over other embodiments or designs.

[0031] I. Overview of Hardware FIG. 1 is a block diagram showing the components of a virtual environment system 100. The system can include any of a head-mounted display 102, gaze tracking sensors 104a-b, base stations 106a-b, hand controllers 108a-b, and a computing device 110.

[0032] The head-mounted display (HMD) 102 can provide a controlled and self-contained environment while being worn by the user (i.e., a controlled optical state, contrast, scene setting during execution). The HMD 102 can prevent ambient light from the real environment from interfering with the VR scene and then enable such a state to be changed in a defined and controlled manner. In some embodiments, the ability to project a scene via the HMD display can be integrated into the head-mounted display device. In some embodiments, the system may utilize an external device (e.g., a handheld device / smartphone) attached to a special case (e.g., cardboard) forming the head-mounted display.

[0033] The HMD 102 can be equipped with a set of electronic sensors such as a rotational speed sensor, gaze tracking sensors 104a - b, and a camera. The sensors of the HMD 102 can record the spatio-temporal dynamics of head movements such as rotational speed and translational motion and, in cooperation with the base station, can reconstruct 3D position information (in time and space). The gaze tracking sensors 104a - b can enable gaze tracking and can capture sensor data related to the eyes such as blinks, pupil size, gaze direction, impulsive movements, and corresponding timestamps. Subsequent analysis of such recorded sensor data can enable the evaluation of the user's spatio-temporal responses and behaviors (head and eyes) related to changes in luminance, contrast, scene, and object characteristics.

[0034] The base stations 106a - b can be equipped with optoelectronic sensors for detecting and reconstructing the position information of the HMD 102 and the hand controllers 108a - b. The base stations 106a - b can reconstruct the position information from the HMD 102 and the hand controllers 108a - b both temporally and spatially. This can enable the analysis of the performance, movement, and trajectory of the subject's hands and head in space and time, as well as the influence of optical and scene states as provided on the HMD 102. In some cases, the HMD 102 can perform the functions described for the base stations 106a - b.

[0035] The hand controllers 108a - b can be equipped with electronic sensors such as accelerometers and tags that enable the reconstruction of 3D position information. The sensors can record the spatio - temporal dynamics of hand movements and, in cooperation with the base station, can derive 3D position information at any point during the use of the system. Subsequent analysis of such recorded sensor data can enable the evaluation of the user's spatio - temporal responses and behavior related to changes in the VR environment projected onto the HMD 102. A system that tracks both hand movements (e.g., via the hand controllers 108a - b) and eye movements (e.g., via the HMD 102) can track the functional visual ability, optical state, and behavior (and subsequent performance) when performing one or more tasks.

[0036] The motion tracking system can track multiple degrees of freedom of the movement of the display (position and orientation), as well as user body landmarks of interest (e.g., positions of hands, torso). The motion tracking system can be, for example, inside-out (e.g., fusion of optical sensors such as depth sensors, light detection and ranging (LiDAR), inertial measurement units, magnetometers). The motion tracking system can track the position and / or orientation and position of points of interest from other body parts (e.g., hands, arms). The motion tracking system can include a single device or a set of external devices (light-based, infrared-based, depth-based, ultra-wideband systems) that can track the position of landmarks on the user and the usage environment directly (e.g., via visual features) or via additional body-worn, hand-held, or environment-placed active (e.g., photodiodes, IMUs, magnetic sensors, UWB receivers) or passive (e.g., reflective markers) tracking devices.

[0037] The computing device 110 (e.g., a personal computer (PC), laptop) can execute at least a part of the control software of the VR system (scene management and projection, communication with the HMD, controller, base station). The computing device 110 can provide functions to register and manage user data, for example, and edit and / or select configuration parameters. The computing device 110 can also, or alternatively, manage recorded data (e.g., data representing head movement, eye movement, pupil dilation) and user data and manage secure data transfer to another data infrastructure. The computing device 110 can include a processing unit for executing commands defined by software programs and processing communication with a display and any available feedback system (e.g., audio, tactile).

[0038] In some embodiments, the system can include any of a set of hand-held controllers, a motion tracking system, an eye tracking system, an auditory system, a tactile feedback system, and an auditory input system.

[0039] The computing device 110 can include software for providing functions to control the nature of the visual scene projected onto the HMD 102, capture and record the user's actions / interactions, and manage the recorded sensor data as well as user data (e.g., the user's height, arm length, user identifier). The software and its configuration can control the VR scene and present the environment as described herein. The software can control the projection of the visual scene onto the user's field of view. The software can convert the user interaction with the visual scene into actions and progress of the task (e.g., proceed to the next stage). The software program can also process the data logging of the available tracking data and meta-information.

[0040] The software can include a module for the administrative user to select one or more tasks for the user to perform. The software can include a module for the administrative user to register the user and input demographic data (e.g., age), body parameters (e.g., arm length, height), eye parameters (e.g., interpupillary distance), etc. The software can also include a module for the administrative user to set up the system according to individual physical parameters (e.g., eye tracking calibration, arm reach, seat height).

[0041] Software can include the function of editing and / or selecting a predetermined system configuration. The configuration can define evaluation parameters such as optical state, task and subtask timing and duration, contrast of scene objects, size of scene objects, object speed, target object distribution, and / or makeup of scene objects. Software can configure visual scene (panorama scene) settings such as a basic scene without decoration, a night view scene of a street, a hotel lobby, and / or a forest. Software can also include the function of storing (configuration, user, sensor records) and transferring data in a secure manner to another infrastructure for further processing and analysis.

[0042] Software can perform tasks for the evaluation of visual functions. The tasks can include displaying a virtual reality environment having a required action to be performed by a user interacting with the virtual reality system. For example, the task can be related to an object selection task. The object selection task can include displaying a scene of a plurality of objects and requesting the user to identify an object in the scene (e.g., by visual identification of the visual scene or reading of a text prompt) of the object type. For example, this can include identifying food (e.g., an apple) in a scene including a table and a plurality of virtual objects of various types.

[0043] Tasks can include the purpose of measuring user performance when selecting the rendering of abstract objects or real-world objects (e.g., cups, plates, keys). For example, tasks can include object / obstacle interaction tasks that have the purpose of measuring the user's performance in recognition, reaction (e.g., selection of a moving object), and avoidance of moving abstract objects (e.g., a moving ball). In a virtual reality environment, multiple different types of tasks can be performed. For example, tasks can be selected for execution based on the user's visual state, selected as part of a predetermined order, randomly selected, etc.

[0044] II. Execution of Selected Tasks in a Visual Scene FIG. 2 shows a flow process 200 illustrating an exemplary method for executing a selected task in a visual scene. As described herein, multiple types of tasks can be performed using a virtual reality system. For example, tasks can include object selection tasks, object interaction tasks, and / or reading tasks, as described herein. In some cases, a series of tasks can be performed in order (e.g., in a random order, in an order defined by user input).

[0045] As described above, tasks can include displaying a scene in a virtual reality display and requesting an operation to be performed in the virtual reality environment. In some cases, a series of tasks can be configured to be performed in sequence, and after completion of the first task in the series of tasks, the second task can be started (e.g., a new scene can be displayed in the virtual reality display and a new requested operation to be performed can be provided).

[0046] In block 210, the system can obtain a selection of tasks. This can include identifying a selection for starting a task or a series of tasks in order. For example, the task can be selected based on input provided by the user (or a supervising user) based on a visual state related to the user.

[0047] In block 220, the visual scene can be displayed with specific visual characteristics. The specific optical characteristics can include any feature of the virtual reality display. Examples of specific optical characteristics can include light level, contrast of virtual objects in the display, addition of text labels, number / size / position of virtual objects in the display, trajectory of movement of virtual objects in the display, and the like.

[0048] The visual scene can display one or more virtual objects so that the user can interact with them (e.g., by identifying a virtual object, by squashing a virtual object moving towards the user). The visual scene displayed can be specific to the selected task.

[0049] One or more specific optical characteristics can be dynamically changed during the execution of the task. For example, the specific optical characteristics that are dynamically changed can include the changed size of a virtual object during the execution of the task (e.g., to make the object smaller, to change a text label). As another example, the specific optical characteristics that are dynamically changed can include reducing the light level of the virtual reality display during the execution of the task.

[0050] Changed optical properties during task execution enable interaction with virtual objects in a virtual reality environment and can examine the user's functional visual ability in the changed optical state. For example, when the light level decreases during task execution, the user's performance during task execution (e.g., performance in identifying virtual objects) may change, thereby enabling further identification of the user's functional visual ability.

[0051] In block 230, sensor data can be obtained from a series of sensors included in the virtual reality system. Sensor data can be obtained from sensors within the system (e.g., a set of eye sensors, a set of base stations, a set of hand controllers). Sensor data obtained from a series of sensors can then be arranged for each data type for subsequent processing. For example, data from eye sensors and data from hand controllers can be separately arranged by the time stamp of the sensor data for subsequent processing.

[0052] The acquired sensor data can be processed to derive characteristics of the user's behavior in the virtual reality environment. For example, data from a gaze tracking sensor can capture the pupil position mapped to real-world coordinates over time.

[0053] Changes over time in the identified coordinates in the real-world coordinate space can be identified and can specify the movement of an object (e.g., the pupil) over time. For example, the change in the identified coordinates of the pupil over a period of time can specify the movement of the pupil over that period. As another example, the change in the identified coordinates of the head (provided by base station sensor data) in the real-world coordinate space can provide the movement of the user's head.

[0054] As will be described in more detail below, spatial movement can be tracked during the execution of a task using sensor data. Spatial movement can include the detected physical movement of a head-mounted display captured by a base station sensor. For example, a user can move their head to perform the required actions associated with a task to compensate for various visual limitations of the user. The set of spatial movements can also identify posture changes, head movements, sudden movements, pupil size, eye movements of the user, etc. Spatial movement can be identified in a second performance metric that can be provided in the output as described below.

[0055] In some cases, the set of spatial movements can specify a user's movement, blink, pupil size, etc. Such movements can include abnormal movements that deviate from the expected range of spatial movement during the performance of a task. Abnormal movements or actions detected by a virtual reality system can be provided as part of the output.

[0056] The identified coordinates of an object in the real-world coordinate space can be mapped to coordinates within a virtual reality environment. For example, the coordinates specifying the pupil position of a user in the real-world coordinate space at a first time instance can be mapped to specify the direction of the pupil within the virtual reality environment coordinate space. The mapped coordinates of an object within the virtual reality environment can be used to identify whether a user interacts with a virtual object, as will be described below.

[0057] In block 240, the coordinates of the user's movement can be mapped to the coordinates of virtual objects within the visual scene. The measured coordinates of the user (e.g., the user's pupil, hand, head) in the virtual reality environment are compared to the coordinates of virtual objects in the virtual reality environment to determine whether the user interacted with the virtual objects in a particular way within the visual scene. For example, if the coordinates of an object within the visual scene are within a threshold proximity of the coordinates of the user's hand within the visual scene at a given point in time, it can be determined that the user interacted with the virtual object in a particular way.

[0058] In some cases, determining that the user interacted with the virtual object in a particular way can include detecting that the mapped virtual space position of the user's hand corresponds to the virtual space position of the virtual object, as well as detecting a trigger. Trigger events can include interactions with a trigger button on a hand controller, audible trigger words detected by the virtual reality system, detection of a line of sight to the virtual object for a specified amount of time, and the like. For example, the criteria can be configured to be met if the virtual space position of the user's hand is within a threshold proximity of the position of the virtual object and the trigger is detected within a threshold time from the virtual - space position of the hand within a threshold proximity of the position of the virtual object.

[0059] In block 250, a performance metric can be derived from the mapped coordinates. The performance metric can quantify the performance of a user for a specified task. For example, if the selected task is an object selection task, the performance metric can quantify the number of virtual objects correctly identified by the user and the time to identify / select each virtual object. In some embodiments, the performance metric can indicate or be based on the number of virtual objects with which the user interacted in a particular way (e.g., selecting a virtual object of the correct object type or trajectory), the number of virtual objects with which the user interacted in another particular way (e.g., selecting a virtual object of the wrong object type or wrong trajectory), the respective virtual space positions of one or more virtual objects with which the user interacted in a particular way (e.g., relative to the virtual space positions of the user and / or target type of objects), etc. The performance metric can indicate the functional visual ability of the user. In some cases, block 250 includes deriving a performance metric for each of a plurality of optical settings.

[0060] When deriving the performance metric, the number of virtual objects with which the user interacted according to the task can be identified. The performance metric can include a value or a series of values that specify the number of virtual objects with which the user interacted during the execution of the task. For example, the performance metric can include a value based on the number of virtual objects interacted with by the user, and the value of the performance metric increases as the number of virtual objects interacted with by the user increases.

[0061] In some embodiments, the performance metric can provide insights into the user's different performance in task completion using optically modified properties that change dynamically in the visual scene. For example, as the light level of the visual scene decreases during task execution, the measured performance of the user in identifying virtual objects may decrease. As another example, as the light level of the visual scene decreases, it can be determined whether the user's performance in selecting virtual objects according to the task or interacting with virtual objects decreases. The performance metric can identify that the user's performance (e.g., the number of virtual objects correctly identified by the user) has decreased as the light level of the virtual reality environment decreases.

[0062] In some embodiments, a second performance metric can be derived based on the user's spatial movement during task execution. The second performance metric can be used together with the first performance metric to generate a plurality of data sets as represented in the output.

[0063] A set of sensor data (e.g., data obtained from the eye tracking sensors 104a - b, hand controllers 108a - b, base station sensors 106a - b) can be processed to derive the spatial movement of the head - mounted display during task execution. The spatial movement can indicate the movement of the user's head when interacting with virtual objects. Such spatial movement can further quantify the user's functional visual ability because more spatial movement generally represents an increase in the level of effort required to correctly identify virtual objects. For example, if the user has a restricted peripheral vision, the user can move the head to identify objects within the visual scene to compensate for the restricted peripheral vision. The detected spatial movement can quantify the limitations such that they can be represented in the output as described below.

[0064] The second performance metric can be generated based on the derived spatial movement. The second performance metric can include a value that quantifies the number and magnitude of spatial movements during the execution of the task, and a value that quantifies the movement of the user's head while the task is being executed. The output can be updated to represent both the first performance metric and the second performance metric. The output can quantify the user's functional visual ability and the user's spatial movement for interacting with virtual objects within the virtual reality environment.

[0065] In block 260, an output can be generated. The output can provide a representation of the user's performance during the execution of the task and / or the user's spatial movement during the execution of the task. For example, the performance metric can include a series of values indicating virtual objects with which the user interacted during the execution of the task. The system can graphically represent the performance metric in the output and provide a visual representation of the virtual objects with which the user interacted during the execution of the task. The output can be analyzed to assist in identifying the user's various optical states. The output is described in more detail with respect to FIG. 5.

[0066] Figure 3 shows an example of a virtual environment display 300 for an object selection task. For example, in Figure 3, several virtual objects can be depicted in the virtual environment display. For example, the visual scene can include a first type of virtual object 302, a second type of virtual objects 304a - b, and a third type of virtual objects 306a - c. The user can interact with the virtual objects in the visual scene by identifying the virtual objects within the scene of the objects. For example, the position of the user 308 can be provided to the virtual environment display 300, and the display 300 can be changed based on the detected movement by the user. Further, the user can select a virtual object by indicating the user's position on the virtual object and providing a trigger (e.g., pressing a button on a hand controller). In some embodiments, a tactile stimulus can be displayed in response to the selection of the object as feedback. For example, an auditory stimulus can be provided based on the correct or incorrect selection of the object.

[0067] As shown in Figure 3, the task can include an object selection task. The objective (or required action) of the object selection task can be to select an abstract target object within a defined number of distractor objects that spread in a defined manner on a virtual table. In some cases, the object selection task can specify various object types to identify and select a position within the scene of the objects. For example, in a scene that includes a table with various object types (e.g., food items, personal items, random objects), the virtual reality system can prompt the selection of a first object type (e.g., prompt to identify an apple located on the table). Thus, in this example, the number of objects with which the user interacts can include the objects of the object type that is prompted for selection during the execution of the object selection task.

[0068] The number of objects, the nature of the objects, the contrast of the objects, the optical state, the timing, the duration of each task and each trial, the spread of the objects, the shape of the objects, the content of the objects (e.g., whether the objects are filled with text), and the geometric shape of the table can be configured. Various measurements can be obtained, such as object selection performance (e.g., both correctly and incorrectly selected ones), the selection time for each object, the line-of-sight direction, the position and movement of the head, the position, movement, and speed of the hand, the posture of the upper body, and eye parameters (pupil size over time, fixation, saccadic movements). The result of the task can be the performance as a function of the light and contrast conditions, as an indicator of visual function suitable for distinguishing users with or without limitations in functional visual ability, evaluating disease states and progression, and evaluating treatment outcomes.

[0069] In some embodiments, the characteristics of the task can be changed based on the user's performance during the execution of the task. For example, the difficulty of the task (e.g., the number of objects in the environment, the speed of moving objects in the environment, the light settings) can be increased or decreased based on the user's performance during the execution of the task.

[0070] Figure 4 shows an example of a virtual environment display 400 for an object interaction task. The purpose of the object interaction task can be to recognize moving virtual objects 404, 406 and avoid collisions with such objects. Avoiding a collision with a virtual object moving towards the user's position can include selecting the virtual object (e.g., to "crush" the virtual object using a hand controller). In some cases, the user's movement can be to avoid the oncoming virtual object.

[0071] During the execution of the task, a user interacting with a head-mounted display can move their eyes / head to move the virtual position of user 402 within the visual scene and select an oncoming object (e.g., select virtual object 404 and crush it). Any of several virtual objects, the nature of the object, the contrast of the object, the light state, the speed of the object, the position where the object was created, the direction and position in which the object is presented to the user (e.g., via the HMD), and the timing and duration of the task and each trial of the task.

[0072] Measurements that can be captured during the execution of this task can include the number of (touched, missed, ignored) selected objects, the time to selection, the scene hemisphere in which the object was selected / missed, the line of sight direction, the position and movement of the head, the position, movement and speed of the hand, the upper body posture, and performance as reflected by eye parameters (pupil size over time, fixation, saccadic movements). The output of the task can be performance as a function of light, contrast, and / or object state as an indicator of visual function suitable for discriminating between users, assessing disease state and progression, and evaluating treatment outcome, regardless of the presence or absence of limitations in functional visual ability.

[0073] In some embodiments, the task can include a reading-based task. A reading-based task can include a requirement to perform a corresponding action based on text displayed in a virtual reality environment. For example, a reading-based task can include displaying a scene that includes text elements (e.g., a bus stop sign indicating bus scheduling). In this task, a user interacting with the virtual reality system is required to, for example, identify the bus indicated on the sign. Aspects of the reading-based task can be incorporated into any of the other tasks described herein.

[0074] In some embodiments, the task can include a calibration task. The calibration task can include presenting a visual scene and changing aspects of the scene to improve the quality of the acquired data, control of the system, etc. For example, in order to calibrate aspects of the task, virtual objects can be changed during the execution of the calibration task. Calibration can consist of performing a standard evaluation of visual functions. Calibration can also include identifying user characteristics such as the user's height, and the task can be adapted based on the user's characteristics.

[0075] III. Output Generation FIG. 5 shows an exemplary output 500 representing the user's performance during the execution of a task. As shown in FIG. 5, the output can quantify the user's performance during the execution of the task. The output can be based on the derived performance metric as described herein.

[0076] In the example shown in FIG. 5, the output 500 can quantify the number of virtual objects with which the user interacts between each part of the execution of the task. The output can be generated based on a performance metric that specifies the user's performance between each part of the task. Each point (e.g., 502a - d) indicating the first trend line (e.g., solid line) can include the number of objects with which the user interacts during a part of the task, as represented by the first performance metric. The output can quantify the user's performance during the execution of the task, which can be analyzed to identify various functional visual abilities of the user.

[0077] The output can also show various spatial movements of the user during the execution of the task. The second performance metric described herein can specify the number / size of spatial movements in the execution of the task between parts of the task. Each point 504a - d indicating the second trend line (e.g., dashed line) can quantify the user's spatial movement during the performance of each part of the task, as specified by the second performance metric.

[0078] For example, an increase in spatial movement can indicate an increase in the effort required for a user to identify virtual objects within a visual scene in low light conditions. Output incorporating a second performance metric can provide a map of spatial movement during the performance of a task that can indicate the burden on the user in performing the task.

[0079] In some cases, a set of sensor data can capture various movements or actions performed by a user, such as sudden movements, blinks, changes in pupil size, etc. Many of such actions can be inherently abnormal (e.g., of a type or magnitude deviating from a predicted sequence of actions) and can indicate various visual limitations of the user. The output can specify the action type and occurrence time of abnormal events detected during the performance of the task.

[0080] In some embodiments, the output can provide a graphical representation of regions within a visual scene where there has been an interaction with virtual objects while a task is being performed. For example, the graphical representation can provide a heat map identifying regions (e.g., quadrants) of the visual scene that include the positions of virtual objects. The heat map can provide insights into the regions where virtual objects were identified by the user and the corresponding regions where the user's vision has various functional visual capabilities.

[0081] IV. Computing Environment FIG. 6 shows an example of a computer system 600 for implementing some of the embodiments disclosed herein. The computer system 600 may have a distributed architecture, and some of the components (e.g., memory and processors) are part of an end-user device, and some other similar components (e.g., memory and processors) are part of a computer server. The computer system 600 includes at least a processor 602, a memory 604, a storage device 606, an input / output (I / O) peripheral device 608, a communication peripheral device 610, and an interface bus 612. The interface bus 612 is configured to communicate, transmit, and transfer data, control, and commands among various components of the computer system 600. The processor 602 may include one or more processing units such as a CPU, a GPU, a TPU, a systolic array, or a SIMD processor. The memory 604 and the storage device 606 include computer-readable storage media such as RAM, ROM, electrically erasable programmable read-only memory (EEPROM), hard drives, CD-ROMs, optical storage devices, magnetic storage devices, electronic non-volatile computer storage devices such as flash memory, and other tangible storage media. Any such computer-readable storage media may be configured to store instructions or program code embodying aspects of the present disclosure. The memory 604 and the storage device 606 also include a computer-readable signal medium. The computer-readable signal medium includes a propagated data signal in which computer-readable program code is embodied. Such a propagated signal may take any of various forms including, but not limited to, electromagnetic, optical, or any combination thereof. The computer-readable signal medium includes any computer-readable medium that can communicate, propagate, or transmit a program for use in connection with the computer system 600, which is not a computer-readable storage medium.

[0082] Furthermore, the memory 604 includes an operating system, programs, and applications. The processor 602 is configured to execute the stored instructions and includes, for example, an arithmetic logic unit, a microprocessor, a digital signal processor, and other processors. The memory 604 and / or the processor 602 may be virtualized and may be hosted, for example, within another computing system of a cloud network or a data center. The I / O peripheral devices 608 include user interfaces such as keyboards, screens (e.g., touchscreens), microphones, speakers, and other input / output devices, as well as computing components such as graphics processing units, serial ports, parallel ports, universal serial buses, and other input / output peripheral devices. The I / O peripheral devices 608 are connected to the processor 602 via any of the ports coupled to the interface bus 612. The communication peripheral devices 610 are configured to facilitate communication between the computer system 600 and other computing devices via a communication network and include, for example, network interface controllers, modems, wireless and wired interface cards, antennas, and other communication peripheral devices.

[0083] Although the subject matter has been described in detail with respect to its specific embodiments, it will be understood by those skilled in the art that, upon achieving the above understanding, they can readily generate changes, modifications, and equivalents to such embodiments. Accordingly, it should be understood that this disclosure is presented for purposes of illustration and not limitation, and is not intended to exclude such changes, modifications, and / or additional incorporations to the subject matter that would be readily apparent to those skilled in the art. In fact, the methods and systems described herein may be embodied in various other forms. Furthermore, various omissions, substitutions, and changes may be made in the form of the methods and systems described herein without departing from the spirit of the disclosure. The appended claims and their equivalents are intended to cover such forms or modifications as fall within the scope and spirit of the disclosure.

[0084] Unless otherwise specified, throughout this specification, discussions using terms such as "processing", "computing", "calculating", "determining", and "identifying" refer to the operations or processes of one or more computers or similar electronic computing devices, such as computers or devices that manipulate or transform data represented as physical electronic or magnetic quantities within the memory, registers, or other information storage devices, transmission devices, or display devices of a computing platform.

[0085] One or more systems described herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provides results conditioned on one or more inputs. Suitable computing devices include general-purpose computing devices, multi-purpose microprocessor-based computing systems that access stored software to program or configure a computing system, and dedicated computing devices that implement one or more embodiments of the subject matter. Any suitable programming, scripting, or other type of language or combination of languages can be used to implement the teachings contained herein in the software used to program or configure the computing device.

[0086] Embodiments of the methods disclosed herein can be executed in the operation of such computing devices. The order of the blocks shown in the above examples can be changed; for example, the blocks can be rearranged, combined, and / or divided into sub-blocks. Certain blocks or processes can be executed in parallel.

[0087] Conditional language used herein, such as, among others, "can", "could", "might", "may", "e.g.", etc., generally conveys that a particular example includes a particular feature, element, and / or step, but other examples do not, unless otherwise specified or understood in a different sense within the context in which it is used. Thus, such conditional language is not generally intended to mean that a feature, element, and / or step is required in any way in one or more examples, or that one or more examples necessarily include logic for determining whether these features, elements, and / or steps are included in, or performed by, any particular example, regardless of the presence or absence of author input or prompting.

[0088] Terms such as "comprising", "including", "having" are synonyms and are used in an inclusive, open-ended manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in an inclusive sense (not an exclusive sense), for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. The use of "adapted to" or "configured to" herein means open and inclusive language that does not exclude a device adapted or configured to perform additional tasks or steps. Further, the use of "based on" means that a process, step, calculation, or other operation "based on" one or more recited conditions or values may in fact be based on additional conditions or values beyond those recited. Similarly, the use of "based at least in part on" means that a process, step, calculation, or other operation "based at least in part on" one or more recited conditions or values may in fact be based on additional conditions or values beyond those recited. The headings, lists, and numbering contained herein are for ease of explanation only and are not meant to be limiting.

[0089] The various features and processes described above may be used independently of each other or may be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Further, in some implementations, certain methods or process blocks may be omitted. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks or states may be executed in other suitable sequences. For example, the described blocks or states may be executed in an order other than the specifically disclosed order, or multiple blocks or states may be combined into a single block or state. The exemplary blocks or states may be executed in series, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed examples. Similarly, the exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed examples.

Claims

1. A method for measuring a user's functional visual ability via a virtual reality environment, comprising: identifying a task to be performed in the virtual reality environment, wherein the virtual reality environment is displayed by a head-mounted display and the display of the virtual reality environment includes at least one optical setting that is dynamically changed during the execution of the task; facilitating the execution of the task, wherein the execution of the task includes displaying one or more virtual objects in the display of the virtual reality environment by the head-mounted display; acquiring a set of sensor data from a set of sensors during the execution of the task; processing the set of sensor data to map a first set of coordinates representing movement in the virtual reality environment as indicated by the user to a second set of coordinates specifying the positions of the one or more virtual objects in the virtual reality environment; processing the set of sensor data to derive a spatial movement of the head-mounted display, wherein the spatial movement indicates the movement of the user's head when interacting with the one or more virtual objects; deriving a first performance metric based on the mapped coordinates and a second performance metric based on the derived spatial movement; generating an output based on the first performance metric and the second performance metric, wherein the output quantifies the user's functional visual ability using the optically set dynamically changed in the virtual reality environment; A method comprising the above steps.

2. The method according to claim 1, wherein the at least one optical setting is dynamically changed from a first setting to a second setting during the execution of the task, and the at least one optical setting includes a light intensity setting, a virtual object contrast setting, a dynamically changed brightness setting, the number of the one or more virtual objects displayed in the virtual reality environment, the movement trajectory of the one or more virtual objects displayed in the virtual reality environment, and / or the positions of the one or more virtual objects in the virtual reality environment.

3. The method according to claim 1, wherein The first performance metric specifies the performance of the user during execution of each part of the task, and the second performance metric specifies the number and magnitude / size or number of the spatial movements during execution of each part of the task. A method in which the at least one optical setting is dynamically changed to generate different sub-settings, and execution of each part of the task corresponds to a different sub-setting. **Claim 4** The method according to claim 1, wherein the set of sensors A gaze tracking sensor disposed within the head-mounted display and configured to track the eye movements of the user; A base station sensor disposed within the head-mounted display and configured to identify the spatial movement of the head-mounted display; A hand controller sensor configured to track the hand movements and / or trigger events of the user; comprising. **Claim 5** The method according to claim 1, wherein the task is identified from a set of tasks, and each task in the set of tasks is related to a specific optical symptom of the user. **Claim 6** The method according to claim 1, wherein the task includes an object selection task, The object selection task maps the movement by the user in the display of the virtual reality environment to a position including a virtual object in order to identify each of the one or more virtual objects. **Claim 7** The method according to claim 1, wherein the task includes an object interaction task, The object interaction task maps the movement by the user in the virtual reality environment to select the position of a virtual object that moves towards the position of the user in the virtual reality environment. **Claim 8** A virtual environment system, comprising: A head-mounted display configured to display a virtual reality environment; A computing device, comprising: One or more data processors; A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to Identifying a task to be performed in the virtual reality environment by the head-mounted display, wherein the display of the virtual reality environment includes at least one optical setting that is dynamically changed during the execution of the task; Facilitating execution of the task, wherein execution of the task includes displaying one or more virtual objects in the display of the virtual reality environment by the head-mounted display; During execution of the task, obtaining a set of sensor data from a set of sensors; Processing the set of sensor data to identify a subset of the one or more virtual objects interacted with by the user and the time of interaction with each such subset of the one or more virtual objects; Processing the set of sensor data to derive a spatial movement of the head-mounted display, wherein the spatial movement indicates movement of the user's head when interacting with the one or more virtual objects; Deriving a first performance metric based on the subset of the one or more virtual objects interacted with by the user and the time of interaction with each such subset of the one or more virtual objects, and a second performance metric based on the derived spatial movement; Generating an output based on the first performance metric and the second performance metric, wherein the output quantifies the user's functional visual ability using the optically set dynamically changed in the virtual reality environment; A non-transitory computer-readable storage medium for causing execution of a method comprising; A computing device comprising; A virtual environment system comprising. **Claim 9** In the virtual environment system according to claim 8, The step of processing the set of sensor data to identify the subset of the virtual objects interacted with by the user further includes Mapping a first set of coordinates representing movement in the virtual reality environment indicated by the user using a second set of coordinates specifying the positions of the one or more virtual objects in the virtual reality environment. A virtual environment system. **Claim 10** In the virtual environment system according to claim 9, the method further comprises detecting a trigger operation in a hand controller sensor configured to track the movement of the user's hand, the trigger operation indicating the identification of one of the one or more virtual objects, including the step of processing the set of sensor data to identify the subset of the one or more virtual objects interacted with by the user, including both mapping the first coordinate set using the second coordinate set and detecting the trigger operation. A virtual environment system.

11. The virtual environment system according to claim 10, further comprising a gaze tracking sensor disposed within the head-mounted display and configured to track the eye movement of the user; a base station sensor disposed within the head-mounted display and configured to track the spatial movement of the head-mounted display; and the gaze tracking sensor and the base station sensor are configured to acquire a set of sensor data. A virtual environment system.

12. In the virtual environment system according to claim 8, the task includes an object selection task, The object selection task maps the movement of the user in the display of the virtual reality environment to a position including a virtual object having a specified virtual object type in order to identify each virtual object having the specified virtual object type in a scene including the one or more virtual objects. A virtual environment system.

13. In the virtual environment system according to claim 9, the task includes an object interaction task, The object interaction task maps the movement of the user in the virtual reality environment when it coincides with the position of the virtual object in the virtual reality environment specified in the second coordinate set. A virtual environment system.

14. In the virtual environment system according to claim 8, The first performance metric specifies the user's performance during the execution of each part of the task, and the second performance metric specifies the number and magnitude / or number of the spatial movements during the execution of each part of the task. A virtual environment system in which the at least one optical setting is dynamically changed to generate different sub-settings, and the execution of each part of the task corresponds to a different sub-setting. **Claim 15** In the virtual environment system according to claim 8, The at least one optical setting is dynamically changed from a first setting to a second setting during the execution of the task, and the at least one optical setting includes a light intensity setting, a virtual object contrast setting, a dynamically changed brightness setting, the number of the one or more virtual objects displayed in the virtual reality environment, the movement trajectory of the one or more virtual objects displayed in the virtual reality environment, and the position of the one or more virtual objects in the virtual reality environment. A virtual environment system.

Images