Focus-based debugging and inspection for display systems

Abstract

The present disclosure relates to focus-based debugging and inspection for display systems. The present disclosure provides a method for debugging, the method comprising determining a user's eye focus depth, determining a virtual focus relative to a virtual viewpoint position in a computer-generated real environment based on the user's eye focus depth, transitioning a first object from the computer-generated real environment from a first rendering mode to a second rendering mode based on a position of the virtual focus relative to the first object, wherein the first object obscures visibility of the second object from the virtual viewpoint position in the first rendering mode, and the first object does not obscures visibility of the second object from the virtual viewpoint position in the second rendering mode, and activating a function of a development interface relative to the second object when the first object is in the second rendering mode.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a divisional application of the invention patent application with Chinese national application number 201910555335..2, application date June 25, 2019, priority date July 2, 2018, and invention name “Focus-based debugging and inspection for display systems”.

[0003] This application claims the benefit of U.S. Provisional Application No. 62 / 692,929, filed on July 2, 2018, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0004] The present disclosure generally relates to the field of display devices capable of outputting content based on gaze direction. Background Art

[0005] Displays that rely on gaze direction are used in computer-generated reality systems, such as virtual reality systems, augmented reality systems, and mixed reality systems. For example, a head-mounted display typically includes an optical assembly that directs light from a display device, such as an LCD, LED, or OLED display panel, to the user's eyes. The head-mounted display is operable to present content to the user and may incorporate head tracking and / or hand tracking capabilities. The head-mounted display can be used to present content, such as a three-dimensional computer-generated reality environment, to the user. Such an environment may include many objects, each with its own properties, settings, features, and / or other characteristics. Summary of the invention

[0006] One aspect of the present disclosure is a method for debugging. A method for debugging includes determining a user's eye focus depth, determining a virtual focus relative to a virtual viewpoint position in a computer-generated real environment based on the user's eye focus depth, wherein the computer-generated real environment includes a first object and a second object, transitioning the first object from the computer-generated real environment from a first rendering mode to a second rendering mode based on the position of the virtual focus relative to the first object, wherein the first object obscures visibility of the second object from the virtual viewpoint position in the first rendering mode, and the first object does not obscures visibility of the second object from the virtual viewpoint position in the second rendering mode, and activating a function of a development interface relative to the second object when the first object is in the second rendering mode.

[0007] Another aspect of the present disclosure is a method for debugging. The method includes determining a user's eye focus depth, and based on the user's eye focus depth, determining a virtual focus relative to a virtual viewpoint position in a computer-generated real environment. The computer-generated real environment includes an object. The method also includes selecting the object in response to determining that the virtual focus is within a threshold distance from the object; and activating a function of a development interface relative to the object based on the selection of the object.

[0008] Another aspect of the present disclosure is a method for debugging. The method includes determining the user's eye focus depth, and based on the user's eye focus depth, determining a virtual focus relative to a virtual viewpoint position in a computer-generated real environment. The computer-generated real environment includes an object. The method also includes defining a visibility plane passing through the virtual focus. A first portion of the object is located on a first side of the visibility plane and is presented using a first rendering mode, in which the first portion of the object is fully visible, and a second portion of the object is located on a second side of the visibility plane and is presented using a second rendering mode, in which the second portion of the object is not fully visible. The method also includes activating a function of a development interface relative to the first portion of the object when the second portion of the object is in the second rendering mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a diagram showing a display system including a head-mounted display and a computing device.

[0010] Figure 2 is a block diagram showing a head mounted display.

[0011] Figure 3 is a block diagram illustrating a computing device.

[0012] Figure 4 is a perspective view illustration of a first scene showing an object in a first selection state.

[0013] Figure 5 is a perspective view illustration of a scene showing an object in a second selection state.

[0014] Figure 6 is a perspective view illustration of a scene showing an object in a third selection state.

[0015] Figure 7 is a perspective view illustration of a scene showing a first object in a first rendering mode.

[0016] Figure 8 is a perspective view illustration of a scene showing a first object in a second rendering mode, wherein a second object is visible.

[0017] Fig. 9is a side view illustration of a scene showing an object in a fully visible rendering mode.

[0018] Fig.10 is a side view illustration of a scene showing an object in a partially visible rendering mode.

[0019] Fig.11 is a side view illustration showing a layered object in which all layers are visible.

[0020] Fig.12 is a side view illustration showing a layered object with some layers not visible.

[0021] Fig.13 is a side view illustration of a scene showing a first object and a second object, wherein the first object is in a fully visible presentation mode.

[0022] Fig.14 is a side view illustration of a scene showing a first object and a second object, wherein the first object is in a modified visibility rendering mode.

[0023] Fig.15 is a side view illustration of a scene showing a first object and a second object.

[0024] Fig.16 is a side view illustration of a scene showing a first object, a second object, and an indicator regarding the location of the second object within the scene.

[0025] Fig.17 is a flowchart illustrating a process for focus-based debugging and inspection of a head-mounted display system according to a first example.

[0026] Fig.18 is a flowchart illustrating a process for focus-based debugging and inspection of a head-mounted display system according to a second example.

[0027] Fig.19 is a flowchart illustrating a process for focus-based debugging and inspection of a head-mounted display system according to a third example. DETAILED DESCRIPTION

[0028] The disclosure herein relates to interaction with a computer-generated reality environment embodied in a development interface. The computer-generated reality environment may include a large number of objects and other assets, and may define complex visual and functional relationships. The systems and methods described herein allow debugging and inspection of the computer-generated reality environment within the development interface, including using a focus-based interaction mechanism to enhance and facilitate viewing and selection of objects and other features in the scene.

[0029] Figure 11 is a block diagram illustrating a display system 100. In the examples described herein, the display system 100 includes a head mounted display 102 (herein "HMD") and a computing device 104. The HMD 102 is worn on the head of a user 106 and includes components that allow images to be displayed to the user 106 as part of a computer generated reality (herein "CGR") experience. The images displayed to the user 106 may depend on the gaze direction and / or gaze depth, which may be measured by components associated with the HMD 102, as will be explained herein. The systems and methods described herein may also be implemented in the form of a device that is operable to display content based on the gaze direction and / or gaze depth, but is not mounted or worn on the head of the user 106. The computing device 104 is connected to the HMD 102 via a wired or wireless connection and is operable to compute, present, and / or otherwise provide content to the HMD 102.

[0030] In the illustrated example, the computing device 104 is shown as being separate from the HMD, and may be, for example, a stand-alone computer, such as a laptop or desktop computer. However, it should be understood that some or all of the functions described herein as being performed by the computing device 104 may be performed by equivalent hardware incorporated into the HMD 102. Furthermore, in some implementations, some of the functions described in conjunction with the HMD 102 and / or the computing device 104 may be performed by a mobile computing device, such as a smartphone, that is removably connected to the HMD 102 (i.e., movable between a connected position and a disconnected position). As an example, computing, image display, and / or sensing functions may be performed by a smartphone that is removably connected to the housing of the HMD 102.

[0031] In addition to causing content to be displayed to the user 106, the HMD 102 also incorporates sensing functionality that can be used to control how content is displayed to the user 106. As will be explained herein, head tracking information 108, eye gaze angle 110, and eye focus depth 112 can be detected by sensors associated with the HMD 102 and used as control inputs. The head tracking information 108 represents the angular orientation and / or three-dimensional position of the HMD 102, which varies corresponding to the angular orientation and position of the head of the user 106. The head tracking information 108 can be an angle represented in three dimensions and / or can be represented as a vector in three-dimensional space. The eye gaze angle 110 represents the direction in which the eyes of the user 106 are pointing and viewing the scene, and represents the rotation of the eyes of the user 106 relative to the head of the user 106. The eye gaze angle 110 can be an angle represented in three dimensions and / or can be represented as a vector in three-dimensional space. The eye focus depth 112 represents the distance from the eyes of the user 106 that the user's eyes are attempting to focus on, and can be represented as a distance.

[0032] The computing device 104 is operable to execute software that provides various types of functionality to the user. In the example shown, the computing device 104 provides functionality corresponding to a development interface 114 and a computer-generated reality environment 116. The development interface 114 may be an integrated development environment (IDE) or another software tool that allows the development of software applications, including defining scenes, objects, and defining and / or parameters associated with objects, as well as computer executable code associated with scenes and / or objects. The computer-generated reality environment 116 includes computer executable program instructions and assets that define a three-dimensional scene that can be displayed and interacted with using the HMD 102 and associated input devices. The display and interaction with the computer-generated reality environment 116 is facilitated by the development interface 114, which allows the user to view, inspect, and modify aspects of the computer-generated reality environment and its constituent objects and assets. In the description herein, the development interface 114 and the computer-generated reality environment 116 are described as being implemented using software executed by the computing device 104. In alternative implementations where the HMD 102 is provided with onboard computing and presentation functionality, software associated with the development interface 114 and the computer-generated reality environment 116 may be executed locally by the HMD 102 and the computing device 104 may be omitted or may be optional.

[0033] Figure 2 is a block diagram illustrating HMD 102. HMD 102 may include housing 218, headband 220, display panel 222, optics 224, sensor 226, eye camera 228, processor 230, memory 232, storage 234, data connection 236, and battery 238. As an example, HMD 102 may be configured as any type of CGR system, such as a virtual reality system, a mixed reality system, an optical see-through augmented reality system, or a video see-through augmented reality system.

[0034] The housing 218 is the physical structure to which the other components of the HMD 102 are connected and supported. Various configurations and materials can be used for the housing 218. The housing 218 is connected to a headband 220, which supports the housing 218 relative to the user's head. As examples, the headband 220 can be a flexible band, a rigid structure that extends around part or all of the user's head, or a multi-part structure that includes different types of (e.g., flexible, rigid, elastic, etc.) components. As examples, common configurations that can be used for the headband 220 include a "visor" type configuration, a "halo" type configuration, or a "mohawk" type configuration.

[0035] The display panel 222 is a device operable to output images such as still images or video images. As an example, the display panel 222 may be an LCD display panel, an LED display panel, or an OLED display panel.

[0036] The optics 224 are configured to direct light emitted from the computing device 104 to the user's eyes, and may also allow light from the external environment to reach the user's eyes. As an example, the optics 224 may include lenses, reflectors, polarizers, waveguides, and / or other components. In one implementation, the optics 224 may include a lens positioned between the display panel 222 and the user's eyes. In another implementation, the optics 224 may be configured as an optical combiner such as an off-axis combiner, a polarization beam combiner, or a waveguide combiner.

[0037] Sensor 226 is a device incorporated into HMD 102, such as by being permanently connected to housing 218 or headband 220. Sensor 226 is capable of outputting a signal representative of a sensed condition. Examples of individual sensors that may be incorporated into sensor 226 include an inertial measurement unit that utilizes an accelerometer, a gyroscope, and a magnetometer to output information describing motion, a visible spectrum camera, an infrared spectrum camera, a structured light stereo device, a depth camera, a lidar device, a radar device, an ultrasonic device, an infrared detector that measures signals from an external infrared source, an infrared beacon that emits a signal that can be measured by an external infrared detector, a biometric sensor, a capacitive sensor, a temperature sensor, a light sensor, and a force sensor.

[0038] The eye camera 228 is a device operable to output an image of the user's eye, such as a visible spectrum video camera or an infrared spectrum video camera. The eye camera 228 may be located in the housing 218 of the HMD 102 and directed toward the user's eye. The output from the eye camera 228 may be used to determine the eye gaze angle 110 and the eye focus depth 112 of the user 106. The eye gaze angle 110 and the eye focus depth 112 may be determined using known methods. As an example, the eye gaze angle 110 may be determined based on a vector constructed orthogonal to the pupil of the user's 106 eye (e.g., as an average). Well-known machine vision techniques may be used to determine the identification, location, and angular orientation of the pupil. The eye focus depth 112 may be determined based on the image obtained by the eye camera 228, for example, by determining the convergence point of the vector constructed orthogonal to the user's 106 eye.

[0039] Processor 230 is incorporated into HMD 102, such as by being located in housing 218 of HMD 102. Processor 230 is operable to execute computer program instructions and perform operations described by the computer program instructions. As an example, processor 230 may be a conventional device such as a central processing unit.

[0040] Memory 232 may be a volatile, high-speed, short-term information storage device such as a random access memory module. Memory 234 may be a non-volatile information storage device such as a flash memory module, a hard disk drive, or a solid state drive.

[0041] The data connection 236 is a communication connection that allows information to be exchanged between the HMD 102, the computing device 104, and / or other devices. The data connection 236 may be a wired connection or a wireless connection using any suitable communication protocol.

[0042] Battery 238 may be incorporated in implementations where HMD 102 operates without a power transfer connection to an external computing device, such as computing device 104, or another power source. For example, HMD 102 may include battery 238 in implementations that utilize wireless operation.

[0043] Figure 3 is a block diagram illustrating computing device 104. Computing device 104 may include processor 340, memory 342, storage 344, one or more input devices 346, one or more output devices 348, data connection 350, development interface software 352, and computer generated reality environment software 354 including assets 356.

[0044] Processor 340 is operable to execute computer program instructions and perform operations described by the computer program instructions. As an example, processor 340 can be a conventional device such as a central processing unit. Memory 342 can be a volatile, high-speed, short-term information storage device such as a random access memory module. Memory 344 can be a non-volatile information storage device such as a flash memory module, a hard disk drive, or a solid-state drive. As an example, input device 346 can include a keyboard, a touch screen input device, a gesture input device, an audio input device (e.g., a microphone), a joystick, or a position tracking controller. As an example, output device 348 can include a display screen, a projection display, an audio output, or a tactile output. Data connection 350 is a wired or wireless communication connection that allows communication with HMD 102 using any type of protocol. For example, computing device 104 can transmit pre-rendered content to HMD 102 and use data connection 350 to receive information such as sensor output from HMD 102.

[0045] The development interface software 352 is executable by the computing device 104 and is operable to provide functionality associated with the development interface 114, which can be displayed and / or interacted with using the HMD 102 and / or the computing device 104 during execution of the development interface software 352. The computer-generated reality environment software 354 is executable by the computing device 104 and is operable to provide functionality associated with the computer-generated reality environment 116, which can be displayed and / or interacted with using the HMD 102 and / or the computing device 104 during execution of the computer-generated reality environment software 354. Assets 356 are associated with the computer-generated reality environment software 354 and include content displayed to a user through the computer-generated reality environment 116 and information that controls how the content is displayed and how the user can interact with the content. Examples of assets 356 include three-dimensional models, animation sequences, materials, textures, shaders, lighting objects, virtual camera objects, colliders, physics controllers, particle systems, interactive objects, and scripts.

[0046] Figure 4 4 is a side view illustration showing an example of a scene 460 including an object 462 in a first selection state. The scene 460 is a three-dimensional scene that a user can view through the HMD 102. The scene 460 represents a portion of the computer-generated reality environment 116 and is presented and displayed using the development interface 114, which allows a developer to debug the computer-generated reality environment 116, such as by viewing, testing, interacting with, and / or modifying the computer-generated reality environment 116. The object 462 is a virtual three-dimensional object that exists in the scene 460. The object 462 can be fixed or movable within the scene 460.

[0047] A virtual viewpoint position 464 (e.g., a virtual camera) is positioned relative to the scene 460 and represents the virtual position of the user's eyes in order to present content that will be displayed to the user 106 through the HMD 102. For example, the position of the virtual viewpoint position 464 relative to the scene 460 may be controlled by tracking the HMD 102 and the user's 106 eyes using the head tracking information 108.

[0048] A virtual gaze ray 466 is defined relative to the virtual viewpoint position 464. The virtual gaze ray 466 can be defined as a vector in three-dimensional space relative to the scene 460 using the head tracking information and the eye gaze angle 110 to determine a line of sight direction corresponding to the head angle and the gaze direction of the user 106 relative to the scene. A virtual focus 468 is defined along the virtual gaze ray 466. The virtual focus 468 is an estimated position representing a point in the scene that the eye of the user 106 is attempting to focus on. The distance of the virtual focus 468 from the virtual viewpoint position 464 in virtual space (which may also be referred to as a virtual eye focus depth) is determined based on the eye focus depth 112. As previously described, the eye focus depth 112 can be determined using an image obtained from the eye camera 228. As an example, the distance of the virtual focus 468 from the virtual viewpoint position 464 can be determined by applying a scaling factor to the eye focus depth 112.

[0049] exist Figure 4 , the first selection state corresponds to the absence of any object selection based on the virtual focus 468. Specifically, the virtual gaze ray 466 is oriented toward the object 462, and the virtual focus 468 is located between the virtual viewpoint position 464 and the object 462. In some implementations, the presence or absence of selection of the object 462 is determined based on the presence of the virtual focus 468 within a predetermined distance of the object 462, and Figure 4 , the virtual focus 468 is at a position greater than a threshold distance from the object 462. Therefore, the object 462 is not selected in the first selection state.

[0050] Figure 5 4 is a side view illustration showing a scene 460 including an object 462 in a second selected state. In response to a change in the eye focus depth 112 measured by the eye camera 228 or otherwise measured, the virtual focus 468 has been moved relative to the Figure 4 Move to the position shown in .

[0051] The second selection state corresponds to selecting object 462. In the example shown, virtual focus 468 is inside object 462, and the second selection state may be activated in response to virtual focus 468 being inside object 462. As another example, the second selection state may be activated in response to the virtual focus being within a predetermined distance from object 462, and the distance may be measured relative to a predetermined point relative to object 462, a center point of object 462, a surface of object 462, and / or a bounded area (e.g., a bounding rectangle, a sphere, a capsule, etc.) defined relative to object 462.

[0052] When the second selection state is active, an action may be performed with respect to object 462, such as modifying object 462 and / or a property associated with object 462. As an example, activating the second selection state may pause code execution associated with object 462 (e.g., a script attached to object 462), while code execution continues for other parts of scene 460, such as other objects contained in scene 460. Pausing code execution may include triggering a breakpoint contained in the code associated with object 462. In other words, the gaze information is used to trigger a breakpoint when executing the code associated with object 462. As another example, activation of the second selection state may allow one or more properties of object 462 to be modified using an input device. An example of modifying a property using an input device includes changing a value associated with a property by a button press or using a joystick (e.g., a thumbstick included as part of a handheld input device). If object 462 is a visual object having a color, the color may be modified using a button press or using a joystick. If object 462 is an illumination object, the intensity of light emitted by the illumination object may be modified using a button press or using a joystick. These are merely examples, and other types of properties may be modified in a similar manner using the same or different input modalities.

[0053] Figure 6 4 is a side view illustration showing a scene 460 including an object 462 in a third selected state. In response to a change in the eye focus depth 112 measured by the eye camera 228 or otherwise measured, the virtual focus 468 has been moved relative to the Figure 4 As shown in Figure 6 As explained, in one implementation, the third selection state may be activated by the virtual focus 468 being within the object 462 or within a predetermined distance from the object 462. In another implementation, the third selection state may be activated by the virtual focus 468 being within the object 462 or within a predetermined distance from the object 462 for greater than a threshold period of time.

[0054] The third selection state corresponds to the display of additional information associated with object 462. The additional information may be displayed by a user interface element that is part of scene 460, a user interface element that is added to scene 460 in response to activation of the third activation state, or a user interface element that is part of a heads-up display presented to user 106 as an overlay relative to scene 460. In the illustrated example, the third selection state includes the display of inspector 670 as part of scene 460. Inspector 670 may be, for example, a two-dimensional user interface displayed within scene 460 (e.g., by projecting the user interface on a plane). As an example, inspector 670 may display a value corresponding to a property of object 462 and / or allow the value of a property corresponding to object 462 to be modified.

[0055] Figure 7 7 is a perspective view illustration of a scene 760 showing a first object 762 in a first rendering mode. The scene 760 is depicted from a virtual viewpoint position and along a virtual gaze ray ( Figure 7 760 is oriented. In the illustrated example, the first object 762 is a virtual three-dimensional object that is rendered and output for display as part of the scene 760. The scene 760 may also include a ground plane 761, which may be a virtual object, or may be a physical ground plane in a specific implementation in which the HMD 102 is configured as an augmented reality system or a mixed reality system. The virtual focus 768 configured according to the description of the virtual focus 468 is positioned between the virtual viewpoint position and the first object 762 (e.g., on the "near side" or "first side" of the first object 762) or within the first object 762. In the first rendering mode, a nominal rendering mode may be applied that shows the first object 762 in the scene in a manner in which the first object 762 is intended to appear in the computer-generated real environment 116 when viewed in final form outside the context of the development interface 114. As an example, the first object 762 may be rendered according to a color, material, texture, and / or shader associated with the first object 762.

[0056] Figure 8 760, which shows a first object 762 and a second object 863 in a second rendering mode. In response to the virtual focus 768 having moved past the first object 762 (e.g., to the "far side" or "second side" of the first object 762), the second rendering mode is activated. If this is no longer the case, the first rendering mode may be returned to.

[0057] When a virtual gaze ray associated with a current view of the scene 760 passes through the first object 762, the first rendering mode is a rendering mode in which the first object 762 blocks the visibility of the second object 863, and the second rendering mode is a rendering state in which the first object does not block the visibility of the second object 863.

[0058] In the second rendering mode, the rendering settings of the first object 762 have been changed in response to the changed position of the virtual focus 768. The second rendering mode modifies the rendering settings of the first object 762 relative to the first rendering mode. The transition from the first rendering mode to the second rendering mode may include transitioning from rendering according to the color, material, texture and / or shader associated with the first object 762 to rendering according to the settings associated with the second rendering mode, and the settings are configured to allow visibility of objects that were partially or completely occluded by the first object 762 when the first object 762 was in the first rendering mode. Thus, as in the example shown, when the first object 762 is in the first rendering mode ( Figure 7), second object 863 is obscured and is visible when first object 762 is in a second rendering mode, wherein first object 762 is rendered in a wireframe rendering mode. When in the second rendering mode, functions associated with development interface 114 may be used to perform operations with respect to second object 863, such as, for example, inspecting properties of second object 863, modifying properties of second object 863, and / or pausing code execution of second object 863.

[0059] The second rendering mode modifies the visibility of the first object 762 relative to the first rendering mode. As an example, the first object 762 may change from being visible in the first rendering mode to being invisible in the second rendering mode. As another example, the first object 762 may change from being opaque in the first rendering mode to being transparent in the second rendering mode. As another example, the first object 762 may change from being nominally rendered according to a color, material, texture, and / or shader associated with the first object 762 to being rendered according to a wireframe rendering mode.

[0060] Fig. 9 960, which shows an object 962 in a fully visible rendering mode. A virtual gaze ray 966 extends from a virtual viewpoint position 964 and is oriented toward the object 962, and a virtual focus 968 is located between the virtual viewpoint position 964 and the object 962. The object 962, the virtual viewpoint position 964, the virtual gaze ray 966, and the virtual focus 968 are configured in the manner previously described with respect to the object 462, the virtual viewpoint position 464, the virtual gaze ray 466, and the virtual focus 468. In the fully visible rendering mode, the object 962 is fully visible and is rendered according to the nominal rendering settings using the properties associated with the object 962.

[0061] Fig.10960, which shows an object 962 in a partially visible rendering mode. In the partially visible rendering mode, a virtual focus 968 is positioned within the object 962, and a visibility plane 1069 is defined at the virtual focus 968. A first portion 1063a of the object 962 located between the virtual viewpoint position 964 and the visibility plane 1069 is hidden from view (e.g., not visible in the scene 960 from the perspective of the user 106), or is rendered so that the object can be seen therethrough, such as by a semi-transparent rendering or a wireframe rendering. A second portion 1063b of the object 962 located on a first side of the visibility plane 1069 relative to the virtual viewpoint position 964 is rendered normally and is visible in the scene 960 from the perspective of the user 106. Thus, the partially visible rendering mode allows viewing of an interior portion of the object 962, if any. By updating the position of the visibility plane 1069 as the position of the virtual focal point 968 changes, a cross-sectional view of the object 962 is defined that accommodates the eye focus depth 112 of the user 106 , thereby allowing the user to control the inspection of the interior portion of the object 962 in real time.

[0062] Visibility plane 1069 may be generated with a desired orientation relative to scene 960. As one example, visibility plane 1069 may be generated with a desired orientation relative to the visibility plane, such as by orienting the visibility plane to be perpendicular to virtual gaze ray 966. As another example, visibility plane 1069 may be generated with a desired orientation relative to a world space of scene 960, such as by orienting visibility plane 1069 to be perpendicular to one of the X-axis, Y-axis, or Z-axis of the world space of scene 960. As another example, visibility plane 1069 may be generated with a desired orientation relative to a local coordinate space of object 962, such as by orienting visibility plane 1069 to be perpendicular to one of the X-axis, Y-axis, or Z-axis of the local coordinate space of object 962.

[0063] While in a partial visibility state, user 106 can access functionality associated with development interface 114 to interact with and modify an object or portion of an object (e.g., an interior surface of object 962) that is otherwise not visible from virtual viewpoint position 964, such as, for example, inspecting properties of object 962, modifying properties of object 962, and / or pausing code execution of object 962.

[0064] Fig.11 1160 is a side view of a scene 1160 showing a layered object 1162, wherein all multiple layers, such as first to third layers 1163a, 1163b, and 1163c of the layered object, are visible. As an example, the layered object 1162 can be a user interface including a layered two-dimensional portion and displayed in a three-dimensional space as part of the scene 1160.

[0065] A virtual gaze ray 1166 extends from the virtual viewpoint position 1164 and is oriented toward the layered object 1162, and a virtual focal point 1168 is located between the virtual viewpoint position 1164 and the layered object 1162. The layered object 1162, the virtual viewpoint position 1164, the virtual gaze ray 1166, and the virtual focal point 1168 are configured in the manner previously described with respect to the object 462, the virtual viewpoint position 464, the virtual gaze ray 466, and the virtual focal point 468. Fig.11 In the illustrated example shown in , all layers of the layered object 1162 are fully visible and rendered according to the nominal rendering settings using the properties associated with the layered object 1162 .

[0066] Fig.12 1162 is a side view showing a layered object 1162 in which some layers are not visible. Specifically, the first layer 1163a and the second layer 1163b have been removed from Fig.11 1163c. In response to the virtual focus 1168 moving past the first layer 1163a and the second layer 1163b (e.g., moving to the far side of the first layer 1163a and the second layer 1163b relative to the virtual viewpoint position 1164), the first layer 1163a and the second layer 1163b have transitioned to the invisible state. As an alternative to transitioning to the invisible state, the first layer 1163a and the second layer 1163b may be moved so that they no longer obstruct visibility of the third layer 1163c.

[0067] When the third layer 1163c is no longer obscured by the first layer 1163a and the second layer 1163b, the user 106 can access functionality associated with the development interface 114 to interact with and modify the third layer 1163c, such as, for example, inspecting properties of the third layer 1163c, modifying properties of the third layer 1163c, and / or pausing code execution of the third layer 1163c.

[0068] Fig.131360 is a side view illustration of a scene 1360 showing a first object 1362 and a second object. The scene 1360 is a CGR scene including virtual objects and physical objects. In the scene 1360, the first object 1362 is a virtual object presented in a fully visible presentation mode. The second object 1363 is a physical object that is part of the environment surrounding the user 106. A virtual gaze ray 1366 extends from a virtual viewpoint position 1364 and is oriented toward the first object 1362, and a virtual focus 1368 is located between the virtual viewpoint position 1364 and the first object 1362. The first object 1362, the virtual viewpoint position 1364, the virtual gaze ray 1366, and the virtual focus 1368 are configured in the manner previously described with respect to the object 462, the virtual viewpoint position 464, the virtual gaze ray 466, and the virtual focus 468. In the fully visible presentation mode, the first object 1362 is fully visible and is presented according to the nominal presentation settings using the properties associated with the first object 1362.

[0069] Fig.14 1360, showing first object 1362 in a modified visibility presentation mode, wherein the first object is transparent, semi-transparent, rendered in wireframe, or otherwise does not obstruct visibility of second object 1363. In the partially visible presentation mode, virtual focus 1368 is positioned on the far side of first object 1362 relative to virtual viewpoint position 1364. Thus, the modified visibility presentation mode allows second object 1363 to be viewed in response to virtual focus moving past first object 1362.

[0070] When second object 1363 is no longer obscured by first object 1362 , user 106 may access functionality associated with development interface 114 to inspect second object 1363 , such as, for example, inspecting properties, measurements, and / or interpreted information (e.g., estimated surface geometry and / or position) of second object 1363 .

[0071] Fig.151560 is a side view illustration of a scene 1560 showing a first object 1562 and a second object 1563. The scene 1560 is a CGR scene including physical objects and virtual objects. In the scene 1560, the first object 1562 is a physical object that is part of the environment surrounding the user 106. The second object 1563 is a virtual object that is positioned relative to the first object 1562 so that it appears invisible within the scene 1560 from a virtual viewpoint position 1564, and is therefore not output for display using the HMD 102 due to occlusion by the first object 1562. A virtual gaze ray 1566 extends from the virtual viewpoint position 1564 and is oriented toward the first object 1562, and a virtual focus 1568 is located between the virtual viewpoint position 1564 and the first object 1562. The first object 1562 , the virtual viewpoint position 1564 , the virtual gaze ray 1566 , and the virtual focus 1568 are configured in the manner previously described with respect to the object 462 , the virtual viewpoint position 464 , the virtual gaze ray 466 , and the virtual focus 468 .

[0072] Fig.16 1560, which shows a first object 1562, a second object 1563, and an indicator 1565 regarding the position of the second object within the scene 1560. For example, the indicator 1565 can be an opaque, semi-transparent, or wireframe representation of the second object 1563 or a graphical symbol representing the second object 1563. A virtual focal point 1568 is positioned on the far side of the first object 1562 relative to the virtual viewpoint position 1564. In response to the position of the virtual focal point 1568 moving past the first object 1562 in the scene 1560, the indicator 1565 is output in the scene 1560 so as to appear between the virtual viewpoint position 1564 and the first object 1562, so that the indicator 1565 is operable to indicate to the user 106 that the second object 1563 is behind and obscured by the first object 1562.

[0073] When indicator 1565 is displayed, user 106 may interact with indicator 1565 to access functionality associated with development interface 114 to inspect and / or modify second object 1563 , such as by viewing and / or modifying properties associated with second object 1563 .

[0074] Fig.17 17 is a flow chart illustrating a process 1780 for focus-based debugging and inspection of a display system 100. The process 1780 may be performed, for example, using an HMD 102 and a computing device 104. Portions of the process 1780 may be implemented as computer program instructions executed by a computing device, such as a processor 340 of the computing device 104.

[0075] In operation 1781, sensor output is obtained from a head mounted display such as HMD 102. The sensor output may include or may be interpreted to determine head tracking information 108, eye gaze angle 110, and eye focus depth 112. In operation 1782, a virtual viewpoint position, a virtual gaze ray, and a virtual focal point are determined, as described with respect to virtual viewpoint position 464, virtual gaze ray 466, and virtual focal point 468. The virtual viewpoint position, virtual gaze ray, and virtual focal point are located within a scene in a computer generated reality environment, as described with respect to computer generated reality environment 116, scene 460, scene 760, scene 960, scene 1160, scene 1360, and scene 1560.

[0076] The eye focus depth may be determined using output from a sensor associated with a head mounted display, such as the eye camera 228 of the HMD 102. A virtual gaze ray may be determined based on the eye gaze angle 110 and the head tracking information 108 from the HMD 102. The virtual focus may be located on the virtual gaze ray.

[0077] In operation 1783, based on the position of the virtual focus relative to the first object, a first object located in the computer-generated real environment is transitioned from a first rendering mode to a second rendering mode. As an example, the first object may be located on a virtual gaze ray. Figure 7-8 The transition between the first rendering mode and the second rendering mode is performed in the described manner. For example, the first object may block the visibility of the second object from the virtual viewpoint position in the first rendering mode, and the first object may not block the visibility of the second object from the virtual viewpoint position in the second rendering mode.

[0078] In some implementations, in response to determining that the virtual focus is located on the far side of the first object relative to the virtual viewpoint position, transitioning the first object from the first rendering mode to the second rendering mode is performed. The first rendering mode may utilize rendering settings associated with the first object. In some implementations of the second rendering mode, the first object is not visible in the second rendering mode. In some implementations of the second rendering mode, the first object is semi-transparent in the second rendering mode. In some implementations of the second rendering mode, the first object is rendered as a wireframe in the second rendering mode.

[0079] In operation 1784 , while the first object is in the second presentation mode, functionality of the development interface is activated with respect to the second object, as described with respect to, for example, development interface 114 and inspector 670 .

[0080] In some implementations, activating the function of the development interface causes displaying information associated with the second object. In some implementations, activating the function of the development interface causes modifying information associated with the second object. In some implementations, activating the function of the development interface causes pausing execution of code associated with the second object.

[0081] Fig.18 18 is a flow chart illustrating a process 1880 for focus-based debugging and inspection of a display system 100. The process 1880 may be performed, for example, using an HMD 102 and a computing device 104. Portions of the process 1880 may be implemented as computer program instructions executed by a computing device, such as a processor 340 of the computing device 104.

[0082] In operation 1881, sensor output is obtained from a head mounted display such as HMD 102. The sensor output may include or may be interpreted to determine head tracking information 108, eye gaze angle 110, and eye focus depth 112. In operation 1882, a virtual viewpoint position, a virtual gaze ray, and a virtual focus are determined, as described with respect to virtual viewpoint position 464, virtual gaze ray 466, and virtual focus 468. The virtual viewpoint position, virtual gaze ray, and virtual focus are located within a scene in a computer generated reality environment, as described with respect to computer generated reality environment 116, scene 460, scene 760, scene 960, scene 1160, scene 1360, and scene 1560.

[0083] The computer generated reality environment includes an object. In operation 1883, in response to determining that the virtual focus is within a threshold distance from the object, the object is selected.

[0084] In operation 1884, while the first object is in the second presentation mode, a function of the development interface is activated with respect to the second object, as described with respect to, for example, development interface 114 and inspector 670. In some implementations, activating the function of the development interface is performed in response to determining that the virtual focus has been within a threshold distance from the object for greater than a threshold period of time.

[0085] Fig.19 19 is a flow chart illustrating a process 1980 for focus-based debugging and inspection of a display system 100. The process 1980 may be performed, for example, using an HMD 102 and a computing device 104. Portions of the process 1980 may be implemented as computer program instructions executed by a computing device, such as a processor 340 of the computing device 104.

[0086] In operation 1981, sensor output is obtained from a head mounted display such as HMD 102. The sensor output may include or may be interpreted to determine head tracking information 108, eye gaze angle 110, and eye focus depth 112. In operation 1982, a virtual viewpoint position, a virtual gaze ray, and a virtual focal point are determined, as described with respect to virtual viewpoint position 464, virtual gaze ray 466, and virtual focal point 468. The virtual viewpoint position, virtual gaze ray, and virtual focal point are located within a scene in a computer generated reality environment, as described with respect to computer generated reality environment 116, scene 460, scene 760, scene 960, scene 1160, scene 1360, and scene 1560.

[0087] The computer-generated reality environment includes an object. In operation 1983, a visibility plane is defined so that it passes through the virtual focus. A first portion of the object is located on a first side of the visibility plane and is rendered using a first rendering mode in which the first portion of the object is fully visible, and a second portion of the object is located on a second side of the visibility plane and is rendered using a second rendering mode in which the second portion of the object is not fully visible.

[0088] In operation 1984 , while the first object is in the second presentation mode, functionality of the development interface is activated with respect to the second object, as described with respect to, for example, development interface 114 and inspector 670 .

[0089] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0090] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that people perceive and / or interact with via electronic systems. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0091] A person may sense and / or interact with CGR objects using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0092] Examples of CGR include virtual reality and mixed reality.

[0093] A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment, and / or through a simulation of a subset of the person's physical movements within the computer-generated environment.

[0094] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0095] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0096] Examples of mixed reality include augmented reality and augmented virtuality.

[0097] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. A person uses the system to indirectly view the physical environment via an image or video of the physical environment and perceives the virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0098] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not true version of the original captured image. For another example, a representation of a physical environment may be transformed by graphically eliminating or blurring a portion thereof.

[0099] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more features of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the position of the sun in the physical environment.

[0100] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0101] As described above, one aspect of the present technology is to collect and use data available from various sources to improve the user experience in a computer-generated reality environment. The present disclosure contemplates that, in some instances, these collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identification or personal information.

[0102] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, the personal information data can be used to allow access to user preferences or customizations to enhance the user's experience in a computer-generated reality environment. In addition, the present disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used to provide insights into the user's overall health or can be used as positive feedback to individuals using technology to pursue health goals.

[0103] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to safeguard and protect access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0104] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, using user profiles in a system that provides a computer-generated reality environment, the present technology may be configured to allow users to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for a service. In another example, a user may choose not to use a user profile or to limit the amount of time that information is stored in a user profile. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then the user may be reminded again just before the personal information data is accessed by the application.

[0105] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how the data is stored (e.g., aggregating data on users), and / or other methods when appropriate.

[0106] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, the user experience in a computer-generated environment may be customized by inferring preferences based on non-personal information data or an absolute minimum amount of personal information (such as content requested by a device associated with a user, other non-personal information available to a service, or publicly available information).

Claims

1. A method for debugging, comprising: Determine the depth of eye focus; using the eye focus depth to determine a focus relative to a viewing position in a virtual environment, wherein the virtual environment includes a computer-generated object; as well as When it is determined that the focus is within a threshold distance from the computer-generated object, a function of a computer-executable code development interface relative to the computer-generated object is activated to pause execution of computer-executable code associated with the computer-generated object while continuing execution of computer-executable code associated with another computer-generated object in the virtual environment.

2. The method according to claim 1, wherein: When it is determined that the focus has been within a threshold distance from the computer-generated object for greater than a threshold period of time, activating the function of the computer-executable code development interface is performed.

3. The method according to claim 1, wherein: Activating the function of the computer-executable code development interface causes display of a value corresponding to a property of the computer-generated object.

4. The method according to claim 1, wherein: Activating the function of the computer-executable code development interface causes modification of a value corresponding to a property of the computer-generated object.

5. The method according to claim 1, wherein: Determining the eye focus depth includes determining an output from a sensor of a head mounted display.

6. The method according to claim 1, wherein: The focus is located along a gaze ray, and the computer-generated object is located along the gaze ray, the gaze ray being determined using eye gaze angle and head orientation information.

7. A computer generated reality device comprising: Display panel; sensor; Memory; as well as a processor configured to execute instructions stored in the memory to: Determine the depth of eye focus; using the eye focus depth to determine a focus relative to a viewing position in a virtual environment, wherein the virtual environment includes a computer-generated object; as well as When it is determined that the focus is within a threshold distance from the computer-generated object, a function of a computer-executable code development interface relative to the computer-generated object is activated to pause execution of computer-executable code associated with the computer-generated object while continuing execution of computer-executable code associated with another computer-generated object in the virtual environment.

8. The computer generated reality device of claim 7, wherein: When it is determined that the focus has been within a threshold distance from the computer-generated object for greater than a threshold period of time, activation of the functionality of the computer-executable code development interface is performed.

9. The computer generated reality device of claim 7, wherein: Activation of the function of the computer-executable code development interface causes display of a value corresponding to a property of the computer-generated object.

10. The computer generated reality device of claim 7, wherein: Activation of the function of the computer-executable code development interface results in modification of a value corresponding to a property of the computer-generated object.

11. The computer generated reality device of claim 7, wherein: Determining the eye focus depth includes determining an output from the sensor.

12. The computer generated reality device of claim 7, wherein: The focus is located along a gaze ray, and the computer-generated object is located along the gaze ray, the gaze ray being determined using eye gaze angle and head orientation information.

13. A display system comprising: Memory; as well as a processor configured to execute instructions stored in the memory to: Determine the depth of eye focus; using the eye focus depth to determine a focus relative to a viewing position in a virtual environment, wherein the virtual environment includes a computer-generated object; as well as When it is determined that the focus is within a threshold distance from the computer-generated object, a function of a computer-executable code development interface relative to the computer-generated object is activated to pause execution of computer-executable code associated with the computer-generated object while continuing execution of computer-executable code associated with another computer-generated object in the virtual environment.

14. The display system according to claim 13, wherein: When it is determined that the focus has been within a threshold distance from the computer-generated object for greater than a threshold period of time, activation of the functionality of the computer-executable code development interface is performed.

15. The display system according to claim 13, wherein: Activation of the function of the computer-executable code development interface causes display of a value corresponding to a property of the computer-generated object.

16. The display system according to claim 13, wherein: Activation of the function of the computer-executable code development interface causes modification of a value corresponding to a property of the computer-generated object.

17. The display system according to claim 13, wherein: Determining the eye focus depth includes determining an output from a sensor of a head mounted display.

18. The display system according to claim 13, wherein: The focus is located along a gaze ray, and the computer-generated object is located along the gaze ray, the gaze ray being determined using eye gaze angle and head orientation information.

Images