Notifying viewing on artificial reality device to fixed interaction mode

Abstract

When a notification is to be displayed, the artificial reality notification system can add the notification to a predefined location in the user’s field of view (e.g., top, side, or bottom) where it stays as a head-locked virtual object until the user’s gaze is directed at the notification. When the user’s gaze is directed at the notification, the artificial reality notification system can make the notification world-locked, allowing the user to move their head to move the notification to the center of their field of view, to move closer to the notification to make it larger, to move around it to view aspects from different angles, etc. The notification can be dismissed if the user does not direct their gaze at the notification for a first threshold amount of time, or when the user looks away from the world-locked version for a second threshold amount of time.

Description

Technical Field

[0001] This disclosure relates to a look-to-pin user interaction mode on an artificial reality device that helps present information and reduce eye strain. Background Technology

[0002] The various objects in an artificial reality environment are "virtual objects," that is, representations of objects that appear in that environment and are generated by a computing system. Virtual objects in an artificial reality environment can be presented to users through head-mounted displays, mobile devices, projection systems, or other computing systems. Some virtual objects can be notifications, such as system notifications, communication notifications, alarms, notifications from running applications, notifications from remote sources, etc. Existing artificial reality systems tend to display notifications in the center or corners of the display area, regardless of the context or the user's actions. Summary of the Invention

[0003] According to a first aspect of this disclosure, a method is provided for implementing a fixed interaction pattern for viewing a notification in an artificial reality device, the method comprising: displaying a minimized version of the notification as head-leashed at a predefined location in the field of view of the artificial reality device; determining that the gaze of a tracked user is directed toward the head-leashed minimized version of the notification, and in response, converting the notification to a maximized and world-locked version, wherein the maximized and world-locked version of the notification appears to the user to have no change in position when the user turns their head; and determining that the gaze of the tracked user has moved away from the maximized and world-locked notification for a threshold amount of time, and in response, canceling the notification or returning the notification to the minimized and head-leashed version at the predefined location.

[0004] Preferably, the minimized version of the notification includes one or more of the following: an indication of the user and / or application source of the notification; a preview of the content of the notification; the time associated with receiving the notification; a control for responding to or canceling the notification; or any combination of the above.

[0005] Preferably, determining that the tracked user's gaze is directed at the minimized version of the head-lock notification is based on determining that the display timer for displaying the minimized version of the head-lock notification has not yet expired.

[0006] Preferably, the user's gaze is tracked by: modeling the user's eye position; defining a line connecting the center of the user's cornea to the back of the user's retina based on the user's eye position; and determining the location of the endpoint of the line in the display.

[0007] Preferably, the maximized version of the notification includes one or more of the following: a message from the notification; a graphic or 3D model included in the notification; a control or input field for responding to the notification; or any combination of the above.

[0008] Preferably, converting the notification to maximized and world-locked includes converting the notification to maximized and 3DoF world-locked such that the user's viewing distance and angle to the notification remain unchanged as the user approaches and moves away from the notification or moves around the notification.

[0009] Preferably, converting the notification to maximized and world-locked includes converting the notification to maximized and 6DoF world-locked, such that the user's viewing distance and angle to the notification are adjusted as the user moves closer to and further away from the notification or moves around the notification.

[0010] Preferably, the method according to the first aspect further includes: in response to converting the notification to maximized and world-locked, determining that the maximized and world-locked notification and one or more other virtual objects in the field of view of the artificial reality device occupy more than a threshold amount beyond the field of view of the artificial reality device; and in response to determining that the maximized and world-locked notification and one or more other virtual objects in the field of view of the artificial reality device occupy more than a threshold amount beyond the field of view of the artificial reality device: A) converting the one or more other virtual objects to a minimized version or removing the one or more virtual objects from the artificial reality environment; and B) moving the indication of the one or more virtual objects to a separate interface.

[0011] According to a second aspect of this disclosure, a computer-readable storage medium is provided storing instructions that, when executed by a computing system, cause the computing system to perform a process for implementing a fixed interactive mode for viewing a notification. The process includes: displaying a first version of the notification as head-locked at a predefined position within the field of view of an artificial reality device; determining that the gaze of a tracked user is directed towards the head-locked first version of the notification, and in response, converting the notification to a second and world-locked version, wherein the second and world-locked version of the notification appears to the user not to change position when the user turns their head; and determining that the gaze of the tracked user has moved away from the second and world-locked version of the notification for a threshold amount of time, and in response, canceling the notification or returning the notification to the first and head-locked version at the predefined position.

[0012] Preferably, the first version of the notification includes one or more of the following: an indication of the user and / or application source of the notification; a preview of the content of the notification; the time associated with receiving the notification; a control for responding to or canceling the notification; or any combination of the above.

[0013] Preferably, determining that the tracked user's gaze is directed at the first version of the head-lock notification is based on the determination that the display timer for the first version of the head-lock notification has not yet expired.

[0014] Preferably, the user's gaze is tracked by: modeling the user's eye position; defining a line connecting the center of the user's cornea to the back of the user's retina based on the user's eye position; and determining the location of the endpoint of the line in the display.

[0015] Preferably, the second version of the notification includes one or more of the following: a message from the notification; a graphic or 3D model included in the notification; a control or input field for responding to the notification; or any combination of the above.

[0016] Preferably, converting the notification to a second and world-locked version includes converting the notification to 3DoF world-locked, such that the user's viewing distance and angle to the notification remain unchanged as the user approaches and moves away from the notification or moves around the notification.

[0017] Preferably, converting the notification to a second and world-locked version includes converting the notification to 6DoF world-locked, such that the user's viewing distance and angle to the notification are adjusted as the user moves closer to and further away from the notification or moves around the notification.

[0018] Preferably, the process according to the second aspect of this disclosure further includes: in response to converting the notification to a second and world-locked version, determining that the second and world-locked notification version and one or more other virtual objects in the field of view of the artificial reality device occupy more than a threshold amount in the field of view of the artificial reality device; and in response to determining that the second and world-locked notification version and one or more other virtual objects in the field of view of the artificial reality device occupy more than a threshold amount in the field of view of the artificial reality device: A) converting the one or more other virtual objects to a minimized version or removing the one or more virtual objects from the artificial reality environment; and B) moving the indication of the one or more virtual objects to a separate interface.

[0019] According to a third aspect of this disclosure, a computing system is provided for implementing a fixed interaction pattern for viewing notifications. The computing system includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: displaying a first version of the notification as head-locked at a predefined location within the field of view of an artificial reality device; determining that a tracked user's gaze is directed at the head-locked first version of the notification for a threshold amount of time, and in response, converting the notification to a second version and world-locked version, wherein the second and world-locked versions of the notification appear to remain in the same position to the user when the user turns their head; and determining that the tracked user's gaze is moving away from the second and world-locked versions of the notification, and in response, canceling the notification or returning the notification to the first and head-locked versions at the predefined location.

[0020] Preferably, the second version of the notification includes one or more of the following: a message from the notification; a graphic or 3D model included in the notification; a control or input field for responding to the notification; or any combination of the above.

[0021] Preferably, converting the notification to a second and world-locked version includes converting the notification to 6DoF world-locked, such that the user's viewing distance and angle to the notification are adjusted as the user moves closer to and further away from the notification or moves around the notification.

[0022] Preferably, the process according to the third aspect of this disclosure further includes: in response to converting the notification to a second and world-locked version, determining that the second and world-locked notification version and one or more other virtual objects in the field of view of the artificial reality device occupy more than a threshold amount in the field of view of the artificial reality device; and in response to determining that the second and world-locked notification version and one or more other virtual objects in the field of view of the artificial reality device occupy more than a threshold amount in the field of view of the artificial reality device: A) converting the one or more other virtual objects to a minimized version or removing the one or more virtual objects from the artificial reality environment; and B) moving the indication of the one or more virtual objects to a separate interface. Attached Figure Description

[0023] Figure 1 This is a block diagram illustrating an overview of various devices on which some embodiments of the present technology can operate.

[0024] Figure 2A This is a line drawing illustrating a virtual reality headset that can be used in some embodiments of the present technology.

[0025] Figure 2BThis is a line drawing illustrating a mixed reality head-mounted device that can be used in some embodiments of this technology.

[0026] Figure 2C The diagram shows a line drawing of the controller. In some implementations, the user can hold the controller with one or both hands to interact with the artificial reality environment.

[0027] Figure 3 This is a block diagram illustrating an overview of the environment in which some embodiments of the present technology may operate.

[0028] Figure 4 This is a block diagram showing some components that, in some embodiments, can be used in systems employing the disclosed technology.

[0029] Figure 5 This is a flowchart illustrating the process of viewing a fixed interaction pattern for notification in some embodiments of this technology.

[0030] Figures 6A to 6G This is a concept diagram showing an example of viewing a notification presented in a fixed interaction pattern.

[0031] Figure 7A and Figure 7B This is a conceptual diagram illustrating an example of minimizing a virtual object in response to reaching the maximum field of view.

[0032] The technology described herein can be better understood by referring to the following specific embodiments in conjunction with the accompanying drawings, in which similar reference numerals indicate the same or functionally similar elements. Detailed Implementation

[0033] This disclosure relates to an artificial reality notification system that enables a fixed interactive pattern for viewing notifications. When a notification is to be displayed, the artificial reality notification system can add the notification to a predefined location (e.g., top, side, bottom, corner, etc.) within the user's field of view, where the notification remains as a head-locked virtual object until the user's gaze is directed at it. A "head-locked" virtual object is an object that remains in the same position within the user's field of view regardless of how the user moves their head or moves around in the artificial reality environment. When the system detects that the user's gaze is directed at the notification, it can world-lock the notification, allowing the user to move their head to move the notification to the center of their field of view (or, in some cases, to initially move the notification automatically to the center of the user's field of view), and in other cases, to move closer to the notification to make it larger, to move around the notification to view it from different angles, etc. A "world-locked" virtual object is one that the artificial reality device modifies to appear to remain at the same geographical location regardless of the user's movement, as the user moves their head or moves around in the artificial reality environment. When a user rotates their head and moves laterally (i.e., parallel to the floor), a fully world-locked (i.e., six-DOF world-locked or 6DoF world-locked) object appears to remain in the same position in the world to the user. Objects can also be three-DOF world-locked or 3DoF world-locked, where the object appears to remain in the same position in the world to the user when they rotate their head, but does not appear to change distance to the user when they move laterally. Therefore, in various implementations, when a user's gaze is detected pointing at a notification, the artificial reality notification system can make the notification 3DoF or 6DoF world-locked, allowing the user to move their head to move the notification into their field of view (or, in some cases, initially automatically move the notification to the center of the user's field of view). In the 3DoF world-locked version, the user's viewing distance and angle to the notification remain constant as the user moves closer to / away from the notification or around the notification. In the 6DoF world-locked version, the user's viewing distance and angle to the notification are adjusted as the user moves closer to / away from the notification or around the notification. The notification can be canceled if the user gazes at it for less than a first threshold amount of time, or if the user looks away from the world-locked version for a second threshold amount of time.

[0034] In an example of the disclosed technology, an AIVR notification system integrated into an AIVR device can receive notifications that a text message has arrived. The AIVR notification system can format the notification as a minimized version and display it with its head locked at the top of the user's field of view. When the user looks at the minimized version of the notification by pointing their eyes to the top of their field of view, the AIVR notification system locks the notification in place as a world-locked object and maximizes the minimized version. This allows the user to then look up to move the notification to the center of their field of view and get closer to it to zoom in—eliminating the eye strain involved in viewing the notification while it is attached to the top of their field of view. Once the user has finished viewing the world-locked notification, they can remove their gaze from it for up to two seconds, at which point the notification closes out of the user's field of view.

[0035] Although the description of the AI ​​Reality Notification System and related systems and methods throughout this disclosure refers to "notification," in each instance, the disclosed techniques can be used equivalently with other non-notification content items. For example, the AI ​​Reality Notification System may provide output using a fixed-view interaction mode from a minimized drafting application, a fixed-view interaction mode for directions from a wayfinding application, or a fixed-view interaction mode for a minimized view of an audio or video call. Therefore, it will be understood that wherever the term "notification" is used in this disclosure, it can be replaced with any other type of content item.

[0036] In some implementations, when the maximum number of virtual objects exists in the user's field of view (or the maximum number of virtual objects is locked into the field of view) or when virtual objects occupy the largest portion of the user's field of view (or are locked into the field of view), the artificial reality notification system may: A) minimize some of these virtual objects (e.g., replace them with smaller versions or icon-only versions), which may include keeping the virtual objects in their locked positions or moving them to one side or a corner of the user's field of view; or B) remove the virtual objects from the artificial reality environment and move the indications of the virtual objects to a separate interface, such as a launcher panel that the user can freely access and cancel. Therefore, such virtual object constraint conditions prevent visual overload of virtual objects, allowing the user to continue seeing the real world and accessing more important virtual objects without excessive crowding.

[0037] Various embodiments of the disclosed technology may include artificial reality systems, or may be implemented in combination with artificial reality systems. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some way before being presented to a user. This artificial reality or extra reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or content generated in combination with captured content (e.g., real-world photographs). Artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that provides a three-dimensional effect to the viewer). Furthermore, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality, and / or for use in artificial reality (e.g., performing activities in artificial reality). Artificial reality systems that deliver artificial reality content can be implemented on a variety of platforms, including head-mounted displays (HMDs) connected to a host computer system, stand-alone HMDs, mobile devices or computing systems, "cave" environments or other projection systems, or any other hardware platform capable of delivering artificial reality content to one or more viewers.

[0038] As used herein, “virtual reality” or “VR” refers to an immersive experience in which the user’s visual input is controlled by a computing system. “Augmented reality” or “AR” refers to a system in which the user views images of the real world after they have passed through a computing system. For example, a tablet computer with a camera on its back can capture images of the real world and then display those images on a screen on the side of the tablet computer opposite the camera. The tablet computer can process and adjust or “enhance” the images as they pass through the system, for example, by adding virtual objects. “Mixed reality” or “MR” refers to a system in which light entering the user’s eyes is partly generated by a computing system and partly constitutes light reflected from objects in the real world. For example, an MR headset can be shaped as a pair of glasses with a pass-through display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR headset, thus allowing the MR headset to present virtual objects that are blended with the real objects the user can see. As used in this article, “artificial reality,” “surreal,” or “XR” means any one of the following: VR, AR, MR, or any combination or mixture thereof.

[0039] In existing artificial reality (AV) systems, notifications and similar content can be distracting because they are typically locked to the center of the user's field of view. In other existing AV systems, content is locked to the edge of the user's field of view, but this can lead to eye strain because users need to direct their gaze to the edge of their field of view for extended periods while viewing the notification. The disclosed AV notification system and related methods promise to address these problems of existing AV systems by providing a fixed-viewing-to-the-edge user interaction pattern that facilitates less distracting information presentation and reduces eye strain. By initially presenting notifications or other content items with their heads locked to a specific field of view location (e.g., the edge), the AV notification system provides notifications with minimal user distraction. Furthermore, by locking the notification or other content item as soon as user gaze is detected, the AV notification system provides a mechanism to move the notification to a more prominent and natural viewing position, thereby improving access to the notification or content item and reducing eye strain.

[0040] Several embodiments are described in more detail below with reference to the accompanying drawings. Figure 1This is a block diagram illustrating an overview of devices on which some embodiments of the disclosed technology may run. The devices may include hardware components of a computing system 100 that enable viewing notifications in a fixed interactive pattern. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing devices 101, 102, and 103) that communicate via wired or wireless channels to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone head-mounted device capable of providing a user with a computer-created or enhanced experience without the need for external processing or sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as head-mounted devices and a core processing unit (e.g., a console, mobile device, or server system), wherein some processing operations are performed on the head-mounted device while other processing operations are offloaded to the core processing unit. The following is in conjunction with… Figure 2A and Figure 2B An example head-mounted device is described. In some embodiments, location and environmental data may be collected solely by sensors incorporated in the head-mounted device, while in other embodiments, one or more of a plurality of non-head-mounted computing devices may include sensor components capable of tracking environmental or location data.

[0041] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which may be located in one device or distributed across multiple devices (e.g., distributed across two or more computing devices 101 to 103).

[0042] The computing system 100 may include one or more input devices 120 that provide input to the processor 110, thereby notifying the processor of actions. These actions may be transmitted by a hardware controller that interprets the signals received from the input devices and transmits the information to the processor 110 using a communication protocol. Each input device 120 may include, for example, a mouse, keyboard, touchscreen, touchpad, wearable input device (e.g., haptic gloves, bracelets, rings, earrings, necklaces, watches, etc.), camera (or other light-based input device, such as an infrared sensor), microphone, or other user input device.

[0043] Processor 110 can be coupled to other hardware devices, for example, using an internal bus or an external bus, such as a Peripheral Component Interconnect Standard (PCI) bus, a Small Computer System Interface (SCSI) bus, or a wireless connection. Processor 110 can communicate with hardware controllers for various devices, such as for display 130. Display 130 can be used to display text and graphics. In some embodiments, display 130 includes an input device, which is part of the display, for example, when the input device is a touchscreen or equipped with an eye-tracking orientation monitoring system. In some embodiments, the display is separate from the input device. Examples of display devices are: liquid crystal display (LCD) screens, light-emitting diode (LED) screens, projection displays, holographic displays, or augmented reality displays (e.g., heads-up displays or head-mounted displays), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as network chips or network cards, video chips or video cards, audio chips or audio cards, Universal Serial Bus (USB), FireWire or other external devices, cameras, printers, speakers, optical disc read-only memory (CD-ROM) drives, digital video disc (DVD) drives, disk drives, etc.

[0044] In some implementations, computing system 100 can use input from I / O device 140 (e.g., camera, depth sensor, inertial motion unit (IMU) sensor, GPS unit, LiDAR, or other time-of-flight sensors) to identify and map the user's physical environment while tracking the user's position within that environment. This simultaneous localization and mapping (SLAM) system can generate maps (e.g., topology, grids, etc.) of an area (which may be a room, building, outdoor space, etc.) and / or obtain maps previously generated by computing system 100 or by another computing system that has already mapped the area. The SLAM system can track the user within the area based on factors such as GPS data, match identified objects and structures with mapped objects and structures, and monitor acceleration and other positional changes.

[0045] The computing system 100 may include a communication device capable of wireless or wired communication with other local computing devices or network nodes. This communication device may use, for example, Transmission Control Protocol / Internet Protocol (TCP / IP) to communicate with another device or server over a network. The computing system 100 may use this communication device to distribute operations across multiple network devices.

[0046] Processor 110 can access memory 150, which may be included on one of a plurality of computing devices of computing system 100, or may be distributed across a plurality of computing devices of computing system 100, or may be distributed across a plurality of other external devices. Memory includes one or more hardware devices for volatile or non-volatile storage, and may include both read-only memory and writable memory. For example, memory may include one or more of the following: random access memory (RAM), various cache memories, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard disk drive, floppy disk, optical disk drive (CD), DVD, magnetic storage device, and tape drive. Memory is not a propagating signal detached from the underlying hardware; therefore, memory is non-transitory. Memory 150 may include program memory 160, which stores programs and software, such as operating system 162, artificial intelligence notification system 164, and other applications 166. The memory 150 may also include a data memory 170, which may include maximized or minimized versions of notifications, timer thresholds, defined notification anchors, configuration data, settings, user options, or user preferences that can be provided to any element of the program memory 160 or the computing system 100.

[0047] Some implementations can operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations that can be used with this technology include, but are not limited to: XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices.

[0048] Figure 2AThis is a line drawing of a virtual reality head-mounted display (HMD) 200 according to some embodiments. The HMD 200 includes a front rigid body 205 and a strap 210. The front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a locator 225, and one or more computing units 230. The position sensors 220, IMU 215, and computing units 230 may be located inside the HMD 200 and may be invisible to the user. In various embodiments, the IMU 215, position sensors 220, and locator 225 may track the movement and position of the HMD 200 in the real world and virtual environments with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the locator 225 may emit infrared beams that produce light spots on real objects around the HMD 200. As another example, IMU 215 may include, for example, one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position sensors, force sensors, or orientation sensors; or combinations thereof. One or more cameras (not shown) integrated with HMD 200 can detect the light spot. Computational unit 230 in HMD 200 can use the detected light spot to infer the position and movement of HMD 200, as well as to identify the shape and location of real objects around HMD 200.

[0049] The electronic display 245 may be integrated with the front rigid body 205 and may provide image light to the user as instructed by the computing unit 230. In various embodiments, the electronic display 245 may be a single electronic display or multiple electronic displays (e.g., one display for each of the user's eyes). Examples of the electronic display 245 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) subpixels, a projector unit (e.g., a micro-LED, a laser, etc.), some other display, or some combination thereof.

[0050] In some implementations, HMD 200 may be coupled to a core processing unit such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors may monitor HMD 200 (e.g., by light emitted from HMD 200), and the PC may use the HMD in conjunction with outputs from IMU 215 and position sensor 220 to determine the position and movement of HMD 200.

[0051] Figure 2B This is a line drawing of a mixed reality head-mounted display (HMD) system 250, which includes a mixed reality head-mounted display (HMD) 252 and a core processing unit (CPU) 254. The HMD 252 and CPU 254 can communicate via a wireless connection (e.g., a 60 GHz link) as indicated by link 256. In other embodiments, the mixed reality HMD system 250 may include only a head-mounted display without external computing devices, or may include other wired or wireless connections between the HMD 252 and the CPU 254. The mixed reality HMD 252 includes a transparent display 258 and a frame 260. The frame 260 may accommodate various electronic components (not shown), such as light projectors (e.g., lasers, LEDs, etc.), cameras, eye-tracking sensors, microelectromechanical systems (MEMS) components, network components, etc.

[0052] A projector can be coupled to a pass-through display 258, for example, via optical elements, to display media to a user. The optical elements may include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc., for directing light from the projector to the user's eyes. Image data can be transmitted from the core processing unit 254 to the HMD 252 via link 256. A controller in the HMD 252 can convert the image data into light pulses from the projector, which can be transmitted as output light to the user's eyes via the optical elements. The output light can be mixed with light passing through the pass-through display 258, thereby allowing the output light to present virtual objects that appear as if they exist in the real world.

[0053] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow HMD system 250 to track itself, for example in 3DoF or 6DoF, track various parts of the user (e.g., hands, feet, head, or other body parts), render virtual objects as if they were stationary when HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0054] Figure 2C A controller 270 is shown. In some embodiments, a user can hold the controller 270 with one or both hands to interact with the artificial reality environment presented by the HMD 200 and / or HMD system 250. The controller 270 can communicate with the HMD directly or via an external device (e.g., core processing unit 254). The controller may have its own IMU unit, position sensors, and / or can emit light spots at greater distances. The HMD 200 or HMD system 250, external sensors, or sensors within the controller can track these controller light spots to determine the controller's position and / or orientation (e.g., tracking the controller in 3DoF or 6DoF). The computing unit 230 in the HMD 200 or the core processing unit 254 can combine the IMU output and position output to use this tracking to monitor the position and movement of the user's hands. The controller may also include various buttons (e.g., buttons 272A to 272F) and / or joysticks (e.g., joysticks 274A and 274B) that actuate to provide input and interact with objects.

[0055] In various implementations, HMD 200 or HMD system 250 may also include additional subsystems (e.g., eye-tracking unit, audio system, various network components, etc.) to monitor user interactions and indications of intent. For example, in some implementations, instead of a controller or other than a controller, one or more cameras included in or from an external camera in HMD 200 or HMD system 250 may monitor the position and posture of the user's hands to determine gestures and other hand and body movements. As another example, one or more light sources may illuminate one or both of the user's eyes, and HMD 200 or HMD system 250 may use an eye-facing camera to capture the reflection of that light (e.g., based on a set of reflections around the user's cornea) to determine eye position, thereby modeling the user's eyes and determining gaze direction.

[0056] Figure 3This is a block diagram illustrating an overview of an environment 300 in which some embodiments of the disclosed technology may operate. Environment 300 may include one or more client computing devices 305A to 305D, examples of which may include computing system 100. In some embodiments, some of the client computing devices (e.g., client computing device 305B) may be HMD 200 or HMD system 250. Client computing device 305 may operate in a network environment using a logical connection via network 330 to one or more remote computers (e.g., server computing devices).

[0057] In some implementations, server computing device 310 may be an edge server that receives multiple client requests and coordinates the fulfillment of these requests through other servers (e.g., server computing devices 320A to 320C). Server computing devices 310 and 320 may include computing systems, such as computing system 100. Although each server computing device 310 and 320 is logically presented as a single server, multiple server computing devices can be distributed computing environments comprising multiple computing devices located in the same physical location or in geographically different physical locations.

[0058] Client computing device 305, and server computing devices 310 and 320 can each act as a server or client for other servers / one or more client devices. Server computing device 310 can connect to database 315. Server computing devices 320A to 320C can each connect to their respective databases 325A to 325C. As discussed above, each server computing device 310 or 320 can correspond to a group of servers, and the individual servers among these servers can share a database or have their own database. Although databases 315 and 325 are logically presented as a single unit, both databases 315 and 325 can be distributed computing environments containing multiple computing devices, which can be located within their respective servers, or in the same physical location or in geographically different physical locations.

[0059] Network 330 can be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. Network 330 can be the Internet or some other public or private network. Client computing devices 305 can connect to network 330 via a network interface, for example, through wired or wireless communication. Although the connection between server computing device 310 and multiple server computing devices 320 is shown as a separate connection, these connections can be any type of LAN, WAN, wired network, or wireless network, including network 330 or a separate public or private network.

[0060] Figure 4 This diagram illustrates a block diagram of component 400, which, in some embodiments, can be used in systems employing the disclosed technology. Component 400 may be included in one device of computing system 100 or may be distributed across multiple devices of computing system 100. Component 400 includes hardware 410, middleware 420, and multiple dedicated components 430. As discussed above, systems implementing the disclosed technology can use various hardware, including processing unit 412, working memory 414, input and output devices 416 (e.g., camera, display, IMU unit, network connection, etc.), and storage memory 418. In various embodiments, storage memory 418 may be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, storage memory 418 may be one or more hard disk drives or flash drives accessible via a system bus, or it may be a cloud storage provider (e.g., located in database 315 or 325) or other network repository accessible via one or more communication networks. In various implementations, component 400 may be implemented in a client computing device (e.g., client computing device 305) or on a server computing device (e.g., server computing device 310 or 320).

[0061] Middleware 420 may include components that facilitate resource transfer between hardware 410 and multiple dedicated components 430. For example, middleware 420 may include an operating system, services, drivers, a basic input / output system (BIOS), controller circuitry, or other hardware or software systems.

[0062] Each dedicated component 430 may include software or hardware configured to perform operations for implementing a fixed interaction pattern for viewing notifications. Each dedicated component 430 may include a notification head-lock display module 434, a gaze detection model 436, a timer module 438, a notification world-lock display module 440, and a component and application programming interface (API) (e.g., interface 432) that can be used to provide a user interface, transmit data, and control the plurality of dedicated components. In some embodiments, component 400 may reside in a computing system distributed across multiple computing devices, or it may be an interface to a server-based application that executes one or more of the multiple dedicated components 430. Although the multiple dedicated components 430 are depicted as separate components, the multiple dedicated components 430 may be logical functional distinctions or other non-physical distinctions, and / or may be submodules or code blocks of one or more applications.

[0063] The notification head-lock display module 434 can format the received notification into a minimized version that is always located at a predefined position in the user's field of view (i.e., as head-locked). The following is in conjunction with... Figure 5 Box 502 in the document provides additional details on setting the format of minimized notifications and displaying them as a header lock.

[0064] The gaze detection model 436 can track a user's gaze to determine whether the user is currently looking at a version of the notification displayed by the notification head-lock display module 434 or the notification world-lock display module 440. In various embodiments, the gaze detection model 436 can use images or other sensor inputs from the device at I / O 416 to identify eye direction (e.g., based on light reflected around the cornea of ​​the user's eye). Based on this eye direction, the gaze detection model 436 can infer which part of the artificial reality device display the user is looking at.

[0065] The timer module 438 can determine, based on gazes tracked by the gaze detection model 436, whether a gaze timer has expired for the amount of time a user has looked at or looked away from the displayed notification. In one instance, the timer module 438 can determine whether the user has not looked at the notification for a first threshold amount of time, thereby canceling the notification. In another instance, the timer module 438 can determine whether the user has looked at the notification for a second threshold amount of time, thereby locking and / or maximizing the notification. In yet another instance, the timer module 438 can determine whether the user has looked away from the notification for a third threshold amount of time, thereby minimizing or canceling the notification. The following is combined with... Figure 5 Boxes 504, 508, and 512 provide additional details for examining the threshold time amount used to control the display of notifications.

[0066] The notification world-locked display module 440 can format received notifications as a maximized output, making the notification appear to be in the same geographic location regardless of user movement (i.e., world-locked). In various implementations, the notification can be 3DoF or 6DoF world-locked, allowing the user to move their head to bring the notification into their field of view (or, in some cases, initially automatically move the notification to the center of the user's field of view). In the 3DoF world-locked version, the user's viewing distance and angle to the notification remain constant as the user moves closer to / away from the notification or around the notification. In the 6DoF world-locked version, the user's viewing distance and angle to the notification are adjusted as the user moves closer to / away from the notification or around the notification. The following is combined with... Figure 5 Box 510 in the document provides additional details on setting the format for maximized notifications and displaying them as world locked.

[0067] Those skilled in the art will understand that the above description Figures 1 to 4 The components shown in the flowcharts shown, and in the various flowcharts discussed below, can be modified in various ways. For example, the order of the logic can be rearranged, multiple sub-steps can be executed in parallel, the logic shown can be omitted, and other logic can be included, etc. In some embodiments, one or more of the components described above can perform one or more of the processes described below.

[0068] Figure 5 This is a flowchart illustrating a process 500 for viewing a fixed interaction pattern for notification in some embodiments of the present technology. In some embodiments, process 500 may be executed by an artificial reality system, for example, as part of the execution of an operating system, a shell application controlling the artificial reality environment, or another application executed by the artificial reality system to control the delivery of notifications. In some embodiments, process 500 may be executed in response to receiving a notification or other content item for display in the artificial reality environment.

[0069] At box 502, process 500 may display a head-lock notification at a predefined location within the field of view of the artificial reality device. In various implementations, the predefined location may be at the edge or corner of the field of view. In some cases, the head-lock notification may be a minimized version of the notification, which may be a predefined minimized notification size or a specific shape, and may provide indications of the notification, such as specifying the user and / or application source of the notification, a preview of the notification content, the time associated with receiving the notification, controls for responding to or canceling the notification, etc. As described above, a head-lock notification may include maintaining the notification in the same location within the user's field of view as the user moves around in the artificial reality environment.

[0070] At box 504, process 500 may determine whether the display timer for the notification has expired. At box 508, if the notification is not being viewed by the user, the display timer may be a threshold amount of time (e.g., 3 seconds, 5 seconds, or 10 seconds) for displaying the head-locked notification before the notification is canceled. If the display timer has expired, process 500 may proceed to box 506, where the head-locked version of the notification may be removed from the field of view before process 500 ends. If the display timer has not expired, process 500 may proceed to box 508.

[0071] At box 508, process 500 can determine whether the tracked user's gaze is directed at a head-lock notification in the field of view. As described above, the artificial reality device can model the user's eye position. Based on this eye position, a line connecting the center of the user's cornea to the back of the user's retina can be determined to determine where the user is looking on the display. At box 508, process 500 can determine whether the endpoint of this line on the display corresponds to a threshold time (e.g., one or two seconds) of the head-lock notification. If not, process 500 can return to box 504. If yes, the process can continue to box 510.

[0072] At box 510, process 500 can convert the notification to world-locked. In some implementations, at box 510, process 500 can also reformat the notification, for example, displaying a maximized version, which may include additional messages from the notification, additional graphics or 3D models included in the notification, additional controls or input fields for responding to the notification, etc. In some cases, process 500 can automatically move the notification to the center of the user's field of view. When a user's gaze is detected pointing at the notification, process 500 can make the notification 3DoF world-locked or 6DoF world-locked, thereby allowing the user to move their head to move the notification into their field of view (if the notification does not automatically move to the center of the user's field of view). In the 3DoF world-locked version, the user's viewing distance and angle to the notification remain constant as the user moves closer to / away from the notification or around the notification. In the 6DoF world-locked version, the user's viewing distance and angle to the notification are adjusted as the user moves closer to / away from the notification or around the notification. Therefore, in both the 3DoF and 6DoF world-locked versions, users can move the notification within their field of view (e.g., by pointing their head towards the world-locked position of the notification to move it closer to the center of their field of view). In the 6DoF version, users can also: A) view the notification from different angles as they move around its world-locked position; and / or B) make the notification appear larger or smaller by moving closer to or further away from its world-locked position. Users can also interact with various controls provided with the world-locked version of the notification, such as message reply controls, notification cancellation controls, controls to open the corresponding application, etc.

[0073] Process 500 may keep the notification in a world-locked state until, at box 512, process 500 determines that the tracked user's gaze has left the notification for a threshold amount of time (e.g., one or two seconds). When process 500 makes this determination, process 500 may proceed to box 514, where, in some embodiments, process 500 may remove the notification from the field of view; and in other embodiments, process 500 may return to box 502 to minimize the notification, head-lock the notification, and restore the notification to a predefined position. Process 500 may terminate after box 514 unless it returns to box 502.

[0074] Figures 6A to 6G This is a conceptual diagram illustrating example 600 of a notification presented in a fixed interaction mode. Example 600 exists. Figure 6AIn this context, the user is within environment 602. The user's gaze direction 604 is being tracked by the AI ​​Reality Notification System. Rectangle 606 indicates the user's entry into the field of view of environment 602. A notification related to a received text message has entered the AI ​​Reality Notification System, which has formatted the notification as a minimized version 608 indicating the sender of the text message and a preview of its content. The minimized version 608 is head-locked to be displayed in the upper left corner of the field of view 606. (Continue) Figure 6B In example 600, the tracked user's gaze direction 604 is identified as pointing towards the minimized version 608 of the notification. Next, in... Figure 6C In Example 600, the artificial reality notification system, in response to the tracked user's gaze direction 604 having lingered on the notification for 1.5 seconds, replaces the minimized version 608 with a maximized version 610 of the notification. The maximized version 610 includes a full-text message and controls for replying to that text message. The maximized version 610 is locked to the position it was in when maximized, in this case, above table 612. (Continuing...) Figure 6D Example 600 also illustrates a 6DoF world-locking implementation, wherein the user has moved close to the world-locked position of the maximized version 610 of the notification, thereby making the maximized version 610 appear larger. Figure 6E and Figure 6F The notification shows the maximized version 610 of the 3DoF world lock ( Figure 6E ) and 6DoF World Lock ( Figure 6F (An alternative version.) Figure 6E and Figure 6F In this version, users have moved away from the maximized notification feature of version 610. Figure 6E The maximized version 610 of the notification maintains its size under 3DoF world-locked conditions as the user moves backward (compared to...). Figure 6D (Compared to). Figure 6F The maximized version 610 of the notification is set to 6DoF world-locked. Moving the user backward reduces the size of the maximized version 610 of the notification (compared to...). Figure 6D (Compared). Finally, in Figure 6G In Example 600, the user has moved to their original position, and the direction of the user's gaze 604 being tracked is identified as being away from the maximized version 610 of the notification for two seconds. In response, the artificial reality notification system returns the notification to the minimized version 608, with the head locked again in the upper left corner of the field of view 606.

[0075] Figure 7A and Figure 7BThis is a conceptual diagram illustrating Example 700, which minimizes a virtual object in response to reaching the maximum field of view. Example 700 begins with... Figure 7A , Figure 7A The display shows the artificial reality environment 702, the user's gaze direction 704, the user's field of view 706, a minimized notification 708, a first virtual object 710, and a second virtual object 712. Figure 7B In the process, the user's gaze direction 704 has moved to the minimized notification 708, causing the minimized notification 708 to become the maximized notification 714 (as described above). Because the maximized notification 714, the first virtual object 710, and the second virtual object 712 occupy more than the maximum amount of the user's field of view (35% in this case), the artificial reality notification system minimizes the virtual objects 710 and 712 to icon versions 716 and 718, and moves icon versions 716 and 718 to the corner of the user's field of view 706.

[0076] In this specification, references to "implementation" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described with respect to that embodiment is included in at least one embodiment of this disclosure. The appearance of these phrases in different places in the specification does not necessarily refer to the same embodiment, nor does it necessarily refer to a separate or alternative embodiment that is mutually exclusive with other embodiments. Furthermore, various features that may be present in some embodiments but not in other embodiments are described. Similarly, various requirements that may be requirements for some embodiments but not for other embodiments are described.

[0077] As used herein, above the threshold means that the value of the comparison item is higher than any other specified value, the comparison item is among a specified number of items with a maximum value, or the value of the comparison item is within a specified highest percentage value. As used herein, below the threshold means that the value of the comparison item is lower than any other specified value, the comparison item is among a specified number of items with a minimum value, or the value of the comparison item is within a specified lowest percentage value. As used herein, within the threshold means that the value of the comparison item is between two other specified values, the comparison item is among a specified number of items in the middle, or the value of the comparison item is within a specified percentage range in the middle. When not otherwise defined, relative terms such as high or unimportant can be understood as assigning a value and determining how that value is compared to an established threshold. For example, the phrase "select fast connections" can be understood as meaning selecting connections with assigned values ​​corresponding to their speeds above the threshold.

[0078] As used in this article, the word “or” refers to any possible permutation of a set of items. For example, the phrase “A, B, or C” refers to at least one of A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B, and C; or multiples of any of the items such as A and A; B, B, and C; A, A, B, C, and C; etc.

[0079] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The specific features and actions described above are disclosed as exemplary forms of implementing the appended claims. Therefore, the embodiments and implementations described are not limited except by the appended claims.

[0080] All patents, patent applications, and other references mentioned above are incorporated herein by reference. Modifications may be made to various aspects to provide further implementations using the systems, functions, and concepts of the aforementioned references. In the event of any conflict between statements or subject matter in the incorporated references and those in this application, this application shall prevail.

Claims

1. A method for implementing a fixed interaction pattern for human-viewed notifications, the method comprising: The minimized version of the notification is displayed with the head locked at a predefined position within the field of view of the artificial reality device; Determine that the tracked user's gaze is directed towards a head-locked minimized version of the notification, and in response, convert the notification to a maximized and world-locked version, wherein when the user turns their head, the maximized and world-locked versions of the notification appear to have no change in position to the user; and Determine that the tracked user's gaze has been away from the maximized and world-locked notification for a threshold amount of time, and in response, cancel the notification or return the notification to the minimized and head-locked version at the predefined location. The method further includes: In response to converting the notification to maximized and world-locked, it is determined that the maximized and world-locked notification and one or more other virtual objects in the field of view of the artificial reality device occupy an amount exceeding a threshold of the field of view of the artificial reality device; and In response to the notification that the virtual object is maximized and world locked, and the one or more other virtual objects in the field of view of the artificial reality device occupy more than the threshold amount of the field of view of the artificial reality device: convert the one or more other virtual objects to a minimized version or remove the one or more virtual objects from the artificial reality environment; and move the indication of the one or more virtual objects to a separate interface.

2. The method according to claim 1, wherein, The minimized version of the notification includes one or more of the following: The notification is an instruction from the user and / or application source; A preview of the notification content; The time associated with receiving the notification; Controls used to respond to or cancel the notification.

3. The method according to claim 1, wherein, The determination that the tracked user's gaze is directed at the minimized version of the head-lock notification is based on the determination that the display timer for displaying the minimized version of the head-lock notification has not yet expired.

4. The method according to claim 1, wherein, The gaze of the user being tracked is tracked in the following way: Model the user's eye position; Based on the user's eye position, a line is defined connecting the center of the user's cornea to the back of the user's retina; as well as Determine the position of the endpoint of the line in the display.

5. The method according to claim 1, wherein, The maximized version of the notification includes one or more of the following: Message from the notification; Includes graphics or 3D models in the notification; Controls or input fields used to respond to the notification.

6. The method according to claim 1, wherein, Converting the notification to maximized and world-locked includes converting the notification to maximized and 3DoF world-locked such that the user's viewing distance and angle to the notification remain unchanged as the user moves closer to and further away from the notification or moves around the notification.

7. The method according to claim 1, wherein, Converting the notification to maximized and world-locked includes converting the notification to maximized and 6DoF world-locked such that the user's viewing distance and angle to the notification are adjusted as the user moves closer to and further away from the notification or around the notification.

8. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for implementing a notification to view a fixed interactive mode, the process comprising the method according to any one of claims 1 to 7.

9. A computing system for implementing a fixed interaction pattern for viewing notifications, the computing system comprising: One or more processors; as well as One or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process including the method according to any one of claims 1 to 7.

Images