Distributed Processing in a Computer-Generated Reality System

Abstract

The present invention discloses technologies related to display devices. In some embodiments, a display device includes a display system configured to display three-dimensional content to a user. The display device is configured to discover, via a network interface, one or more computing nodes capable of operating to facilitate rendering of the three-dimensional content, and receive information identifying capabilities of the one or more computing nodes to facilitate the rendering. Based on the received information, the display device evaluates a set of tasks to identify one or more tasks in the set to be offloaded to the one or more computing nodes to facilitate the rendering, and allocates the identified one or more tasks to the one or more computing nodes via the network interface for processing by the one or more computing nodes.

Description

Background Art Technical Field

[0001] The present disclosure generally relates to computing systems, and more particularly to computer-generated reality systems.

[0002] Description of Related Technologies

[0003] Augmented reality (AR), mixed reality (MR), virtual reality (VR), and cross reality (XR) can allow a user to interact with an immersive environment having artificial elements such that the user can feel a part of the environment. For example, a VR system can display a stereoscopic scene to a user to create an illusion of depth, and a computer can adjust the scene content in real time to provide an illusion that the user is moving within the scene. When a user views an image through a VR system, the user can thus feel as if they are moving within the scene from a first-person perspective. Similarly, an MR system can combine computer-generated virtual content with real-world images or real-world footage to enhance the view of the world that the user sees, or alternatively combine virtual representations of real-world objects with a view of a three-dimensional virtual world. Thus, the simulated environment of virtual reality and / or the mixed environment of mixed reality can provide an interactive user experience for a variety of applications. Brief Description of the Drawings

[0004] Figure 1 is a block diagram showing an example of a system for distributing the processing of content being displayed on a display device among multiple computing nodes.

[0005] Figure 2 is a block diagram showing an example of a distribution engine capable of operating to distribute tasks between a computing node and a display device.

[0006] Figure 3 is a block diagram showing an example of a discovery engine that may be included in the distribution engine.

[0007] Figures 4A to 4C is a block diagram showing an example of a task graph that may be used by the distribution engine.

[0008] Figure 5 is a block diagram showing an example of components included in a display device and a computing node.

[0009] Figures 6A to 6D is an illustration showing different examples of processing content being displayed.

[0010] Figures 7A to 7D is a flowchart showing an example of a method performed by components of a distribution system.

[0011] Figure 8 is a block diagram showing an example of a distribution engine evaluating the capabilities of a computing node before offloading a task to the computing node.

[0012] Figure 9 is a flowchart showing an example of a method for evaluating the capabilities of a computing node.

[0013] Figure 10 is a block diagram of an example of a personalization engine.

[0014] This disclosure includes references to "one embodiment" or "an embodiment". The appearances of the phrase "in one embodiment" or "in an embodiment" do not necessarily refer to the same embodiment. Specific features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

[0015] Within this disclosure, different entities (which may be variously referred to as "units", "circuits", other components, etc.) may be described or claimed as "configured to" perform one or more tasks or operations. This expression—the [entity] [configured to [perform one or more tasks]]—is used herein to refer to a structure (i.e., a physical thing, such as an electronic circuit). More specifically, this expression is used to indicate that this structure is arranged to perform one or more tasks during operation. A structure may be said to be "configured to" perform a certain task even if the structure is not currently being operated. A "display system configured to display three-dimensional content to a user" is intended to cover, for example, a liquid crystal display (LCD) that performs this function during operation, even if the LCD under consideration is not currently in use (e.g., power is not connected to it). Thus, an entity described or stated as "configured to" perform a certain task refers to a physical thing for implementing that task, such as a device, a circuit, a memory storing executable program instructions, and so on. This phrase is not used herein to refer to intangible things. Thus, the "configured to" construct is not used herein to refer to software entities, such as application programming interfaces (APIs).

[0016] The term "configured to" is not intended to mean "capable of being configured to". For example, an unprogrammed FPGA would not be considered "configured to" perform a particular function, although it may be "capable of being configured to" perform that function and can be "configured to" perform that function after programming.

[0017] The recitation in the appended claims of the structure "configured to" perform one or more tasks is expressly intended not to invoke 35 U.S.C. § 112(f) for that claim element. Thus, no claim in the present application is intended to be construed as having a means-plus-function element. If the applicant wishes to invoke section 112(f) during the application process, it will use the "means for [performing a function]" construct to recite the claim element.

[0018] As used herein, the terms "first", "second", etc. serve as labels for the nouns that follow them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless explicitly stated. For example, in a processor having eight processing cores, the terms "first" processing core and "second" processing core can be used to refer to any two of the eight processing cores. In other words, the "first" processing core and the "second" processing core are not limited to, for example, processing core 0 and processing core 1.

[0019] As used herein, the term "based on" is used to describe one or more factors that affect a determination. This term does not exclude the possibility that there may be additional factors that affect the determination. That is, the determination may be based solely on the specified factors or on the specified factors and other unspecified factors. Consider the phrase "determine A based on B". This phrase specifies that B is a factor used to determine A or that B affects the determination of A. This phrase does not exclude the possibility that the determination of A may also be based on some other factor such as C. This phrase is also intended to cover embodiments where A is determined solely based on B. As used herein, the phrase "based on" is thus synonymous with the phrase "at least partially based on".

[0020] As used herein, a "physical environment" refers to the physical world that people can sense and / or interact with without the assistance of an electronic system. A physical environment such as a physical park includes physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0021] As used herein, a "computer-generated reality (CGR) environment" refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In CGR, a subset of a person's physical movements or representations thereof are tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner consistent with at least one physical law. For example, a CGR system can detect a person's head rotation, and in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds would change in the physical environment. In some cases (e.g., for accessibility reasons), the adjustment of the characteristics of virtual objects in the CGR environment can be made in response to a representation of a physical movement (e.g., a voice command).

[0022] A person can use any of their senses to sense and / or interact with a CGR object, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of point audio sources in 3D space. As another example, the audio object can enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person can sense and / or interact only with audio objects.

[0023] Examples of CGR include virtual reality and mixed reality.

[0024] As used herein, a virtual reality (VR) environment refers to a simulated environment that is designed to be completely computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with the virtual objects in a VR environment by way of a simulation of the person's presence within the computer-generated environment and / or by way of a simulation of a subgroup of the person's physical movements within the computer-generated environment.

[0025] As used herein, a mixed reality (MR) environment refers to a simulated environment that is designed to incorporate sensory input or a representation thereof from the physical environment in addition to including computer-generated sensory input (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any condition between a completely physical environment at one end and a virtual reality environment at the other end, but excluding these two ends.

[0026] In some MR environments, the computer-generated sensory input can respond to changes in the sensory input from the physical environment. Additionally, some electronic systems for presenting an MR environment can track the position and / or orientation relative to the physical environment so that virtual objects can interact with real objects (i.e., physical items from the physical environment or their representations). For example, the system can cause movement such that a virtual tree appears stationary relative to the physical ground.

[0027] Examples of mixed reality include augmented reality and augmented virtuality.

[0028] As used herein, an augmented reality (AR) environment is a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation of a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or translucent display such that the person using the system perceives the virtual objects superimposed on the physical environment. Alternatively, the system can have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with virtual objects and presents the combination on the opaque display. The person uses the system to indirectly view the physical environment via the images or videos of the physical environment and perceives the virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "passthrough video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Further alternatively, the system can have a projection system that projects virtual objects into the physical environment, such as as a hologram or on a physical surface, such that the person using the system perceives the virtual objects superimposed on the physical environment.

[0029] An augmented reality environment is also a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a passthrough video, the system can transform one or more sensor images to impose an alternative perspective (e.g., viewpoint) that is different from the perspective captured by the imaging sensor. As another example, a representation of a physical environment can be transformed by graphically modifying (e.g., magnifying) portions thereof such that the modified portions can be a representative but not a true version of the originally captured image. As yet another example, a representation of a physical environment can be transformed by graphically removing portions thereof or blurring portions thereof.

[0030] An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs can be representations of one or more characteristics of the physical environment. For example, an AV park can have virtual trees and virtual buildings, but a person's face is a realistic reproduction from an image of a physical person. As another example, a virtual object can adopt the shape or color of a physical item imaged by one or more imaging sensors. As yet another example, a virtual object can adopt a shadow that conforms to the positioning of the sun in the physical environment. Detailed Description

[0031] Providing an excellent CGR experience (such as an AR, MR, VR, or XR experience) may require using a large amount of hardware and software resources to provide dynamic and vivid content. However, the resources available for providing such content operate within limited constraints. For example, a display device may have limited processing power, operate using battery power, and have a network connection with limited bandwidth. The management of these resources can be particularly important for a CGR system because problems such as jitter and latency can quickly ruin the experience. For example, if there is a significant delay between an event occurring at one user's display device and an event occurring at another user's display device, it may be difficult for the two users to interact with each other.

[0032] This disclosure describes embodiments in which a display device attempts to discover computing devices available to assist the display device and offloads tasks to these computing devices to extend the amount of available computing resources for delivering content. As will be described in more detail below, in various embodiments, the display device may collect information identifying the capabilities of one or more computing devices to assist the display device. For example, the display device may determine that the user has a tablet computer and a laptop computer nearby that are currently unused and both have a graphics processing unit (GPU). Based on this discovery, the display device may evaluate a set of tasks associated with the content being displayed and may offload one or more tasks to the discovered devices. In various embodiments, the display device may continue to collect computing capability information from available computing devices because operating conditions may change over time. For example, if the display device is wirelessly communicating with a tablet computer and the user operating the display device walks out of the room, the display device may detect this change and reallocate tasks accordingly. When evaluating which tasks to offload, the display device may consider many factors related to computing resources, energy budget, quality of service, network bandwidth, security, etc. in an effort to meet various goals related to, for example, accuracy, precision, fidelity, processing time, power consumption, privacy considerations, etc. Dynamically discovering computing resources and reallocating tasks in real time based on these factors can provide a much richer experience for the user than when the user is limited to the resources of the display device and, for example, a desktop computer connected to the display device.

[0033] Turning now to Figure 1 , the figure shows a block diagram of an allocation system 10. In the illustrated embodiment, the allocation system 10 includes a display device 100 that includes an environmental sensor 110, a user sensor 120, and an allocation engine 150. As shown, the system 10 may also include one or more computing nodes 140A - 140F. In some embodiments, the system 10 may be implemented differently than shown. For example, multiple display devices 100 may be used, more (or fewer) computing nodes 140 may be used, etc.

[0034] In various embodiments, the display device 100 is a computing device configured to display content such as a three-dimensional view 102 to a user and, in some embodiments, provide audio content 104. In the illustrated embodiment, the display device is depicted as a telephone; however, the display device can be any suitable device, such as a tablet computer, a television, a laptop computer, a workstation, etc. In some embodiments, the display device 100 is a head-mounted display (HMD) configured to be worn on the head and display content to a user. For example, the display device 100 can be a head-mounted earphone, a helmet, goggles, glasses, a telephone inserted into a housing, etc., worn by a user. As will be described with respect to Figure 5 as set forth below, the display device 100 can include a near-eye display system that displays a left image and a right image on a screen in front of the user's eyes to present a 3D view 102 to the user. In other embodiments, the device 100 can include a projection-based system, a vehicle windshield with integrated display capabilities, a window with integrated display capabilities, a display formed as a lens (e.g., similar to a contact lens) designed to be placed on a person's eye, earphones / headsets, a speaker array, an input system (e.g., a wearable or handheld controller with or without haptic feedback), etc. The display device 100 can be used to provide any one of a variety of user experiences to a user. In various embodiments, these experiences can utilize AR, MR, VR, or XR environments. For example, the display device 100 can provide collaborative and creative experiences that can allow users to work together in an AR environment to create content. The display device 100 can provide a co-presence experience in which multiple users can be individually connected in an MR environment. As used herein, the term "co-presence" refers to a shared CGR experience in which two people can interact with each other using their respective devices. The display device 100 can provide a gaming experience in which a user performs activities in a VR environment. In various embodiments, the display device 100 can provide other non-CGR experiences. For example, a user can operate the display device 100 to stream media content that can be displayed in three dimensions or two dimensions, such as music or a movie. To facilitate the delivery of these various experiences, the display device 100 can incorporate the use of environmental sensors 110 and user sensors 120.

[0035] In various embodiments, the environmental sensor 110 is a sensor configured to collect various information about the environment in which the user operates the display device 100. In some embodiments, the environmental sensor 110 may include one or more visible light cameras that capture video information of the user's environment. This information may be used, for example, to provide a virtual view of the real environment, detect objects and surfaces in the environment, provide depth information of the objects and surfaces in the real environment, provide the position (e.g., location and orientation) and movement (e.g., direction and speed) information of the user in the real environment, and so on. In some embodiments, the display device 100 may include a left camera and a right camera located at positions on the front surface of the display device 100, and in embodiments where the display device 100 is an HMD, these positions are substantially in front of each of the user's eyes. In other embodiments, more or fewer cameras may be used in the display device 100 and may be located at other positions. In some embodiments, the environmental sensor 110 may include one or more world mapping sensors (e.g., an infrared (IR) sensor with an IR illumination source or a light detection and ranging (LIDAR) transmitter and receiver / detector) that capture depth information or range information of the objects and surfaces in the user's environment, for example. This range information may be used, for example, in combination with the frames captured by the cameras to detect and identify objects and surfaces in the real-world environment, and to determine the position, distance, and speed of the objects and surfaces relative to the user's current position and movement. This range information may also be used to position the virtual representation of the real-world objects to be synthesized into the virtual environment at the correct depth. In some embodiments, this range information may be used to detect the possibility of collisions with real-world objects and surfaces to redirect the user's walking. In some embodiments, the environmental sensor 110 may include one or more light sensors (e.g., on the front and top of the display device 100) that capture lighting information (e.g., direction, color, and intensity) in the user's physical environment. For example, this information may be used to change the brightness and / or color of the display system in the display device 100.

[0036] In various embodiments, user sensor 120 is a sensor configured to collect various information about a user operating display device 100. In some embodiments where display device 100 is an HMD, user sensor 120 may include one or more head pose sensors (e.g., IR or RGB cameras) that can capture information about the position and / or movement of the user and / or the user's head. The information collected by the head pose sensors can be used, for example, to determine how to render and display views of the virtual environment and the content within the views. For example, different views of the environment can be rendered at least in part based on the position of the user's head, regardless of whether the user is currently walking through the environment, etc. As another example, enhanced position and / or movement information can be used to composite virtual content into a scene at a fixed position relative to a background view of the environment. In some embodiments, there may be two head pose sensors located on the front or top surface of display device 100; however, in various embodiments, more (or fewer) head pose sensors can be used, and these head pose sensors can be positioned at other locations. In some embodiments, user sensor 120 may include one or more eye tracking sensors (e.g., IR cameras with IR illumination sources) that can be used to track the position and movement of the user's eyes. In some embodiments, the information collected by the eye tracking sensors can be used to adjust the rendering of the image to be displayed and / or to adjust the display of the image through the display system of display device 100 based on the direction and angle of the user's eye gaze. In some embodiments, the information collected by the eye tracking sensors can be used to match the direction of the eyes of the user's avatar with the direction of the user's eyes. In some embodiments, the brightness of the displayed image can be adjusted based on the user pupil dilation determined by the eye tracking sensors. In some embodiments, user sensor 120 may include one or more eyebrow sensors (e.g., IR cameras with IR illumination) that track the expression of the user's eyebrows / forehead. In some embodiments, user sensor 120 may include one or more jaw tracking sensors (e.g., IR cameras with IR illumination) that track the expression of the user's mouth / jaw. For example, in some embodiments, the expressions of the forehead, mouth, jaw, and eyes captured by sensor 120 can be used to simulate the expressions on the user's avatar in a co-presence experience and / or to selectively render and composite virtual content for viewing at least in part based on the user's reaction to the content displayed on display device 100. In some embodiments, user sensor 120 may include one or more hand sensors (e.g., IR cameras with IR illumination) that track the position, movement, and gesture of the user's hand, fingers, and / or arm. For example, in some embodiments, the detected position, movement, and gesture of the user's hand, fingers, and / or arm can be used to simulate the movement of the user's avatar's hand, fingers, and / or arm in a co-presence experience.As another example, the detected hand and finger poses of a user can be used to determine the user's interaction with virtual content in a virtual space, including but not limited to poses for manipulating virtual objects, poses for interacting with virtual user interface elements displayed in the virtual space, etc.

[0037] In various embodiments, the display device 100 includes one or more network interfaces for establishing a network connection with the computing node 140. Any suitable network communication protocol can be used to establish this network connection, including wireless protocols such as Long Term Evolution TM etc., or wired protocols such as Ethernet, Fibre Channel, Universal Serial Bus TM (USB), etc. In some embodiments, the connection can be implemented according to a proprietary wireless communication technology (e.g., 60 gigahertz (GHz) wireless technology) that provides a highly directional wireless link between the display device 100 and one or more computing nodes 140. In some embodiments, the display device 100 is configured to select between different available network interfaces based on the connectivity of the interface and the specific user experience delivered by the display device 100. For example, if a particular user experience requires a large amount of bandwidth, the display device 100 can select a radio component that supports the proprietary wireless technology when wirelessly communicating with the high-performance computing device 140E. However, if the user is only streaming a movie from the laptop 140B, then it may be sufficient and selected by the display device 100. In some embodiments, in cases such as limited bandwidth, for example, the display device 100 can use compression techniques to communicate over the network connection.

[0038] In various embodiments, the computing node 140 is a node that can be used to assist in generating the content used by the display device 100, such as facilitating the rendering of the 3D view 102. The computing node 140 can be or can include any type of computing system or computing device. As Figure 1As shown, the computing nodes 140 can generally be classified into primary, secondary, and tertiary computing grids 142. In the illustrated embodiment, the primary computing grid 142A includes the computing nodes 140 belonging to the user of the display device 100. These computing nodes 140 may provide less computing power than the computing nodes 140 in the other grids 142, but may be readily available to the user of the display device 100. For example, a user operating the display device 100 at home may be able to utilize the computing power of his or her phone, watch 140A, laptop computer 140B, and / or tablet computer 140C, which may be in the same room or a nearby room. Other examples of such computing nodes 140 may include wireless speakers, set-top boxes, gaming consoles, game systems, Internet of Things (IoT) devices, home network devices, etc. In the illustrated embodiment, the secondary computing grid 142B includes nearby computing nodes 140 that can provide greater computing power at a higher cost and, in some cases, may be shared by multiple display devices 100. For example, a user operating the display device 100 may enter a retail store with a workstation 140D and / or a high-performance computing (HPC) device 140E and may be able to receive assistance from such nodes 140 to interact with the store products in the AR environment. In the illustrated embodiment, the tertiary computing grid 142C includes high-performance computing nodes 140 that can be made available to the user through cloud-based services. For example, a server cluster 140F may be established on the basis of a server farm located far from the display device 100 and may implement one or more services for the display device 100, such as rendering three-dimensional content, streaming media, storing the rendered content, etc. In such an embodiment, the computing nodes 140 may also include logical computing nodes, such as virtual machines, containers, etc., that can be provided by the server cluster 140F.

[0039] Thus, computing nodes 140 can vary significantly in their ability to assist display device 100. Some computing nodes 140, such as watch 140A, may have limited processing power and be power-constrained, such as limited to a one-watt battery power source, while other nodes, such as server cluster 140F, may have almost unlimited processing power and have few power limitations, such as being able to deliver several kilowatts of computing. In various embodiments, computing nodes 140 can vary in their ability to perform specific tasks. For example, workstation 140D may execute specialized software, such as a VR application that can provide specialized content. HPC 140E may include designated hardware, such as multiple high-performance central processing units (CPUs), graphics processing units (GPUs), image signal processors (ISPs), circuitry supporting neural network engines, security hardware (e.g., security elements, hardware security modules, security processors, etc.), and so on. In some embodiments, computing nodes 140 can vary in their ability to securely perform operations. For example, tablet 140C may include a security element configured to securely store and operate on confidential data, while workstation 140D may be untrusted and accessible via an unencrypted wireless network connection. In various embodiments, the ability of computing nodes 140 to assist display device 100 can be dynamic. For example, when the user operating display device 100 walks into another room, display device 100 may lose connectivity with tablet 140C. Initially idle, laptop 140B may provide some assistance to display device 100, but after someone else starts using laptop 140B for some other purpose, it provides less or no assistance.

[0040] In various embodiments, allocation engine 150 may execute to discover computing nodes 140 and determine whether to offload task 154 to the discovered computing nodes 140. In the illustrated embodiment, allocation engine 150 makes this determination based on computing power information 152 and the specific task 154 to be offloaded. Computing power information 152 can generally refer to any suitable information that engine 150 can use to evaluate whether task 154 should (or should not) be offloaded to a particular computing node 140. As will be described below with respect to Figure 3More specifically described, the computing power information 152 may include information about resource utilization, power constraints of the computing node 140, specific hardware or software present at the computing node 140, the ability to perform specialized tasks 154, etc. Since the capabilities of the computing node 140 may change over time, in some embodiments, the allocation engine 150 may continuously receive the computing power information 152 in real time while the display device 100 is displaying content. If a particular computing node 140, for example, refuses to accept a task 154 or leaves the grid 142, the allocation engine 150 may determine to dynamically re - allocate the task 154 between the computing node 140 and the display device 100.

[0041] The allocation engine 150 may evaluate any of the various tasks 154 for possible offloading. These tasks 154 may be related to the rendering of the content being displayed on the display device 100, such as performing mesh assembly, shading, texturing, transformation, lighting, clipping, rasterization, etc. These tasks 154 may also be related to the rendering in that they affect the content being displayed. For example, as will be discussed below in conjunction with Figure 4A what is discussed, the display device 100 may deliver an AR experience that uses an object classifier to identify specific objects captured in video frames collected by the camera sensor 110. The allocation engine 150 may offload one or more tasks 154 related to the classifier to one or more computing nodes 140 rather than fully implementing the classifier at the display device 100. Then, the display device 100 may indicate the result of the object classification in the 3D view 102. The tasks 154 may also be related to other content provided by the display device 100, such as audio or tactile content provided to the user. For example, as will be discussed below in conjunction with Figure 4B what is discussed, one or more tasks related to speech recognition may be offloaded to the computing node 140. The tasks 154 may also be related to other operations, such as storing the rendered content for subsequent retrieval by the same display device 100 or other devices such as a friend's phone. Thus, the tasks 154 performed in the allocation system 10 may be consumed by algorithms / components that produce visual elements (feed the display), auditory elements (e.g., room acoustics), and interactions (e.g., gestures, speech) to meet the experience goals. As will be discussed below with respect to Figure 2As discussed, the allocation engine 150 can evaluate the computing power information 152 in combination with a graphical structure defining a set of tasks to be executed, the interdependencies of these tasks, and their corresponding constraints (e.g., the perceptual latency and thresholds of the visual, audio, and interaction elements of the experience), as well as one or more user-specific quality of service (QoS) parameters. In various embodiments, the engine 150 provides this information to a cost function that attempts to minimize, for example, power consumption and latency while ensuring the delivery of an optimal user experience. In some embodiments, the allocation engine 150 may also handle the collection of the results generated by the nodes 140 executing the tasks 154 and route the results to the appropriate consuming hardware and / or software in the display device 100.

[0042] Although shown within the display device 100, the allocation engine 150 may reside elsewhere and, in some embodiments, in multiple locations. For example, a first instance of the allocation engine 150 may reside at the display device 100, and a second instance of the allocation engine 150 may reside at the laptop computer 140B. In such an embodiment, the allocation engine 150 at the laptop computer 140B may collect instances of the computing power information 152 from one or more other computing nodes 140 (such as Figure 1 the illustrated tablet computer 140C), and provide a set of tasks 154 that are offloaded from the display device 100 to the other computing nodes 140. In some embodiments, the allocation engine 150 at the laptop computer 140B may forward the received computing power information 152 (or a combination of this computing power information with the computing power information sent by the laptop computer 140B) to the allocation engine 150 at the display device 100, which may determine what to allocate to the other computing nodes 140. In some embodiments, the allocation engine 150 at the laptop computer 140B may alternatively make a determination locally as to what should be offloaded to the other nodes 140.

[0043] Turning now to Figure 2 FIG., this figure illustrates a block diagram of the allocation engine 150. In the illustrated embodiment, the allocation engine 150 includes a discovery engine 210, a graphics selector 220, a personalization engine 230, a constraint analyzer 240, and a task publisher 250. In other embodiments, the engine 210 may be implemented differently than shown.

[0044] In various embodiments, the discovery engine 210 processes the discovery of the available computing nodes 140 by exchanging discovery information 202. The discovery engine 210 may use suitable techniques to discover the computing nodes 140. For example, the engine 210 may employ protocols such as the Simple Service Discovery Protocol (SSDP), Aware, zero-configuration networking (zeroconf), etc. As will be combined with Figure 3 As described, the discovery engine 210 may issue broadcast requests to the compute nodes 140 and / or receive broadcast notifications from the compute nodes 140. In some embodiments, the discovery engine 210 also processes the collection of computing capacity information 152 received from the compute nodes 140. In the illustrated embodiment, the engine 210 aggregates this information 152 into a dynamic constraint vector 212, which is provided to the constraint analyzer 240. Also as will be combined with Figure 3 As discussed, the constraint vector 212 may include multiple factors related to the computing capacity of the compute nodes 140 and is dynamically updated as the state of the available compute nodes 140 changes.

[0045] In various embodiments, the graphics selector 220 identifies a set of tasks 154 for performing the requested experience and determines a corresponding task graph 222 for use by the constraint analyzer 240. As described above, the display device 100 may support providing multiple different types of user experiences to the user. When the user requests a specific experience (e.g., a co-presence experience between two users), the selector 220 may receive the corresponding indication 204 of the request and identify an appropriate set of tasks 154 to facilitate the experience. In doing so, the selector 220 may determine one or more task graphs 222. As will be described below with respect to Figure 4A and Figure 4B As described, in various embodiments, the task graph 222 is a graph data structure that includes multiple interdependent graph nodes, and each graph node defines a set of constraints for performing a corresponding one of the set of tasks 154. In some embodiments, the selector 220 may dynamically compose the task graph 222 based on the requested experience indication 204 and one or more context factors regarding the experience. However, in some embodiments, the selector 220 may select one or more pre-created static task graphs 222.

[0046] In various embodiments, the personalization engine 230 generates user-specific QoS parameters 232 related to a particular user's preferences or tolerances for a particular quality of service. When a user operates a display device to enjoy a CGR experience, the user may have specific tolerances for factors such as latency, jitter, resolution, frame rate, etc. to avoid the experience becoming unpleasant. For example, if a user attempts to navigate a three-dimensional space in a VR game, the user may become dizzy and disoriented if the movement through the space is jittery. Additionally, the user's tolerances for these factors may vary from one another. To ensure that a given user has an enjoyable experience, the allocation engine 150 (or some other element of the display device 100) may collect user-specific parameters 232 related to the user's preferences or tolerances for these user-specific factors. For example, for a given experience, the engine 150 may determine a minimum frame rate for displaying three-dimensional content, a minimum latency for displaying three-dimensional content, and a minimum resolution for displaying three-dimensional content. If the engine 150 cannot allocate a particular set of tasks 154 in a manner that meets these requirements, the engine 150 may indicate that the experience cannot currently be provided, or evaluate a different set of tasks 154 to ensure that the parameters 232 can be met. In some embodiments, the parameters 232 may be determined by prompting the user for input. For example, the display device 100 may present content associated with a particular QoS and ask if this is acceptable to the user. In other embodiments, the parameters 232 may be determined when the user experiences a particular QoS and based on sensors 110 and 120. For example, sensors 110 and / or 120 may provide various information indicating that the user is experiencing discomfort, and the engine 150 may adjust the QoS of the experience to account for this detected discomfort.

[0047] In various embodiments, the constraint analyzer 240 determines how tasks 154 should be allocated between the display device 100 and the computing nodes 140 based on the dynamic constraint vector 212, the task graph 222, and the QoS parameters 232. Thus, the analyzer 240 can analyze the specific computing capabilities of the nodes 140 identified in the vector 212 and match those capabilities with the constraints in the task graph 222 while ensuring that the QoS parameters 232 are met. In some embodiments, the matching can include determining a plurality of different allocation plans 244 for allocating tasks 154 between the display device 100 and the computing nodes 140 and calculating a cost function 242 for each different allocation plan 244. In various embodiments, the cost function 242 is a function (or set of functions) that determines the specific cost of a given allocation plan 244. The cost of a given plan 244 can be based on any of a variety of factors, such as the total power consumption for implementing the plan 244, the latency for implementing the plan 244, the quality of service, etc. Based on the calculated cost functions for the different plans 244, the analyzer 240 can select a particular allocation 244 that is determined to have the lowest cost (or the highest cost below a certain threshold amount).

[0048] In various embodiments, the task publisher 250 facilitates the implementation of the allocation plan 244 selected by the constraint analyzer 240. Thus, the publisher 250 can examine the allocation plan 244 to determine that a particular task 154 has been allocated to a particular node 140 and contact that node 140 to request that it execute the allocated task 154. In some embodiments, the publisher 250 also processes the collection of the appropriate data to execute the allocated task 154 and transmits that data to the node 140. For example, if a given task 154 depends on information from the environmental sensors 110 and / or the user sensors 120 (e.g., images collected by the outward-facing camera sensor 110), then the publisher 250 can aggregate that information from the sensors 110 or 120 and transmit that information to the computing node 140 to which the task 154 has been allocated via a network connection.

[0049] Turning now to Figure 3 , the figure illustrates a block diagram of the allocation engine 210. In the illustrated embodiment, the discovery engine 210 includes a recruiter 310 and a collector 320. In some embodiments, the discovery engine 210 can be implemented differently than shown.

[0050] In various embodiments, the recruiter 310 processes discovery of computing nodes 140 and obtains assistance from the computing nodes. Although the recruiter 310 may use any suitable technique as described above, in the illustrated embodiment, the recruiter 310 sends a discovery broadcast 302 that solicits assistance from any available computing node 140 and identifies the computing node 140 based on the response from the computing node. As used herein, the term "broadcast" will be interpreted according to its established meaning and includes communications involving more than one recipient. For example, if the communication over the network connection is using IPv4, the recruiter 310 may send the discovery broadcast 302 to a broadcast address having a host portion consisting of all 1s. In various embodiments, the discovery broadcast 302 may be transmitted over a local area network accessible to the display device 100 to identify other nodes 140 that are part of that network. In some embodiments, the recruiter 310 may receive a broadcast notification 304 from the computing node 140. That is, instead of responding to any solicitation from the recruiter 310, the computing node 140 may send a notification 304 that indicates that the computing node is available to assist any display device 100 that happens to need assistance. In some embodiments, the recruiter 310 receives additional information about the available computing node 140, such as user information 306. In various embodiments, the computing node 140 may provide information 306 about the user (or users) of the computing node 140 such that the recruiter 310 can determine whether the computing node is part of the primary grid 142A described above. In such embodiments, the allocation engine 150 may confirm that the display device 100 shares the same user with the given computing node 140 (or is using a computing node 140 of a friend or family member) before attempting to allocate a task 154 to the given computing node 140. For example, in some embodiments, computing nodes 140 belonging to the primary grid 142A may indicate that they share a common household account that may be associated with a certain service. In response to receiving the information 306, the engine 150 may determine that the display device 100 is also associated with the household account in order to identify the computing node 140 as part of the primary grid 142A. In some embodiments, the recruiter 310 may also send a request for assistance from the server cluster 140F, which may implement cloud-based services for rendering three-dimensional content and providing other services as described above. In some embodiments, after discovering the node 140, the discovery engine 210 may begin receiving computing power information 152.

[0051] In various embodiments, collector 320 may be operative to compile the dynamic constraint vector 212 and transmit it to constraint analyzer 240. In some embodiments, constraint vector 212 may include information about a single node 140; in other embodiments, vector 212 may be multi-dimensional and include information 152 from multiple nodes 140. As shown, a given vector 212 may include one or more past entries 300A related to previous computing power information 152 and current real-time information 152 in entry 300B. In some embodiments, collector 320 may also analyze the current and past information 152 to predict the future capabilities of computing node 140 to facilitate display device 100, as shown in entry 300C. For example, collector 320 may employ a learning algorithm that evaluates past and present information 152 over time. In the illustrated embodiment, dynamic constraint vector 212 includes processor capabilities 332, memory capabilities 334, power budget 336, network capabilities 338, security capabilities 338, specific task affinity 342, and task latency 344. In other embodiments, vector 212 may include more (or fewer) elements than 332 through 344; aspects described below with respect to one element may also apply to other elements.

[0052] In various embodiments, processor capabilities 332 identify the processor information of a given computing node 140. Capabilities 332 may, for example, identify the number of processors, the type of processor, the operating frequency, etc. In some embodiments, capabilities 332 may identify the processor utilization of computing node 140. For example, capabilities 332 may identify that the processor is at 60% utilization. In another embodiment, capabilities 332 may represent the amount that a given computing node 140 is willing to allocate to display device 100. For example, capabilities 332 may identify that a given computing node is willing to allocate 10% of its processor utilization.

[0053] In various embodiments, memory capabilities 334 identify the memory information of a given computing node 140. Capabilities 334 may, for example, identify the type of memory and its storage capacity. In some embodiments, capabilities 334 may also identify the current utilization of the space. For example, capabilities 334 may identify that computing node 140 is capable of storing data of a specific size.

[0054] In various embodiments, the power budget 336 identifies constraints related to the power consumption of the compute node. For example, in the case where the compute node 140 is using battery power, the power budget 336 can identify the current charge level of the battery and its total capacity. In the case where the compute node 140 has a plug-in power supply, the power budget 336 can identify the plug-in situation and the number of watts being delivered. In some embodiments, the power budget 336 can indicate thermal information about the compute node 140. Thus, if a given node 140 is operating well below its thermal constraint, it may be able to accommodate a greater number of tasks 154. However, if a given node 140 is reaching its thermal constraint, it may be necessary to reallocate tasks 154 between other nodes 140 and the display device 100.

[0055] In various embodiments, the network capabilities 338 include information about the network interfaces of the compute node 140. For example, the capabilities 338 can identify the type of network interface supported by a given compute node 140, such as and so on. The capabilities 338 can also indicate the network bandwidth that can be provided via the network interface, which can be dynamic based on communication channel conditions. The capabilities 338 can also identify the network latency for communicating with the display device 100. For example, the capabilities 338 can indicate that an Internet Control Message Protocol (ICMP) echo request takes 20 ms to receive a response.

[0056] In various embodiments, the security capabilities 340 include information regarding the ability of the computing node 140 to perform tasks 154 in a secure manner. As described above, sensors 110 and 120 may collect sensitive information, and it may be necessary to protect the sensitive information to ensure user privacy. For example, when providing an MR experience, the camera sensor 110 may collect images of the user's surroundings. In various embodiments, the allocation engine 150 may verify the security capabilities 340 before offloading tasks 154 that include processing images (or some other form of sensitive information). In some embodiments, the capabilities 340 may identify the ability of the node 140 to securely process information by identifying the presence of specific hardware such as security elements, biometric authentication sensors, hardware security modules (HSMs), secure processors, secure execution environments, and the like. In some embodiments, the capabilities 340 may provide a signed certificate from the manufacturer of the computing node 140 that attests to the security capabilities of the computing node 140. In some embodiments, the certificate may also attest to other capabilities of a given node 140, such as the presence of specific tasks (as discussed in connection with task affinity 342), the ability to perform biometric authentication, whether the device includes the user's confidential data, and the like. In some embodiments, the capabilities 340 may identify the presence of a secure network connection due to the use of encryption or dedicated physical connections. In some embodiments, the capabilities 340 may identify whether the computing node 140 includes a biometric sensor and is configured to perform biometric authentication of the user.

[0057] In various embodiments, the specific task affinity 342 includes information regarding the ability of the computing node 140 to process a specific task 154. Thus, the affinity 342 may identify the presence of specific hardware and / or software for performing a specific task 154. For example, the affinity 342 may identify that a given node 140 has a GPU and may thus be more suitable for performing three-dimensional rendering tasks 154. As another example, the affinity 342 may identify that a given node 140 has a security element that has the user's payment credentials and may thus assist in performing payment transactions for the user. As another example, the affinity 342 may identify that a given node 140 supports a neural network engine that supports one or more tasks, such as object classification as described below.

[0058] In various embodiments, task latency 344 includes information regarding how long it may take a compute node to process a given task 154. For example, latency 344 may identify that a particular task 154 is expected to be 20 ms based on previous instances of the compute node 140 executing task 154 and the current utilization of the resources of node 140. In some embodiments, latency 344 may include network connectivity information such as the latency of a network connection as discussed above in connection with network capabilities 338. In such an embodiment, if the time required to offload and execute a given task 154 as indicated by task latency 344 exceeds a certain threshold, the allocation engine 150 may determine, for example, not to offload task 154.

[0059] Turning now to Figure 4A , the figure shows a block diagram of task graph 222A. As described above and as Figure 4A shown, in various embodiments, task graph 222 is a graph data structure having a plurality of nodes 400 corresponding to a set of tasks 154 being considered for offloading. In the illustrated embodiment, task graph 222A is an example of task graph 222 for a set of tasks 154A - 154C that are to be executed to classify objects present in one or more video frames 402 from camera sensor 110. For example, a user operating display device 100 may have walked into a store selling products. When the user observes a product using display device 100, display device 100 may attempt to classify the object and present AR content regarding the product being sold. As shown, task graph 222A includes a graph node 400A for object detection task 154A, where an object is detected in video frame 402 and a bounding box is placed around the object for subsequent analysis. Task graph 222A then includes a graph node 400B for image cropping task 154B, where content outside the bounding box is removed from frame 402 to produce a cropped frame 406. Finally, task graph 222A includes a graph node 400C for object classification task 154C, where the cropped frame 406 is analyzed to identify the classification 408 of the object in cropped frame 406 - for example, the user is observing a pair of shoes.

[0060] As shown, each graph node 400 may define a corresponding set of task constraints 410 for its respective task 154. In the illustrated embodiment, task constraints 410 include type 412, desired task latency 414, energy profile 416, desired network connection 418, security requirements 420, desired computing power 422, and task chain 424. In some embodiments, more (or fewer) constraints 410 may be defined for a given node 400. Additionally, the constraints defined for one graph node 400 may be different from the constraints defined in another graph node 400.

[0061] In various embodiments, the type 412 identifies the type of task 154 associated with a particular node 400. For example, node 400A may indicate that its type 412 is object detection, while node 400B may indicate that its type 412 is image cropping.

[0062] In various embodiments, the desired task latency 414 identifies the maximum allowable latency for performing a given task 154. For example, the latency 414 specified in node 400C may indicate that the object classification task 154 should be completed within 200 ms. Thus, if the task latency 344 in vector 212 indicates that a given computing node 140 cannot meet that latency 414, the analyzer 240 may prevent the computing node 140 from being considered a candidate for offloading the object classification task 154C.

[0063] In various embodiments, the energy profile 416 indicates the expected energy consumption for performing a given task 154. For example, the profile 416 for node 400A may indicate that object detection is a less energy-intensive task 154, while the profile 416 for node 400C may indicate that object classification is a more energy-intensive task 154. Thus, the analyzer 240 may assign task 154A to a more power-constrained computing node 140 or display device 100, while assigning task 154C to a less power-constrained node 140, as indicated, for example, by the power budget 336 in vector 212.

[0064] In various embodiments, the desired network connection 418 indicates the desired characteristics of the network connection associated with a given task 154. These characteristics can be the type of network connection (e.g., etc.), the required bandwidth of the connection, and / or the required latency of the network connection. For example, a task 154 that requires high bandwidth (e.g., streaming media content to the display device 100) may indicate a desire for a higher bandwidth connection. Thus, the analyzer 240 may attempt to match the characteristics identified in the desired network connection 418 with those identified in the network capabilities 338 of the computing node 140.

[0065] In various embodiments, the security requirement 420 indicates the requirement to perform a given task 154 in a secure manner. For example, considering that the video frame 402 may include sensitive content, each of the nodes 400A - 400C may specify the requirement 420 for the tasks 154A - 154C to be performed in a secure manner. Thus, the analyzer 240 may assign the tasks 154A - 154C to the computing nodes 140 based on the security capabilities 340 in the vector 212. Other examples of sensitive content may include keychain data, passwords, credit card information, biometric data, user preferences, other forms of personal information. Thus, if a particular task 154 is being performed using such information, the security requirement 420 may be set to ensure that any node 140 processing the information, for example, can be protected using some form of secure hardware such as a secure element, a hardware security module (HSM), a secure processor, etc. In various embodiments, the security requirement 420 may be important for assigning the task 154 to a given node 140 and may be continuously evaluated by the engine 150 when the set of available nodes 140 changes. For example, if the first node 140 is processing a task 154 with a security requirement 420 and the node 140 becomes unavailable, then the display device 100 may determine to abort a particular experience if another node 140 that can meet the requirement 420 cannot be found.

[0066] In various embodiments, the desired computing capability 422 indicates the desire for the computing node 140 to have specific hardware and / or software for offloading the processing of the task 154. For example, the node 400C may specify the hardware (or software) for implementing a neural network classifier capable of operating to perform the object classification task 154C. In some cases, the capability 422 may include more general specifications (e.g., general-purpose hardware for implementing a neural network), or may include more specific specifications (e.g., dedicated hardware specifically designed for implementing a convolutional neural network (CNN) for object classification). Thus, the analyzer 240 may evaluate the desired computing capability 422 for the specific task affinity 342 specified in the vector 212.

[0067] In various embodiments, the task chain 424 indicates that when two or more tasks 154 are assigned to the display device 100 or the computing node 140, they should be grouped together. For example, although Figure 4A not shown, the task chain 424 of the node 400A may indicate that the task 154A should be executed at the same computing node 140 as the task 154B. Thus, the analyzer 240 may not assign the task 154A and the task 154B to different nodes 140. As will be discussed below in connection with Figure 5 In some embodiments, the data for the tasks 154 linked together may be arranged contiguously in memory to improve the efficiency and / or security of accessing the data when the tasks 154 are executed.

[0068] As described above, after evaluating task graph 222 in combination with the dynamic constraint vector 212 and user-specific QoS parameters 232, the constraint analyzer 240 can determine an allocation plan 244 for offloading task 154. In some embodiments, the allocation plan 244 can be recorded in node 400. For example, the analyzer 240 can indicate in node 400A that task 154A has been allocated to display device 100, in node 400B that task 154B has been allocated to watch 140A, and in node 400C that task 154C has been allocated to HPC 140E. In other embodiments, the plan 244 can be indicated differently, and in some embodiments, the plan is provided separately from task graph 222A.

[0069] Turning now to Figure 4B , the figure shows a block diagram of another task graph 222B. As described above, the allocation engine 150 can evaluate tasks 154 related to content other than the visual content being presented on display device 100. For example, in the illustrated embodiment, task graph 222B relates to a set of tasks 154 for performing audio classification, which can be used for speech recognition. As shown, task graph 222B includes a graph node 400D for an audio detection task 154D, where an audio stream 432 recorded for speech analysis has a bounding box 434 placed around the speech. Task graph 222B also includes an audio cropping task 154E, where the recorded audio 432 is cropped based on the bounding box 434. Task graph 222B then includes a node 400F for an audio classification task 154F, where the speech in the cropped audio 436 is classified and an indication 438 of the classification is presented—for example, the user asks for the current weather today. Similar to task graph 222A, the analyzer 240 can analyze the task constraints 410 defined by nodes 400D - 400F in combination with vector 212 and parameters 232 to determine the allocation plan 244. For example, as Figure 4B shown, the analyzer 240 has selected a plan 244 that allocates the audio detection task 154D to display device 100, the audio cropping task 154E to watch 140A, and the audio classification task 154F to server cluster 140F. The results of the audio classification can then be presented, such as by announcing the current weather, via, for example, the audio system of display device 100.

[0070] Turning now to Figure 4C , the figure shows a block diagram of a larger task graph 222. In various embodiments, task graph 222 can be substantially larger than just a few nodes 400, and in some embodiments, even larger than Figure 4CThe number of nodes 400 shown in. In the illustrated embodiment, nodes 400 have been allocated among display device 100, watch 140A, HPC 140E, and server cluster 140F, as indicated by the different gray shadings. As shown, graph 222 can start from a root node 400, which can be selected based on a particular experience requested by a user, and it ends with a plurality of terminal nodes 400 that provide output to a plurality of systems such as another display device 100, the display system of display device 100, the audio system of display device 100, and so on. In some embodiments, graph 222 of tasks can be implemented differently than shown. For example, graph 222 can include more branches of nodes 400, the edges of nodes 400 can be connected to previous nodes 400 in a way that forms a loop, and so on.

[0071] In various embodiments, graph 222 of tasks can include nodes 400 that receive input from various sources. Thus, in the illustrated embodiment, HPC 140E can store cached content 442 that was previously generated and can be used to facilitate subsequent CGR experiences. For example, in a museum exhibit depicting a city map with a rendered building overlaid on a map, HPC 140E can cache the pre-generated content 442 to speed up future rendering of the map. In the example discussed below with respect to Figure 6D a user can store the previously generated content 442 to share the content with another device that can redisplay the content 442.

[0072] As described above and as Figure 4C shown, graph 222 of tasks can also include one or more instances of linked tasks 444 that are executed at the same computing node 140. For example, in the illustrated embodiment, all of the linked tasks 444 have been allocated to watch 140a. In some embodiments, linked tasks 444 can be changed based on task chain parameters 424 specified in a set of nodes 400 as described above. In some embodiments, allocation engine 150 can determine that a set of tasks 154 should be linked because the set of tasks can be executed more efficiently, more quickly, consume less power, reduce network traffic, etc. when executed at the same computing node 140.

[0073] Now turning to Figure 5, the figure shows a block diagram of components within display device 100 and computing node 140. In the illustrated embodiment, in addition to the environmental sensor 110 and user sensor 120 described above, display device 100 further includes a display system 510, a controller 520, a memory 530, a security element 540, and a network interface 550. As shown, a given computing node 140 includes a controller 560, a memory 570, and a network interface 580. In some embodiments, display device 100 and computing node 140 may be implemented differently than shown. For example, display device 100 and / or computing node 140 may include multiple network interfaces 550, display device 100 may not include security element 540, computing node 140 may include security element 540, etc. In some embodiments, display device 100 and / or computing node 140 may include one or more speakers for presenting audio content 104.

[0074] In various embodiments, display system 510 is configured to display the rendered frames to a user. Display 510 may implement any of a variety of display technologies. For example, as described above, display system 510 may include a near-eye display that presents a left image and a right image to create the effect of a three-dimensional view 102. In some embodiments, the near-eye display may use digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), or light emitting diode (LED). As another example, display system 510 may include a direct retinal projector that scans a frame including a left image and a right image pixel-by-pixel directly onto the user's eye via a reflective surface (e.g., a reflective eyeglass lens). To create a three-dimensional effect in view 102, objects at different depths or distances in the two images are shifted left or right as a function of triangulation of the distances, with nearer objects shifting more than farther objects. Display system 510 may support any medium, such as optical waveguides, holographic media, optical combiners, optical reflectors, or any combination thereof. In some embodiments, display system 510 may be transparent or translucent and configured to selectively become opaque.

[0075] In various embodiments, the controller 520 includes circuitry configured to facilitate the operation of the display device 100. Thus, the controller 520 may include one or more processors configured to execute program instructions, such as the allocation engine 150, to cause the display device 100 to perform the various operations described herein. These processors may be CPUs configured to implement any suitable instruction set architecture and may be configured to execute instructions defined in that instruction set architecture. For example, in various embodiments, the controller 520 may include a general-purpose processor or an embedded processor that implements any of a variety of instruction set architectures (ISAs), such as ARM, x86, PowerPC, SPARC, RISC, or MIPS ISA, or any other suitable ISA. In a multi-processor system, each processor may, but need not, together implement the same ISA. The controller 520 may employ any microarchitecture, including scalar, superscalar, pipelined, superpipelined, out-of-order, in-order, speculative, non-speculative, etc., or combinations thereof. The controller 520 may include circuitry implementing microcode techniques. The controller 520 may include one or more levels of cache, which may employ any size and any configuration (set-associative, direct mapped, etc.). In some embodiments, the controller 520 may include at least a GPU, which may include any suitable graphics processing circuitry. Generally, the GPU may be configured to render display objects into a frame buffer (e.g., a frame buffer including pixel data for an entire frame). The GPU may include one or more graphics processors, which may execute graphics software to perform some or all of the graphics operations or hardware-accelerate certain graphics operations. In some embodiments, the controller 520 may include one or more other components for processing and rendering video and / or images, such as an image signal processor (ISP), an encoder / decoder (codec), etc. In some embodiments, the controller 520 may be implemented as a system-on-chip (SOC).

[0076] In various embodiments, the memory 530 is a non-transitory computer-readable medium configured to store data and program instructions (such as the allocation engine 150) executed by a processor in the controller 520. The memory 530 can include any type of volatile memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of SDRAM, such as mDDR3, etc., or low-power versions of SDRAM, such as LPDDR2, etc.), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. The memory 530 can also be any type of non-volatile memory, such as NAND flash memory, NOR flash memory, nano-RAM (NRAM), magnetoresistive RAM (MRAM), phase change RAM (PRAM), racetrack memory, memristor memory. In some embodiments, one or more memory devices can be coupled to a circuit board to form a memory module, such as a single in-line memory module (SIMM), dual in-line memory module (DIMM), etc. Alternatively, these devices can be mounted with the integrated circuit implementing the system in a chip stack configuration, a package stack configuration, or a multi-chip module configuration.

[0077] In some embodiments, data related to a particular task 154 can be stored in the memory 530 based on the particular task 154. As described above, a set of tasks 154 can be linked together to be executed by the same computing node 140 or display device 100. In such embodiments, the data for this set of tasks 154 can be located together to facilitate access. For example, the data can be juxtaposed in the same physical storage device, the same storage page, a contiguous block of storage addresses, etc. In some embodiments, tasks 154 associated with security operations can be encrypted and / or stored in a portion of the memory 530 with restricted access. For example, the encryption provided by the security element 540 can be used to protect this portion of the memory 530.

[0078] In various embodiments, the security element (SE) 540 is a security circuit configured to perform various security operations on the display device 100. As used herein, the term "security circuit" refers to a circuit that protects isolated internal resources from direct access by external circuits such as the controller 520. The internal resources may be a memory that stores sensitive data such as personal information (e.g., biometric information, credit card information, etc.), encryption keys, random number generator seeds, etc. The internal resources may also be a circuit that performs services / operations associated with sensitive data (such as encryption, decryption, generation of digital signatures, etc.). For example, the SE 540 may maintain one or more cryptographic keys for encrypting data stored in the memory 530 to enhance the security of the display device 100. As another example, the security element 540 may also maintain one or more cryptographic keys to establish a secure connection, authenticate the display device 100 or the user of the display device 100, etc. As yet another example, the SE 540 may maintain biometric data of a user and be configured to perform biometric authentication by comparing the maintained biometric data with the biometric data collected by one or more user sensors 120. As used herein, "biometric data" refers to data that uniquely identifies a user (at least to a high degree of accuracy) among other people based on the user's physical or behavioral characteristics, such as fingerprint data, voice recognition data, facial data, iris scan data, etc.

[0079] In various embodiments, the network interface 550 includes one or more interfaces configured to communicate with external entities such as the computing node 140. As described above, the network interface 550 may support any suitable wireless technology, such as Long Term Evolution TM etc., or any suitable wired technology, such as Ethernet, Fibre Channel, Universal Serial Bus TM (USB), etc. In some embodiments, the interface 550 may implement a proprietary wireless communication technology (e.g., 60 gigahertz (GHz) wireless technology) that provides a highly directional wireless connection between the display device 100 and one or more computing nodes 140.

[0080] In various embodiments, the controller 560 includes a circuit configured to facilitate the operation of the display device 100. The controller 560 may implement any of the functions described above with respect to the controller 520. For example, the controller 560 may include one or more processors configured to execute program instructions to cause the computing node 140 to perform various operations described herein, such as processing the code 566 to process the offloaded task 145.

[0081] In various embodiments, the memory 570 is configured to store data and program instructions that are executed by a processor in the controller 560. The memory 570 may include any suitable volatile and / or non-volatile memory, such as those described above with respect to the memory 530. The memory 570 may be implemented in any suitable configuration, such as those described above with respect to the memory 530.

[0082] In various embodiments, the network interface 580 includes one or more interfaces configured to communicate with external entities such as the display device 100 and other computing nodes 140. The network interface 580 may also implement any suitable techniques, such as those described above with respect to the network interface 550.

[0083] Turning now to Figure 6A , this figure shows an illustration of on-device processing 600A. In the illustrated embodiment, on-device processing 600A is an example where the display device 100 cannot use the available computing nodes 140 to assist in rendering the 3D view 102A. In this particular example, the user is participating in a co-presence experience where the user is viewing some buildings of a city skyline with one or more other users represented by respective avatars 602A. Since the display device 100 is limited by its local computing power, the avatars 602A may be depicted as having only heads, and fewer buildings may be rendered in the view 102A.

[0084] Turning now to Figure 6B , this figure shows an illustration of computing-node processing 600B. In the illustrated embodiment, computing-node processing 600B is an example where the display device 100 is able to utilize the computing power of other computing nodes 140. In this example, the user may participate in a similar co-presence experience as described above, but the display device 100 discovers a nearby watch 140A and a workstation 140D and offloads the task 154 to them. Now, the 3D view 102B is rendered in more detail, such as including more buildings. In addition to the heads, the avatars 602B of other participants now also have bodies.

[0085] Turning now to Figure 6C, which shows an illustration of shared node processing 600C. As described above, in some cases, two or more display devices 100 may share a computing node 140. In the illustrated embodiment, shared node processing 600C is an example of display devices 100A and 100B that share HPC 140E but use separate watches 140A1 and 140A2. For example, both users may be in a museum hosting an MR exhibition where the users view some buildings. To facilitate the use of display devices 100A and 100B by the users, the museum may operate HPC 140E, and when the users enter the exhibition, display devices 100A and 100B detect the HPC. HPC 140E may allow display devices 100 to provide more vivid content than if display devices 100 only used their respective watches 140A. In such an example, HPC 140E may provide first computing capability information 152 to display device 100A and second computing capability information 152 to display device 100B in order to execute one or more tasks offloaded from display device 100A while executing one or more tasks offloaded from display device 100B. As more display devices 100 discover HPC 140E, its computing power may become more restricted, and this restriction is indicated in subsequent communications of computing capability information 152. Thus, display devices 100 may reassign more tasks 154 to their respective watches 140A. Alternatively, the user operating display device 100A may walk away from HPC 140E such that the network connection to HPC 140E degrades to the point where it can no longer be used, and display device 100A may dynamically reassign tasks 154 between itself and watch 140A.

[0086] Now turning to Figure 6D , which shows an illustration of stored processing 600D. As described above, computing node 140 may assist display device 100 by storing content for display device 100. In some embodiments, the content may be used to facilitate subsequent content rendering on display device 100. In some embodiments, the content may be used to facilitate presenting content on other devices that may include other display devices 100. For example, in the illustrated embodiment, the user operating display device 100 may be viewing display content 604 that includes a three-dimensional mixed reality (MR) environment that includes a collection of buildings rendered on a surface. In some examples, the user may want to share the display content 604 for replay on another device such as a friend's phone 610. In response to receiving such a request, display device 100 may request that computing node 140 such as server cluster 140F store the display content 604. In some embodiments, server cluster 140F may then receive a request for display content 604 from phone 610 and provide content 604 to phone 610 for presentation on the display of phone 610.

[0087] In some embodiments, the content 604 displayed on the display device 100 and the telephone 610 is rendered based on data provided by one or more sensors 110 and / or 120 in the display device. For example, the data may include data collected by a sensor in the display device configured to measure the orientation of the device 100, such as, in embodiments where the device 100 is an HMD, measuring the pose of the user's head. Thus, when a user operating the display device 100 changes the orientation of the device 100, such as by changing his or her head, the display content on both the display device 100 and the telephone 610 can be adjusted to reflect the changing view in front of the display device 100. As another example, the data may include data collected by an outward-facing camera in the display device, the outward-facing camera being configured to capture video frames of the environment in which the display device operates. Thus, the real-world content included in the frames can also be included in the content 604 displayed on both the display device 100 and the telephone 610. In various embodiments, in addition to storing the display content 604, the server cluster 140F may also receive one or more tasks 154 unloaded from the display device 100 to facilitate rendering content for the display device 100. In some embodiments, receiving the one or more tasks may include receiving data collected by one or more sensors 110 and / or 120 and using the received data to perform one or more offloaded tasks 154.

[0088] Turning now to Figure 7A , the figure shows a flowchart of method 700. Method 700 is one embodiment of a method that can be performed by a display device such as display device 100 or other examples of the devices described above. In many cases, the execution of method 700 (or methods 720 to 770 discussed below) can significantly improve the user experience by extending the computing available to deliver content such as AR, MR, VR, or XR content to the display device.

[0089] In step 705, the display device discovers one or more computing nodes (e.g., computing node 140) via a network interface (e.g., network interface 550), and the one or more computing nodes are operable to facilitate rendering of three-dimensional content displayed on a display system (e.g., display system 510) of the display device. In such an implementation, the discovery includes receiving information (e.g., computing capability information 152) identifying the rendering-facilitating capabilities of the one or more computing nodes. In some implementations, when the display system is displaying three-dimensional content, the display device receives real-time information identifying the current rendering-facilitating capabilities of the one or more computing nodes. In various implementations, the real-time information includes one or more power constraints (e.g., power budget 336) of the computing nodes facilitating rendering. In some implementations, the one or more power constraints (e.g., power budget 336) include constraints associated with a battery supplying power to the computing node, constraints associated with the processor utilization of the computing node, or thermal constraints of the computing node. In various implementations, the real-time information includes one or more latency constraints (e.g., network capabilities 338 and task latency 344) of the computing nodes facilitating rendering. In some implementations, the one or more latency constraints include the latency of the network connection between the computing node and the display device, the bandwidth of the network connection, or a time value identifying an expected time for performing a distributed task at the computing node.

[0090] In various implementations, the discovery includes sending, via the network interface, a request (e.g., discovery engine 210) soliciting assistance from computing nodes to facilitate rendering, and identifying the one or more computing nodes based on responses received from the one or more computing nodes. In some implementations, identifying the one or more computing nodes includes determining whether the one or more computing nodes share a common user with the display device (e.g., being part of primary grid 142A). In some implementations, the sending includes broadcasting the request on a local area network accessible via the network interface (e.g., discovery broadcast 302). In some implementations, the discovery includes sending, via the network interface, a request soliciting assistance from a computer cluster (e.g., server cluster 140F) implementing a cloud-based service to render three-dimensional content, and allocating one or more tasks of the set of tasks to the computer cluster.

[0091] In step 710, the display device evaluates a set of tasks (e.g., task 154) based on the received information to identify one or more tasks to offload to one or more compute nodes to facilitate rendering. In various embodiments, the display device determines a plurality of different allocation plans (e.g., allocation plan 244) for task allocation between the display device and the one or more compute nodes, calculates a cost function (e.g., cost function 242) for each of the plurality of different allocation plans based on the received information, and selects one of the plurality of allocation plans for allocation based on the calculated cost function.

[0092] In various embodiments, the display device receives a request from a user of the display device to perform a particular operation (including displaying three-dimensional content), and based on the particular operation, determines a graph data structure including a plurality of graph nodes, each of the plurality of graph nodes defining a set of constraints for performing a corresponding one of the set of tasks. In such an embodiment, the evaluation of the set of tasks includes analyzing the graph data structure to determine an allocation plan for allocation. In some embodiments, one of the plurality of graph nodes specifies a constraint (e.g., desired computing power 422) for using particular hardware to perform one of the set of tasks, and the evaluation includes identifying a compute node having the particular hardware for performing the task. In some embodiments, the particular hardware is a graphics processing unit (GPU). In some embodiments, the display device includes a camera configured to capture an image of the environment in which the user operates the display device, the task is to classify an object present in the image (e.g., object classification 154C), and the particular hardware is hardware implementing a neural network classifier capable of operating to classify the object. In some embodiments, the display device includes a camera configured to capture an image of the environment in which the user operates the display device, one of the plurality of graph nodes specifies a constraint (e.g., security requirement 420) for performing the task in a secure manner using the image, and the evaluation includes identifying a compute node capable of operating to perform the task in a secure manner. In some embodiments, the identification of the compute node includes determining that the network connection between the display device and the compute node is encrypted.

[0093] In various embodiments, the display device collects one or more user-specific parameters (e.g., parameter 232) related to the user's tolerance for rendering three-dimensional content according to a particular quality of service, and the evaluation of the set of tasks is based on the one or more collected user-specific parameters. In some embodiments, the one or more user-specific parameters include a minimum frame rate for displaying three-dimensional content, a minimum latency for displaying three-dimensional content, or a minimum resolution for displaying three-dimensional content.

[0094] In step 715, the display device assigns the identified one or more tasks to one or more computing nodes via a network interface for the one or more computing nodes to process. In some embodiments, step 715 includes the display device dynamically identifying some of the tasks for offloading based on real-time information and reallocating the dynamically identified tasks between the display device and the one or more computing nodes. In some embodiments, the display device analyzes the received real-time information to predict the future ability of the one or more computing nodes to facilitate rendering (e.g., predicting entry 300C), and based on the predicted future ability, reallocates the dynamically identified tasks between the display device and the one or more computing nodes.

[0095] Turning now to Figure 7B , the figure shows a flowchart of method 720. Method 720 is an embodiment of a method executable by a computing device (such as one of display device 100 or computing node 140) executing program instructions (such as the program instructions of allocation engine 150).

[0096] In step 725, the computing device receives computing information (e.g., computing power information 152) identifying the ability of one or more computing nodes to facilitate rendering of three-dimensional content (e.g., 3D view 102) displayed on the display device. In some embodiments, the computing information is continuously received when three-dimensional content is being displayed on the display device, and the computing information includes (e.g., processor power 332, memory power 334, or network power 338) the utilization of one or more hardware resources included in the one or more computing nodes. In some embodiments, prior to receiving the computing information, the computing device discovers the one or more computing nodes by sending a broadcast (e.g., discovery broadcast 302) requesting assistance in rendering the three-dimensional content.

[0097] In step 730, the computing device determines whether to offload one or more tasks (e.g., task 154) associated with the rendering of three-dimensional content based on the computing information. In some embodiments, the computing device calculates a cost function (e.g., cost function 242) for multiple different allocation plans (e.g., allocation plan 244) for distributing one or more tasks among one or more computing nodes, and based on this calculation, selects one allocation plan from the multiple allocation plans that is determined to have the lowest power consumption. In some embodiments, the computing device receives an indication of the desired experience to be provided to the user (e.g., requested experience indication 204) from the display device of the user, and based on this indication, determines a graph data structure (e.g., task graph 222) that has multiple graph nodes corresponding to a set of tasks for providing the experience, and determining whether to offload the one or more tasks includes evaluating parameters (e.g., task constraints 410) specified in the multiple graph nodes. In some embodiments, one graph node among the multiple graph nodes identifies a specific task to be performed (e.g., type 412) and identifies a specific latency (e.g., desired task latency 414) for performing the task, and determining whether to offload the one or more tasks includes determining whether the computing node can meet the specific latency. In some embodiments, the computing device evaluates the user's interaction with the three-dimensional content to determine a user-specific tolerance for latency associated with rendering (e.g., user-specific QoS parameter), and based on the determined user-specific tolerance for latency, determines whether to offload the one or more tasks.

[0098] In step 735, the computing device offloads the one or more tasks to one or more computing nodes to cause the one or more computing nodes to execute the one or more offloaded tasks. In some embodiments, the computing device receives an image (e.g., video frame 402) collected from the environment in which the display device operates from a camera attached to the display device, and offloads a task to the computing node that includes using the content of the collected image to generate mixed reality content to be displayed on the display device.

[0099] Now turning to Figure 7C , the figure shows a flowchart of method 740. Method 740 is one embodiment of a method executable by a computing device implementing a computing node (such as computing node 140).

[0100] In step 745, the computing device provides computing information (e.g., computing capability information 152) that identifies the computing device's ability to facilitate rendering of three-dimensional content (e.g., 3D view 102) displayed on a display device (e.g., display device 100). In various embodiments, the computing device continuously provides the computing information while the computing device is performing one or more tasks. In some embodiments, the computing information includes a value indicating the current charge of a battery that powers the computing device (e.g., power budget 336). In some embodiments, the computing information includes latency information (e.g., task latency 344) that can be used to determine an expected time for the computing device to perform an offloaded task. In some embodiments, the computing device receives a request to assist in rendering three-dimensional content (e.g., discovery broadcast 302), and in response to the request, provides information about the user of the computing device (e.g., user information 306), and the information about the user can be used to determine whether the display device is being used by the same user.

[0101] In step 750, the computing device receives one or more tasks (e.g., task 154) offloaded from the display device based on the provided computing information. In some embodiments, step 750 includes the computing device receiving image information (e.g., video frame 402, bounding box 404, or cropped frame 406) collected from a camera (e.g., camera sensor 110) embedded in the display device.

[0102] In step 755, the computing device executes the one or more tasks to facilitate rendering of the three-dimensional content. In some embodiments, step 755 includes the computing device processing the received image information to produce content to be mixed with the three-dimensional content, thereby presenting a mixed reality environment on the display device.

[0103] In step 760, the computing device provides the results of executing these tasks.

[0104] In various embodiments, method 740 further includes the computing device receiving computing information that identifies the ability of one or more other computing devices to facilitate rendering of three-dimensional content displayed on the display device, and providing a set of tasks offloaded from the display device to the one or more other computing devices. In some embodiments, the computing device provides the received computing information to another computing device that is configured to determine whether to offload the set of tasks to the one or more other computing devices. In some embodiments, the computing device determines whether to offload the set of tasks to the one or more other computing devices based on the received computing information. In some embodiments, the computing device: provides first computing information that identifies the computing device's ability to facilitate rendering of three-dimensional content displayed on a first display device (e.g., Figure 6Cthe ability to render three-dimensional content on a first display device (e.g., display device 100A); provide second computing information that identifies the ability of the computing device to facilitate rendering of three-dimensional content on a second display device (e.g., display device 100B); and perform one or more tasks offloaded from the first display device while performing one or more tasks offloaded from the second display device.

[0105] Turning now Figure 7D , the figure shows a flowchart of method 770. Method 770 is one implementation of a method executed by a computing system (such as one of computing nodes in system 10 or computing node 140) to facilitate sharing of display device content on other devices.

[0106] Method 770 begins at step 775, where the computing system stores three-dimensional content (e.g., display content 604) rendered for a display device (e.g., display device 100). In various implementations, the three-dimensional content is rendered based on data provided by one or more sensors in the display device (e.g., ambient sensor 110 or user sensor 120). In some implementations, the three-dimensional content includes mixed reality (MR) content rendered based on the environment in which the user operates the display device. In some implementations, the data includes data collected by a sensor in the display device configured to measure the user's head pose. In some implementations, the data includes video frames captured by an outward-facing camera in the display device, the outward-facing camera being configured to capture the environment in which the display device operates. At step 780, the computing system receives a request for the three-dimensional content from a computing device other than the display device (e.g., phone 610). At step 785, the computing system provides the three-dimensional content to the computing device for presentation on the computing device's display. In various implementations, method 770 further includes the computing system receiving one or more tasks (e.g., task 154) offloaded from the display device to facilitate rendering of the three-dimensional content. In some implementations, receiving the one or more tasks includes receiving data collected by one or more sensors, and the computing system using the received data to perform one or more offloaded tasks, thereby facilitating rendering.

[0107] Turning now Figure 8, the figure shows a block diagram of the ability exchange 800. As described above, the computing node 140 can provide computing ability information 152 to the allocation engine 150 to facilitate the determination of which tasks 154 should be offloaded. In some embodiments, to ensure the accuracy of the information 152, some of this information may be included in a signed attestation provided by the computing node 140. Thus, in the illustrated embodiment, the computing node 140 (such as the tablet computer 140C) can contact the trusted certificate authority 820 to obtain a signed certificate 812 that attests to its capabilities, and provide the certificate 812 to the allocation engine 150.

[0108] In various embodiments, the trusted certificate authority (CA) 810 is a trusted computing system configured to issue signed certificates 812. In some embodiments, the CA 810 may be operated by the manufacturer of the display device 100 and / or the computing node 140; however, in other embodiments, the CA 810 may be operated by some other entity. In various embodiments, the computing node 140 can obtain the certificate 812 by generating a public key pair having a public key 814A and a corresponding private key 814B and issuing a certificate signing request (CSR) to the CA 810. In some embodiments, the CSR is further signed by a trusted key maintained by the computing node 140 to establish trust with the CA 810. Such a trusted key may be stored in the computing node, for example, during the manufacture of the computing node 140. In some embodiments, the trusted key may be unique to a given computing node 140 (or, in another embodiment, unique to devices of the same model of a particular generation, i.e., devices of the same model and generation may store the same key). Once the CSR can be successfully verified, the CA 810 can issue the corresponding certificate 812, which can be signed using a trusted private key maintained by the CA 810.

[0109] The certificate 812 can include any suitable information that can be used by the allocation engine 150, such as one or more of the parameters 332 to 344 discussed above. For example, the certificate 812 can specify that the computing node 140 includes secure hardware (e.g., SE, HSM, secure processor, etc.) as a security capability 340. As another example, the certificate 812 can specify a task affinity 342 for performing neural network-related tasks 154, since the computing node 140 can include dedicated hardware that implements a neural network engine. In some embodiments, the certificate 812 can include manufacturer information that attests that the computing node 140 is a genuine device, such as identifying the name of the manufacturer and confirming that the authenticity of the computing node 140 has been verified. The certificate 812 can also include the public key 814A, a digital signature generated using the private key 814B, and the digital signature of the above-mentioned CA 810. In some embodiments, the certificate 812 can be X.509 compatible; however, in other embodiments, the certificate 812 can be implemented using some other form of signed attestation.

[0110] Once the certificate 812 is received, the distribution engine 150 can verify the certificate 812 to ensure its authenticity. This can include verifying the signature of the CA 810 to ensure the integrity of the content of the certificate 812. In some embodiments, the distribution engine 150 can further verify the computing node 140 by issuing a challenge to the computing node 140 to perform a cryptographic operation using the private key 814A in the public key pair and verifying the result of the cryptographic operation (e.g., digital signature) using the public key 814A in the public key pair. If the verification is successful, the distribution engine 150 can then attempt to identify a task 154 having task constraints 410 that match the capabilities identified in the certificate 812. In some embodiments, the display device 100 can also use the public key 814A to establish a secure connection with the computing node 140, such as using an Elliptic Curve Diffie-Hellman (ECDH) exchange to establish a shared cryptographic key.

[0111] Turning now to Figure 9 , the figure shows a flowchart of a method 900. The method 900 is an embodiment of a method that can be performed by a computing device such as the display device 100 or other examples of the above-described devices. In many cases, the execution of the method 900 can improve the security of the computing device when interacting with other computing nodes to present a CGR experience.

[0112] In step 905, the computing device identifies a plurality of tasks (e.g., task 154) to be performed for presenting computer-generated reality (e.g., 3D view 102) to a user. In various embodiments, the plurality of tasks include tasks that require specific capabilities. In some embodiments, step 905 includes evaluating a graph data structure (e.g., task graph 222) having graph nodes corresponding to the plurality of tasks, where the graph nodes specify criteria (e.g., task constraints 410) for performing the plurality of tasks. In such embodiments, the computing device determines from some of the graph nodes that one or more tasks require one or more capabilities (e.g., based on the desired computing capabilities 422).

[0113] In step 910, the computing device receives a signed attestation (e.g., an attestation certificate 812) from a computing node (e.g., computing node 140) specifying that the computing node has one or more capabilities. In some embodiments, the signed attestation specifies that the computing node includes secure hardware (e.g., secure element 540) configured to cryptographically isolate data operating during an offloading task performed by the computing node. In some embodiments, the signed attestation specifies that the computing node includes a neural network engine available for performing offloading tasks. In some embodiments, the signed attestation attests that the computing node is a genuine product of a particular manufacturer. In various embodiments, the signed attestation is issued by a certificate authority (e.g., certificate authority 810) in response to a certificate signing request for a public key pair generated by the computing node.

[0114] In step 915, in response to successful verification of the signed attestation, the computing device offloads to the computing node one or more tasks of the plurality of tasks determined to require one or more capabilities specified in the signed attestation. In some embodiments, the computing device verifies the signed attestation by issuing a challenge to the computing node to perform a cryptographic operation using the private key (e.g., private key 814B) in the public key pair and verifying the result of the cryptographic operation using the public key (e.g., public key 814A) in the public key pair.

[0115] Turning now to Figure 10 , the figure shows a block diagram of the personalization engine 230. As described above, the personalization engine 230 can generate user-specific QoS parameters 232 related to a particular user's preferences or tolerances for a particular quality of service. In the illustrated embodiment, the engine 230 includes one or more likelihood estimators 1010, a signal encoder 1020, and a personal cache 1030. In other embodiments, the engine 230 may be implemented differently than shown.

[0116] In various embodiments, likelihood estimator 1010 analyzes signals related to the user experience and condition-specific features (e.g., to maintain object shape, enhance audio, smooth, filter, compress, etc.). In the illustrated embodiment, estimator 1010 receives sensor stream 1002, system constraints 1004, and context cues 1006. Sensor stream 1002 may include raw multimodal sensor data (e.g., from a camera, an inertial measurement unit (IMU), an audio sensor, or another of environmental sensor 110 and user sensor 120) and computed metadata (e.g., related to statistical properties of the signal). System constraints 1004 may include constraints related to power, computation, latency, or various other constraints discussed above. Context cues 1006 may provide cues about salience and attributes that may be more relevant, such as user context (e.g., content preferences, security, privacy, emotional state, health-related, audio volume), perceptual tolerance thresholds (e.g., sensing discomfort), security (e.g., warnings to avoid hazards), etc. Context cues 1006 may also include information about specific locations / regions where display device 100 may provide a particular experience (e.g., in a store, museum, etc.), and thus, personalization engine 230 may customize / personalize QoS parameters 232 based on delivering a curated experience in a specific location / region. In the illustrated embodiment, estimator 1010 outputs probability map 1012 to signal encoder 1020.

[0117] In various embodiments, signal encoder 1020 uses probability map 1012 and dynamic QoS estimate 1022 to generate user-specific parameters 206. QoS estimate 1022 may be based on location and network conditions (or other conditions). In various embodiments, parameters 206 may be output as QoS vector values that can be applied to meet overall system constraints (e.g., related to location, power, latency, bandwidth, fidelity, etc.).

[0118] In various embodiments, personal cache 1030 stores various parameter information that may have been previously determined by likelihood estimator 1010 and signal encoder 1020 and analyzed in subsequent determinations. In the illustrated embodiment, these parameters include previously determined probability map 1012 and previously determined user-specific QoS parameters 206, which may be combined with other phases (e.g., estimation, training, inference, adaptation). In various embodiments, personal cache 1030 is implemented in a manner that preserves the privacy of the stored information, as the information may include user-related information.

[0119] ***

[0120] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of the disclosure, even where only a single embodiment is described with respect to a particular feature. The examples of features provided in this disclosure are intended to be illustrative and not limiting unless stated otherwise. The foregoing description is intended to cover such alternative forms, modifications, and equivalents as would be apparent to a person of ordinary skill in the art who is aware of the effective effects of this disclosure.

[0121] The scope of the disclosure includes any feature or combination of features (explicitly or implicitly) disclosed herein or any generalization thereof, regardless of whether it alleviates any or all of the problems addressed herein. Accordingly, new claims may be made during the prosecution of this patent application (or a patent application claiming priority thereto) with respect to any such combination of features. Specifically, referring to the appended claims, the features of the dependent claims may be combined with the features of the independent claims, and the features from the corresponding independent claims may be combined in any suitable manner other than only by the specific combinations recited in the appended claims.

Claims

1. A display device, comprising: A display system configured to display three - dimensional content to a user; A network interface; One or more processors; And A memory having program instructions stored therein, the program instructions being executable by the one or more processors to cause the display device to perform operations including: Discovering, via the network interface, one or more computing nodes capable of operating to facilitate rendering of the three - dimensional content, and When the display system displays the three - dimensional content: Receiving information identifying the capabilities of the one or more computing nodes to facilitate the rendering; Based on the received information, evaluating a set of tasks to dynamically identify one or more tasks in the set of tasks to be offloaded to the one or more computing nodes to facilitate the rendering; And Re - allocating, via the network interface, the dynamically identified one or more tasks to the one or more computing nodes for processing by the one or more computing nodes.

2. The display device according to claim 1, wherein the information is real - time information.

3. The display device according to claim 2, wherein the real - time information includes one or more power constraints of the computing nodes facilitating the rendering, and wherein the one or more power constraints include constraints associated with a battery supplying power to the computing nodes, constraints associated with the processor utilization of the computing nodes, or thermal constraints of the computing nodes.

4. The display device according to claim 2, wherein the real - time information includes one or more latency constraints of the computing nodes facilitating the rendering, and wherein the one or more latency constraints include the latency of the network connection between the computing nodes and the display device, the bandwidth of the network connection, or a time value identifying an expected time for executing a distributed task at the computing nodes.

5. The display device according to claim 1, wherein the evaluation includes: Determining a plurality of different allocation plans for allocating the tasks between the display device and the one or more computing nodes; Based on the received information, calculating a cost function for each of the plurality of different allocation plans; And Based on the calculated cost functions, selecting one of the plurality of allocation plans for the allocation.

6. The display device according to claim 1, wherein the operations include: Receiving, from a user of the display device, a request to perform a specific operation including displaying the three - dimensional content; And Based on the specific operation, determining a graph data structure including a plurality of graph nodes, wherein each of the plurality of graph nodes defines a set of constraints for performing a corresponding one of the set of tasks; and Wherein the evaluation of the set of tasks includes analyzing the graph data structure to determine an allocation plan for the allocation.

7. The display device according to claim 6, further comprising: A camera configured to capture an image of the environment in which the user operates the display device; One of the plurality of graph nodes specifies constraints for performing a task using the image in a secure manner; and wherein the evaluation includes identifying computing nodes that can operate to perform the task in the secure manner.

8. The display device according to claim 1, wherein the operation includes: collecting one or more user-specific parameters related to the user's tolerance for rendering the three-dimensional content according to a specific quality of service, wherein the one or more user-specific parameters include a minimum frame rate for displaying the three-dimensional content, a minimum latency for displaying the three-dimensional content, or a minimum resolution for displaying the three-dimensional content; and wherein the evaluation of the set of tasks is based on the one or more collected user-specific parameters.

9. The display device according to claim 1, wherein the discovery includes: sending, via the network interface, a request for assistance from computing nodes to facilitate the rendering; and identifying the one or more computing nodes based on responses received from the one or more computing nodes.

10. The display device according to claim 1, wherein the display device is a head-mounted display (HMD).

11. A non-transitory computer-readable medium having program instructions stored therein, the program instructions being executable by a computing device to cause the computing device to perform operations including: when the computing device displays three-dimensional content: receiving computing information identifying the ability of one or more computing nodes to facilitate rendering of the three-dimensional content to be displayed on the computing device; dynamically determining, based on the computing information, whether to offload one or more tasks associated with the rendering of the three-dimensional content; and reassigning the dynamically determined one or more tasks to the one or more computing nodes to cause the one or more computing nodes to perform the offloaded one or more tasks.

12. The computer-readable medium according to claim 11, wherein the computing information is continuously received when the three-dimensional content is displayed on the computing device, and wherein the computing information includes utilization of one or more hardware resources included in the one or more computing nodes.

13. The computer-readable medium according to claim 11, wherein the operations include: evaluating a user's interaction with the three-dimensional content to determine a user-specific tolerance for latency associated with the rendering; and determining whether to offload the one or more tasks based on the determined user-specific tolerance for the latency.

14. The computer-readable medium according to claim 11, wherein the operations include: receiving an indication of a desired experience to be provided to the user from a user of the computing device; and determining, based on the indication, a graph data structure having a plurality of graph nodes corresponding to a set of tasks for providing the experience; and wherein determining whether to offload the one or more tasks includes evaluating parameters specified in the plurality of graph nodes.

15. The computer-readable medium according to claim 11, wherein the computing device is a display device.

16. A method, comprising: providing, by a computing device, real-time computing information that identifies the computing device's ability to facilitate rendering of three-dimensional content displayed on a display device; receiving, by the computing device, one or more tasks offloaded from the display device that are dynamically identified based on the provided computing information; executing, by the computing device, the one or more dynamically identified tasks to facilitate the rendering of the three-dimensional content; and providing, by the computing device, the results generated by executing the tasks.

17. The method according to claim 16, further comprising: continuously providing, by the computing device, the computing information while the computing device is executing the one or more tasks.

18. The method according to claim 16, further comprising: receiving, by the computing device, a request to assist in rendering the three-dimensional content; and in response to the request, the computing device providing information about a user of the computing device, wherein the information about the user can be used to determine whether the display device is being used by the same user.

19. The method according to claim 16, further comprising: providing, by the computing device, first computing information that identifies the computing device's ability to facilitate rendering of three-dimensional content displayed on a first display device; providing, by the computing device, second computing information that identifies the computing device's ability to facilitate rendering of three-dimensional content displayed on a second display device; and executing, by the computing device, one or more tasks offloaded from the first display device while executing one or more tasks offloaded from the second display device.

Images