Mixed reality system with reduced power rendering

Abstract

Embodiments of the present disclosure provide a mixed reality system that achieves less power. In some examples, a passive state of the mixed reality system can have predictable content that does not require processing by a CPU. In such examples, the predictable content can be identified and rendered by a GPU while the CPU is in a low power mode. Thus, embodiments of the present disclosure provide a benefit that is not available with conventional techniques because a CPU can consume more power than a corresponding GPU. In some examples, the passive state can have the fact that the predictable content is identified and rendered without using the CPU. In such examples, the passive state can render predictable content that does not require processing by the CPU.

Description

[0001] This application is a divisional application of Chinese patent application No. 201780070885.4, entitled "Mixed Reality System with Reduced Power Rendering" (filed on November 15, 2017).

[0002] Cross-references to related applications

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 423,154, filed November 16, 2016, entitled “Mixed-Reality System with Reduced Power Rendering,” the disclosure of which is incorporated herein by reference in its entirety for all purposes. Background Technology

[0004] Mixed reality combines real-time digital information with the user's field of view of the physical environment. Digital information can be anchored to the physical environment to give the impression that the digital information is present within the physical environment. To achieve mixed reality, the physical environment can be analyzed and digital information generated. However, to create a realistic combination, analysis and generation must be performed frequently, which inevitably consumes significant power. Therefore, there is a need in the art for improved methods and systems to reduce power consumption when implementing mixed reality. Summary of the Invention

[0005] Methods, systems, and computer programs for implementing mixed reality are provided. In some examples, the graphics processing unit (GPU) can at least partially avoid using the central processing unit (CPU). In such examples, a passive state can render predictable content that does not require processing by the CPU. In such examples, predictable content can be rendered by the GPU while the CPU is in a low-power mode.

[0006] In some examples, passive state signifies the intention to push as much imaging determination and rendering computation as possible to the GPU (rather than the CPU). The GPU can consume less power than the CPU because it requires less data input to provide sufficient output. For example, while the CPU needs complete information about the current state, the GPU may only need to determine how to render content (e.g., timing, field of view limitations, depth planes, content type, etc.). And because the GPU requires less data input, it can use different categories of sensors and architectures. For example, due to the computation cycles involved, CPUs with larger memory volumes and component integration are limited to a category of sensors and architectures (component types) corresponding to the CPU's requirements. Or in other words, a class of sensors and architectures that can execute faster and use less power may be of little use to the CPU, while the GPU may be able to use this class of sensors and architectures and provide sufficient output. Therefore, the capabilities of a GPU can be diverse because it can enable different kinds of sensors and architectures, not just a better version of a CPU.

[0007] Embodiments of this disclosure provide methods for implementing mixed reality with reduced power consumption. In some examples, the GPU can avoid some CPU usage. In such examples, a passive power state can be provided between an idle state and an active state. In some examples, the passive state allows the GPU to render predictable content that does not require CPU processing. In such examples, predictable content can be identified and rendered by the GPU while the CPU is in a low-power mode. Therefore, embodiments of this disclosure provide benefits not available in conventional techniques, as the CPU may consume more power than the corresponding GPU. In some examples, the passive state can leverage the fact that predictable content can be identified and rendered without CPU usage. In such examples, the passive state can render predictable content that does not require CPU processing.

[0008] For example, a method for rendering mixed reality content using a mixed reality device can be provided. This method may include acquiring sensor data associated with a physical environment. In some examples, one or more sensors may be used to acquire the sensor data. The method also includes having a vision processing unit (VPU) use the sensor data to determine the pose of the mixed reality device and sending the pose of the mixed reality device to a graphics processing unit (GPU). In some examples, the pose of the mixed reality device includes the position and orientation of the mixed reality device. In some examples, the GPU may be located away from the mixed reality device. In some examples, the method also includes sending the pose of the mixed reality device to a central processing unit (CPU). In some examples, the CPU may operate in a low-power mode throughout the method. The method also includes having the GPU identify content corresponding to the pose. In some examples, the content may be stored in the GPU's cache before identification. The method further includes having the GPU align the content with the physical environment, sending the aligned content to the mixed reality device, and using the display of the mixed reality device to render the aligned content. In some examples, the method may further include: determining that the identified content is predictable content, and in response to determining that the identified content is predictable, retrieving the identified content from the GPU's cache. In other examples, the method may also include: determining that the identified content is dynamic, and in response to determining that the identified content is dynamic, sending a message to the CPU. In such an example, the message could cause the CPU to switch from a low-power mode to a high-power mode.

[0009] For another example, a system can be provided that uses a mixed reality device to render mixed reality content. This system may include a mixed reality device and a graphics processing unit (GPU), which may be included in or separate from the mixed reality device. In some examples, the mixed reality device may include one or more sensors, a vision processing unit (VPU), and a display. In such an example, one or more sensors may be configured to capture sensor data. The VPU may be configured to determine a pose associated with the mixed reality device. The pose can be determined using the sensor data. The GPU may be configured to identify content to be rendered by the display of the mixed reality device, align the content with the physical environment, and send the aligned content to the mixed reality device for rendering by the display.

[0010] This disclosure provides numerous benefits compared to conventional technologies. For example, embodiments of this disclosure offer power savings by allowing a high-power CPU to operate in a low-power mode for an extended period, while maintaining consistent performance in perception and graphics rendering via the GPU. These power savings can be applied to many applications (e.g., productivity software, web browsing, multimedia interaction) where power usage can be minimized when visual updates are not triggered by the user's physical movements. This power saving enables always-on mixed reality, maintaining a consistent visual environment, which is crucial for users to perceive digital information as part of their physical environment. These and other embodiments of this disclosure, along with their advantages and features, can be described in more detail below and with reference to the accompanying drawings. Attached Figure Description

[0011] The illustrative embodiments are described in detail below with reference to the accompanying drawings.

[0012] Figure 1 An example of a mixed reality system for implementing passive power states, according to one embodiment, is shown.

[0013] Figure 2 An example of a process for presenting mixed reality, performed by a vision processing unit (VPU) according to one embodiment, is shown.

[0014] Figure 3 An example of a process for rendering mixed reality, executed by a central processing unit (CPU) according to one embodiment, is shown.

[0015] Figure 4 An example of a process for using a mixed reality device to render dynamic content, according to one embodiment, is shown.

[0016] Figure 5 An example of a process for using a mixed reality device to present predictable content, according to one embodiment, is shown.

[0017] Figure 6 An example block diagram of a central processing unit (CPU) according to one embodiment is shown.

[0018] Figure 7 An example block diagram of the internal components of a central processing unit (CPU) according to one embodiment is shown.

[0019] Figure 8 An example block diagram of a graphics processing unit (GPU) according to one embodiment is shown.

[0020] Figure 9 An example block diagram of a single-stream multiprocessor according to one embodiment is shown.

[0021] Figure 10 An example block diagram of a vision processing unit (VPU) according to one embodiment is shown. Detailed Implementation

[0022] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of embodiments of the present disclosure. However, it will be apparent that various embodiments may be practiced without these specific details. The accompanying drawings and description are not limiting.

[0023] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing the exemplary embodiments. It should also be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this disclosure as set forth in the appended claims.

[0024] Traditionally, mixed reality devices (e.g., wearable mixed reality glasses) manage power consumption using two states: idle and active. In the idle state, no content is rendered. In the active state, the central processing unit (CPU) determines / identifies the content, and the graphics processing unit (GPU) renders the determined / identified content. However, this two-state system consumes a significant amount of power in the active state.

[0025] Embodiments of this disclosure provide examples of implementing mixed reality with less power. In some examples, the GPU can avoid some CPU usage. In such examples, a passive power state can be provided between an idle state and an active state. In the passive state, the GPU determines / identifies / predicts content and renders predictable content, while the CPU does not determine / identify / predict content. In such examples, predictable content can be identified and rendered by the GPU while the CPU is in an idle or semi-idle state (also known as a low-power mode). Therefore, embodiments of this disclosure provide benefits not available in conventional technologies where a CPU in a conventional technology may consume more power than a corresponding GPU. In some examples, the passive state can leverage the fact that predictable content is identified and rendered by the GPU without using the CPU. In such examples, in the passive state, the GPU can render predictable content that does not require processing by the CPU.

[0026] Figure 1 An example of a mixed reality system for implementing passive power states according to an embodiment of the present invention is shown. The mixed reality system can provide a persistent mixed reality environment comprising virtual content overlaid on a physical environment. For example, the virtual content may appear in the physical environment over time.

[0027] The mixed reality system includes a mixed reality device 110. The mixed reality device 110 can present virtual content to a user. The mixed reality device 110 also allows the user to see at least a portion of the physical environment in front of them while presenting virtual content. In some examples, one or more displays (e.g., display 112) may be used to present the virtual content to the user. In some examples, the mixed reality system may include one or more mixed reality devices.

[0028] In some examples, the mixed reality device 110 may include one or more sensors for generating sensor data. The sensor data can be used to identify the gesture (e.g., position or orientation) of the mixed reality device 110. As an example, the sensor data may be collected by a camera capable of providing one or more images of the physical environment. The physical environment may include an area within the user's field of vision. In some implementations, the physical environment may also include an area outside the user's field of vision. In such implementations, one or more cameras may be used to image the area outside the user's field of vision. Although... Figure 1 The sensor shown is located in a specific position, but sensors can be located in different positions. Furthermore, there can be more or fewer sensors.

[0029] In some examples, one or more sensors may be normally open sensor 114 or vision sensor 116. Normally open sensor 114 may be a low-power sensor capable of detecting one or more properties of the physical environment. For example, normally open sensor 114 may be a temperature sensor, pressure sensor, flow sensor, level sensor, proximity sensor, displacement sensor, biosensor, gas sensor, chemical sensor, acceleration sensor, moisture sensor, humidity sensor, speed sensor, mass sensor, tilt sensor, force sensor, viscosity sensor, or any combination thereof.

[0030] In some examples, the normally open sensor 114 can be configured to operate in one or more data generation rate modes (sometimes referred to as power modes), which specify the rate at which the normally open sensor 114 generates data. For example, the normally open sensor 114 can be configured to operate in a low-rate mode, which causes the data generation rate to be lower than the data generation rate when the normally open sensor 114 is configured to operate in a high-rate mode. In some examples, different normally open sensors 114 can be configured to operate at different rates to generate data. For example, a first normally open sensor can be configured to operate at a first rate, and a second normally open sensor can be configured to operate at a second rate. In such examples, the first rate and / or the second rate can be variable rates (e.g., varying over time).

[0031] A vision sensor 116 (e.g., a camera) can capture images of the physical environment. The vision sensor 116 can also be configured to operate in one or more modes that specify the rate at which the vision sensor 116 captures images. For example, the vision sensor 116 can be configured to operate in a low-rate mode, which causes the image capture rate to be lower than the image capture rate when the vision sensor 116 is configured to operate in a high-rate mode.

[0032] In some examples, the mixed reality device 110 may include one or more combined sensors (e.g., combined sensor 118), which may include the functionality of a normally open sensor 114 and a vision sensor 116. In such examples, the combined sensor 118 may be configured to operate in one or more modes. For example, a first mode of the combined sensor 118 may correspond to the functionality of the normally open sensor 114. A second mode of the combined sensor 118 may correspond to the functionality of the vision sensor 116. In some examples, different modes may provide different rates of data acquisition. In one example, the vision sensor functionality may operate at a frame rate of 50 frames per second in the first mode and at a frame rate of 5 frames per second in the second mode. In another example, the vision sensor functionality may operate at a frame rate of 5 frames per second in the second mode, while the normally open sensor functionality may operate at a rate of 10 senses per second.

[0033] The mixed reality system may also include one or more processors (e.g., a visual processing unit (VPU) 120, a CPU 130, and / or a GPU 140). It should also be noted that while a single VPU 120, CPU 130, and GPU 140 are described, one or more VPUs 120, one or more CPUs 130, and / or one or more GPUs 140 may be used. The VPU 120 may be included in or located away from the mixed reality device 110. In some examples, the VPU 120 may be configured to operate in one or more power modes (e.g., low and high). In some examples, a VPU 120 operating in any power mode may consume less power than a CPU 130.

[0034] In some examples, CPU 130 may be included in or located away from the mixed reality device 110. When CPU 130 is located away from the mixed reality device 110, it may be located, for example, in a battery pack used by the mixed reality device 110. In some examples, CPU 130 may consume more power than VPU 120 and / or GPU 140.

[0035] In some examples, GPU 140 may be included in or located away from mixed reality device 110. When GPU 140 is located away from mixed reality device 110, it may be situated within, for example, a battery pack used by mixed reality device 110. In some examples, GPU 140 may be configured to operate in one or more power modes (e.g., low and high). In such examples, GPU 140 operating in any power mode may consume less power than a CPU (e.g., CPU 130).

[0036] In some examples, VPU 120 may include a perception front-end module 122. Perception front-end module 122 may receive sensor data from one or more sensors of the mixed reality device 110 (e.g., normally open sensor 114, vision sensor 116, or combined sensor 118). Perception front-end module 122 may use the sensor data to identify the posture of the mixed reality device 110. For example, posture may be identified using data from normally open sensor 114, vision sensor 116, combined sensor 118, or any combination thereof. In some examples, posture may define the position and / or orientation of the mixed reality device 110 (e.g., x-axis, y-axis, application, pitch, yaw, roll, or any combination thereof).

[0037] In some examples, the perceptual front-end module 122 can also identify one or more features of the physical environment (sometimes referred to as extracted information). For example, corners or edges of objects in the physical environment can be identified. One or more features of the physical environment allow the VPU 120 to send one or more features to other processors instead of sending an image of the physical environment. In some examples, one or more features can describe the physical environment without necessarily including all the pixel information contained in an image. In such examples, one or more features can be used to create a representation of the physical environment for transfer between components.

[0038] In some examples, VPU 120 may also include a normally open sensing module 124. In such examples, the normally open sensing module 124 may receive pose and / or one or more features from the sensing front-end module 122. In some examples, the normally open sensing module 124 may send pose to CPU 130 and / or GPU 140. In some examples, the normally open sensing module 124 may switch CPU 130 and / or GPU 140 from a low-power mode to a high-power mode in response to receiving pose from the normally open sensing module 124. In some examples, the normally open sensing module 124 may also send one or more features to CPU 130 and / or GPU 140.

[0039] In some examples, one or more functions of the sensing front-end module 122 or the normally open sensing module 124 may occur on different components of the VPU 120 (e.g., posture may be determined by the normally open sensing module 124). In other examples, one or more functions of the sensing front-end module 122 or the normally open sensing module 124 may occur on components (e.g., modules or devices) remote from the VPU 120.

[0040] As described above, the CPU 130 may, for example, receive the posture and / or one or more features of the physical environment of the mixed reality device 110 from the VPU 120. In some examples, the CPU 130, alone or in combination with the VPU 120, may create a representation of the physical environment. In such examples, the representation may be a 3D reproduction of the physical environment.

[0041] In some examples, CPU 130 can identify virtual content to be placed into a representation of the physical environment. In such examples, the virtual content can be identified based on an application running on CPU 130. For example, a game can be run on CPU 130, controlling the experience of a user using mixed reality device 110. In some examples, a world view can be generated that includes a representation of the physical environment and virtual content. In such examples, the world view can be virtual content inserted into a representation of the physical environment, such as defining where the content should be located in the physical environment. In some examples, the virtual content can be a three-dimensional representation.

[0042] In some examples, CPU 130 can determine how the virtual content should be viewed at a specific time. In such examples, CPU 130 can associate one or more time indicators with the virtual content, allowing GPU 140 to determine when to render the virtual content and send it to one or more displays of the mixed reality device 110 for presentation to the user.

[0043] In some examples, the content may have predictable motion over a period of time (e.g., a clock, clouds, the surface of a lake, etc.). In such examples, CPU 130 may identify and generate content over one or more time periods. One or more time periods may be associated with predictable content. In an illustrative example, the predictable content may be a clock mounted on a wall. The content representing the clock may include time periods during which the clock can be re-rendered (e.g., once per minute) so that the clock appears to the user of the mixed reality device 110 at the correct time. In some examples, when the digital information represents a static object (e.g., a table), the digital information does not have an associated time period for rendering content.

[0044] In some examples, CPU 130 can send a representation of the physical environment, virtual content, and / or a world view to GPU 140. In such examples, virtual content can be in the form of pixel information.

[0045] In some examples, GPU 140 may include a world operator 142, a perceptual view module 144, a content cache 146, a rendering module 148, a world interaction module 152, or any combination thereof. In such an example, content cache 146 may receive and store data received from CPU 130 (such as content to be rendered by the aforementioned mixed reality device 110 and world view) or data received from VPU 120 (such as gestures). The representation of the physical environment and the content to be rendered by mixed reality device 110 and / or world view can be accessed and consumed separately in content cache 146.

[0046] In some examples, the world manipulator 142 may request a world view from the content cache 146. In such examples, the content cache 146 may send the world view stored in the content cache 146 to the world manipulator 142 in response to a request. In other examples, the content cache 146 may send the world view to the world manipulator 142 periodically, or the content cache 146 may send the world view to the world manipulator 142 when the world view is updated (e.g., via the CPU 130). In some examples, the world view may be only a part of the physical environment associated with the mixed reality device 110. When the world view is a partial world view, the world manipulator 142 may use information received directly from the VPU 120 and / or based on other information stored in the content cache 146 to generate a complete world view.

[0047] In some examples, the world manipulator module 142 may receive pose and / or one or more features from the normally open sensing module 124. In some examples, the world manipulator module 142 may update the world view stored in the content cache 146 based on pose and / or one or more features.

[0048] Using gestures and a world view, the world manipulator module 142 can identify the content to be acquired. In some examples, the world manipulator 142 can also determine the gesture of the virtual content corresponding to the gesture of the mixed reality device 110. For example, when the mixed reality device 110 is facing a specific direction at a specific location, the world manipulator module 142 can determine the position and / or orientation of the virtual content.

[0049] In some examples, when the world manipulator module 142 identifies content, it can also determine the type of content (e.g., whether the content is dynamic or predictable).

[0050] Dynamic content can include objects with motion that is not easily captured by a logically operable GPU. In some examples, dynamic content may depend on user interaction and / or the specific pose of the mixed reality device 110. In such examples, the CPU 130 will need to analyze the interaction and / or pose to determine the content. In some examples, dynamic content includes unpredictable content. In such examples, dynamic content can be content with a visual appearance that changes over time. Additionally, dynamic content can include content that the GPU 140 may not be able to render effectively using existing cached content and input from the always-on perception module 124, which can be provided to the world manipulator module 142. Examples of such dynamic content could be chess game applications or messaging applications where updated rendering depends on input from a remote user. Another example of dynamic content could be complex 3D structures where the GPU 140 may not have sufficient storage to cache information in the content cache 146 and determine the location of the complex 3D structure.

[0051] Predictable content can include objects that move in a standard or predictable manner over time. In some examples, predictable content can be content with a spatial and temporal visual appearance, which can be determined using existing cached content in content cache 146 and input from normally open perception module 124, which can be provided as input to world manipulator module 142. For example, predictable content can have motion that depends on its own physics and physical environment rather than on external things (e.g., user interaction). In some examples, predictable content can have its motion pre-programmed so that the additional processing power required to determine its motion is unnecessary. In such examples, the motion of predictable content can be included in content cache 146, so that CPU 130 is not required. Predictable content can also include static content that does not move over time. In some examples, static content can still move according to the orientation of the static content, caused by changes or updates in the pose of the mixed reality device and the pose of the static content.

[0052] Examples of predictable content include clocks, flowing water, and torches. To illustrate an example of predictable content, a clock will be described. While a clock is not stationary, its movement is predictable. In fact, the pixel information associated with the clock can be stored in content cache 146. Furthermore, the logic for identifying how the clock should move can also be included in content cache 146. In some examples, the logic could correspond to the movement of the clock hands every minute. Another example of predictable content is the surface of a lake when there is no external turbulence. The wave characteristics of the surface can be reconstructed based on simple randomization. For predictable content, the predictable content can be stored in content cache 146 so that when displaying predictable content to a user, CPU 130 does not need to generate the pixel information associated with the predictable content. In such an example, CPU 130 does not need to switch from a low-power mode to a high-power mode, thus saving power.

[0053] In some examples, the type of content can be determined by identifying whether the pose content for the mixed reality device 110 is located in the content cache 146. In such examples, the world manipulator module 142 can also identify whether the content cache 146 contains content corresponding to the pose of the mixed reality device 110. If the world manipulator module 142 determines that the content is dynamic, it can send a message to the CPU 130 to switch from a low-power mode to a high-power mode, thereby generating the state of the dynamic content. In such examples, the message may include a world view. In some examples, the world view included in the message can instruct the CPU 130 what content it needs to generate. Limiting the content that the CPU 130 needs to generate can reduce the amount of time the CPU 130 is in high-power mode.

[0054] In response to receiving a message, CPU 130 can analyze the pose received from normally open perception module 124. In some examples, CPU 130 can also analyze the world view received from world manipulator module 142. By analyzing the pose and / or world view, CPU 130 can determine the content to be rendered by mixed reality device 110. In such an example, CPU 130 can generate pixel information associated with the content and send it to content cache 146 of GPU 140.

[0055] In some examples, the world manipulator module 142 can determine that determining the world view is no longer necessary for a certain period of time or in a specific area. In such examples, the world manipulator module 142 can use the mixed reality device 110 to send one or more frames of pixel information to the rendering module 148 to be displayed. In some examples, the world manipulator module 142 can also send an indication to the rendering module 148 of when to display each of the one or more frames. In such examples, the GPU 140 can avoid determining the world view and / or redirecting content. In some examples, the indication can correspond to both time and a specific pose. In such examples, when both time and a specific pose are met, the rendering module 148 can cause the mixed reality device 110 to display the frame.

[0056] In some examples, the world manipulator module 142 can send the world view to the perception view module 144. In such examples, the world manipulator module 142 can also send the pose of the mixed reality device 110 to the perception view module 144. In some examples, the perception view module 144 can retrieve the content indicated in the world view from the content cache 146 based on the pose.

[0057] In some examples, the perceptual view module 144 can modify the content based on the pose, ensuring that the content is correctly oriented for rendering by the mixed reality device 110. After the perceptual view module 144 has modified the content to correspond to the pose, the modified content can be sent to the rendering module 148. The rendering module 148 can render the modified content so that it can be displayed using the mixed reality device 110 (e.g., ensuring that the modified content is glued to the physical environment). In some examples, the rendering module 148 can align the modified content with a representation of the physical environment to ensure that the modified content is in the correct position.

[0058] As described above, in some examples, GPU 140 may also include a world interaction module 152. World interaction module 152 can receive events 150. Events 150 can affect content presented by the mixed reality device 110. In some examples, world interaction module 152 can send events 150 to world manipulator module 142 for processing. As described above, gestures can be sent from normally open perception module 124 to world manipulator module 142. Therefore, processing performed by world manipulator module 142 can utilize events 150 and / or gestures.

[0059] In some examples, event 150 may be associated with an action that is linked to data received by one or more sensors (e.g., normally open sensor 114 and / or vision sensor 116), one or more remote devices, etc. The event may be a gesture. Gestures may include movements of a user's appendages (e.g., moving a finger from left to right to indicate a swipe). As an example of an action associated with data received by one or more remote devices, a signal from a totem may be event 150. The totem may be a device that sends event 150 to GPU 140. In some examples, the totem may include a communication module (e.g., Bluetooth). Examples of totems are controllers or Internet of Things (IoT) devices. To illustrate an IoT device, an alarm device in a home may be described. For example, an alarm device may indicate that someone is at the door. Based on the indication from the alarm device, GPU 140 may render a notification of the person to be displayed by mixed reality device 110.

[0060] Figure 2 An example of a process 200 for rendering mixed reality, performed by a VPU (e.g., VPU 120) according to one embodiment, is shown. Process 200 includes receiving sensor data (e.g., images, GPS location, etc.) from one or more sensors (205). The one or more sensors may be included in a mixed reality device. In some examples, the sensor data may include data generated by the one or more sensors. The one or more sensors may include normally open sensor 114 (e.g., normally open sensor 114), a vision sensor (e.g., vision sensor 116), and / or a combination sensor (e.g., combination sensor 118). In some examples, the sensor data may be received by a perception front-end module of the VPU (e.g., perception front-end module 122). In some examples, the one or more sensors may be configured to operate in a high-power mode (e.g., vision sensor 116 may operate at a high frame rate in a high-power mode and at a low frame rate in a low-power mode).

[0061] Process 200 also includes identifying the gesture of the mixed reality device using sensor data (210). In some examples, the gesture may be generated by a perception front-end module. In such examples, the perception front-end module may send the gesture to a normally open perception module (e.g., normally open perception module 124). In some examples, the gesture at the current time may differ from the gesture when content is presented because components of the mixed reality system may take time to process. In such examples, the gesture may be associated with a future time. Specifically, the gesture may be a prediction of the mixed reality device's field of view at the time when content is presented using the mixed reality device.

[0062] Process 200 also includes sending the poses of the mixed reality device to a CPU (e.g., CPU 130) and / or a GPU (e.g., GPU 140) (215). In such an example, the poses may be sent to the world manipulator module of the GPU (e.g., world manipulator module 142). In some examples, the poses may be sent to both the CPU and the GPU simultaneously. In other examples, the poses may be sent to one of the CPU and the GPU and then to the other.

[0063] Process 200 also includes identifying one or more features of the physical environment using sensor data (220). For example, corners or edges of objects in the physical environment can be identified. In some examples, one or more features may relate to the field of view associated with the pose and / or one or more areas not included in the field of view. One or more features of the physical environment may allow the VPU to send one or more features to other processors instead of sending an image of the physical environment. In some examples, one or more features may describe the physical environment without necessarily including all pixel information that the image would include. In such examples, one or more features may be used to create a representation of the physical environment for transfer between components. Process 200 also includes sending one or more features to the CPU (225).

[0064] Figure 3 An example of a process 300 for rendering mixed reality, executed by a CPU (e.g., CPU 130) according to one embodiment, is shown. In such an example, the CPU may begin by switching from a low-power mode to a high-power mode. In other examples, the CPU may already be in a high-power mode. In some examples, process 300 may occur after process 200. Process 300 may assume that the initial recognition of the content rendered using the mixed reality device indicates that the content is dynamic and not stored in the content cache (e.g., content cache 146) of the GPU (e.g., GPU 140).

[0065] Process 300 includes generating a representation of the physical environment using one or more features (330) of the physical environment and / or one or more images of the physical environment. The one or more features may describe the physical environment without necessarily including all pixel information that the images would include. The representation may be a computer simulation of the three-dimensional physical environment. In some examples, a CPU and / or a VPU (e.g., VPU 120) may generate the representation.

[0066] Process 300 also includes determining the content to be presented along with the physical environment (335). In some examples, the content may be virtual content overlaid on a view of the physical environment, making the content appear to be in the physical environment. In some examples, an application executing on the CPU may be used to determine the content. The application may determine the content based on the representation and / or pose of the mixed reality device.

[0067] Process 300 also includes updating the representation with content (340). In some examples, the representation can be updated by inserting specific content into it. In other examples, the representation can be updated by associating location information corresponding to the representation with the content, such that the content can include information about the correct position and orientation that needs to be placed in the representation.

[0068] Process 300 also includes sending content and an updated representation to the GPU (345). In some examples, the CPU may send content to the GPU. In such examples, the content may be pixel information that allows the GPU to render content for a mixed reality device. Process 300 also includes switching the CPU to a low-power mode (350). In some examples, the CPU may switch to a low-power mode in response to the content being sent and the updated representation.

[0069] Figure 4 An example of a process 400 for rendering dynamic content using a mixed reality device (e.g., mixed reality device 110) is shown. At the beginning of process 400, the CPU (e.g., CPU 130) and / or VPU (e.g., VPU 120) may be set to operate in a low-power mode.

[0070] Process 400 includes receiving the pose (405) of a receiving device (e.g., a mixed reality device). In some examples, the pose may be received by a world manipulator module (e.g., world manipulator 142) of a GPU (e.g., GPU 140). In such examples, the pose may be received from a VPU. In some examples, the pose may be identified using sensor data of the physical environment surrounding the mixed reality device as described above. In such examples, the pose may be the orientation and / or position of the mixed reality device.

[0071] Process 400 also includes obtaining a representation of the physical environment (410). In some examples, the world manipulator module can obtain the representation. In such examples, the representation of the physical environment can be stored in the GPU's content cache (e.g., content cache 146). In some examples, the representation can be the updated representation described above.

[0072] Process 400 also includes identifying the content to be presented by the mixed reality device (415). In some examples, identification may be performed by the GPU's world manipulator module and / or perceptual view module (e.g., perceptual view module 144). Identification may include a portion of a representation corresponding to a pose. For example, the mixed reality device's field of view may be a portion of the representation. In such an example, the pose may indicate the direction the mixed reality device is facing.

[0073] Once a portion of the representation is identified, the World Manipulator Module and / or the Perceptual View Module can identify the content within that portion of the representation. If the World Manipulator Module identifies the content, it can send one or more instructions regarding the content, along with the content's pose or the pose of the mixed reality device, to the Perceptual View Module. If the Perceptual View Module identifies the content, it may have already received the representation and / or pose to identify the content.

[0074] Process 400 also includes determining that the content is dynamic (420). In some examples, the dynamic nature of the content can be determined based on one or more indications included in the representation. In other examples, the dynamic nature of the content can be determined based on whether the content is located in a content cache. In some examples, the dynamic nature of the content can be determined based on whether a world interaction has occurred. In such examples, the world interaction (e.g., event 150) can be received and identified by the GPU's world interaction module (e.g., world interaction module 152). If a world interaction has occurred, the dynamic nature of the content can be determined.

[0075] Process 400 also includes sending a message (425) to the CPU. In some examples, the message may be sent in response to determining that the content is dynamic. In some examples, the message may include representations and / or gestures. In some examples, the message may cause the CPU to perform... Figure 3 One or more steps described herein include determining the content to be rendered and sending the content to the GPU.

[0076] Process 400 also includes receiving content (430) by the GPU. In some examples, the content may be received by a content cache. In such examples, the content may be determined by the CPU. In some examples, the perceptual view module may obtain the content from the content cache after it has been received. In some examples, the content may be obtained by sending an instruction to the content cache. In such examples, the instruction may include the pose of the content. In other examples, the content may be a 3D representation of the content in a default pose. In such examples, the perceptual view module may change the pose of the content to correspond to the pose of the mixed reality device. In some examples, the perceptual view module may also ensure proper occlusion between the first content and the second content, as well as between the first content and the physical environment. Occlusion may describe when an object in the physical environment will at least partially obstruct a view of the content to be overlaid on the physical environment.

[0077] Process 400 also includes aligning the content with a representation of the physical environment (435). Alignment may include positioning the content within the field of view of the mixed reality device such that the content appears in the position corresponding to the representation. In other words, alignment can make the content stick relative to the physical environment. In some examples, alignment may be performed by the GPU's rendering module (e.g., rendering module 148). Process 400 also includes sending the aligned content to the mixed reality device (440) and using the mixed reality device to render the aligned content (445).

[0078] Figure 5 An example of a process 500 using a mixed reality device to present predictable content is shown according to one embodiment. Process 500 includes providing a mixed reality device. The mixed reality device may include a VPU (e.g., VPU 120), one or more sensors (e.g., normally open sensor 114, visual sensor 116, and / or combined sensor 118), and a display (505). In some examples, the sensors may be cameras, accelerometers, gyroscopes, totems, or GPS. Other sensors may be used.

[0079] Process 500 also includes acquiring sensor data associated with the physical environment (510). In some examples, the sensor data may be received from normally open sensors (e.g., normally open sensor 114), vision sensors (e.g., vision sensor 116), combined sensors (e.g., combined sensor 118), or any combination thereof. In some examples, the sensor data may be acquired by a sensing front-end module of the VPU (e.g., sensing front-end module 122).

[0080] Process 500 also includes using sensor data to determine the pose of the mixed reality device (515). In some examples, the pose can be within the configuration space of the mixed reality device. In such examples, the pose can define the position and orientation of the mixed reality device.

[0081] Process 500 also includes sending the pose of the mixed reality device to the GPU (e.g., GPU 140) (520). In some examples, the GPU may be located remotely from the device. Process 500 also includes identifying content corresponding to a representation of the pose and / or physical environment (525). In some examples, the content may be identified as predictable content. In such examples, the content may be identified as predictable content based on one or more indicators in the representation. In other examples, the content may be identified as predictable content based on whether the content is stored in the GPU's content cache (e.g., content cache 146) before being identified by the GPU.

[0082] In some examples, this indication can suggest that content should not be rendered. In such examples, steps 530 to 540 can be skipped. Additionally, in some examples, a duration for which content should not be rendered can be determined. In such examples, the duration can be temporal or spatial.

[0083] To illustrate when content should not be displayed, we can describe a user's behavior in the living room. When a user walks towards the bathroom, it can be determined that there will be no digital content for the next 10 meters. During this time, everything can be idle except for what is needed to determine if the 10 meters have been passed.

[0084] Process 500 also includes aligning the content with a representation of the physical environment (530), sending the aligned content to a mixed reality device (535), and rendering the aligned content using the mixed reality device (540). In some examples, aligning the content may include determining an orientation for displaying the content relative to the mixed reality device, and modifying the content based on the orientation. In some examples, the content may be modified by a perceptual view module of the GPU (e.g., perceptual view module 144). Modification may include configuring the content for the correct orientation for display.

[0085] In some examples, the VPU can determine to receive sensor data at a slower rate. In such examples, one or more sensors can be configured to operate in a low-power mode. In some examples, the VPU can configure one or more sensors to operate in a low-power mode. In some examples, a low-power mode for one or more sensors can cause the sensors in one or more sensors to generate less data (at a lower rate). In some examples, a low-power mode for one or more sensors can cause the sensors in one or more sensors to not generate data. Additionally, in some examples, a set of one or more sensors can be idle when a set of sensors is not needed.

[0086] It should be understood that, Figure 5 The specific steps shown provide a particular method for presenting mixed reality content using a mixed reality device according to one embodiment. According to alternative embodiments, other orders of steps may also be performed. For example, alternative embodiments of this disclosure may perform the steps outlined above in a different order. Furthermore, Figure 5 The steps shown may include multiple sub-steps, which can be performed in various orders suitable for the respective steps. Furthermore, additional steps may be added or removed depending on the specific application. Many variations, modifications, and substitutions will be recognized by those skilled in the art.

[0087] Processes 200, 300, 400, and 500 are shown as logic flowcharts, whose operations represent the sequence of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the operation. Typically, computer-executable instructions include routines, programs, objects, components, data, structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the operations described can be combined in any order and / or in parallel to implement the process.

[0088] Furthermore, processes 200, 300, 400, and 500 can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) executed jointly by hardware or a combination thereof on one or more processors. The code can be stored on a computer-readable storage medium, for example, in the form of a computer program comprising one or more instructions executable by one or more processors. The computer-readable storage medium can be non-transitory.

[0089] Figure 6-10Possible configurations of components of a mixed reality system according to various embodiments are described. While specific details of each possible configuration are provided, those skilled in the art will recognize that other configurations of the components are covered in this disclosure.

[0090] Figure 6 An example block diagram of a central processing unit (CPU) 600 according to one embodiment is shown. The CPU 600 can execute software instructions stored in local and / or external memory. The CPU 600 includes at least one or more of the following: a control component (e.g., control component 608), one or more functional units (e.g., functional unit 604), memory (e.g., cache 612 and random access memory (RAM) 616), or any combination thereof. The control component 608 can direct the operation of the CPU 600 by controlling input and output (I / O), instruction scheduling, decoding, load / store operations, and passing information on the CPU 600. Furthermore, the control component 608 can coordinate instruction processing via an internal clock and generate timing signals using functional unit 604 (e.g., a synchronization functional unit).

[0091] Functional unit 604 may represent the basic functions of CPU 600. In some examples, functional unit 604 may be a unit of operations and / or calculations performed according to software instructions, and may vary in type and capability. For example, functional unit 604 may be an arithmetic logic unit (ALU), a floating-point unit (FPU), a memory interface unit, etc. An ALU may perform one or more basic arithmetic and / or bitwise operations on one or more integers based on binary numbers. Conversely, an FPU may perform one or more arithmetic and / or bitwise operations on one or more floating-point numbers (e.g., approximations of real numbers). A memory interface unit may provide the control component 608 with an interface to external memory such as RAM or a hard disk drive (HDD) and an interface to onboard memory such as cache 612 and RAM 616. In some examples, functional unit 604 may include a group of similar elements such as one or more ALUs, one or more FPUs, one or more memory interface units, etc. In other examples, functional unit 604 may include any combination of different types of the aforementioned functional units, such as, but not limited to, two ALUs, FPUs, and memory interface units. Typically, CPU 600 can be a general-purpose execution unit with low latency, capable of executing any type of instruction. Furthermore, while CPU 600 is depicted with four functional units 604, CPU 600 may include any number of functional units 604 without departing from the spirit and scope of this disclosure.

[0092] In some examples, functional unit 604 cannot process program instructions stored in permanent memory (e.g., RAM 616). In such examples, control unit 608 can decode software instructions stored in memory and pass one or more individual operators, operands, and / or control signals to the functional unit according to the instruction set architecture of CPU 600. For example, control unit 608 can analyze software instructions according to the instruction set architecture and determine which one or more bits correspond to operators and which operators are defined by one or more bits (e.g., addition, subtraction, etc.). Control unit 608 can similarly manipulate operands and / or control signals. In some examples, the instruction set architecture may require fixed-length instructions (e.g., all instructions have the same number of bits); however, in other examples, control unit 608 can be used to decode variable-length instructions (e.g., the first instruction may include a different number of bits compared to the second instruction). When the instruction set architecture enables variable instruction length, instructions may include multiple bytes and may include more or fewer bytes than other instructions. For example, the first instruction may include four bytes (i.e., 32 bits), and the second instruction may include two bytes (i.e., 16 bytes). The control component 608 can identify the end of a variable instruction by recognizing a special character in the last byte. The CPU 600 can then anticipate instructions based on the instruction set architecture in order to appropriately decode and / or execute software instructions.

[0093] The ALU and FPU can include at least four inputs representing two operands, an operator, and at least one control signal (e.g., a timing signal). The timing signal allows the control component 608 to manage the inputs as the functional units process them. For example, the control component 608 can use the timing signal to force the ALU (or other functional units) to operate simultaneously, in a specific order, asynchronously, or in any other manner that software instructions or the control component 608 may require. The ALU or FPU can output the result of an arithmetic operation and at least one control signal. The output control signal can indicate the state of the CPU 600 and can be used by the ALU or FPU (as input) in subsequent calculations, passed to another ALU for related operations, or stored in a register. For example, the output signal can indicate that the arithmetic result is negative, zero, or overflow (e.g., the operation results in a quantity too large to be represented by the functional unit).

[0094] Control component 608 can fetch, decode, and / or schedule instructions from memory (e.g., cache 612 and RAM 616). Control component 608 can operate the instruction pipeline to schedule one or more decoded instructions on each functional unit in a parallel or semi-parallel manner. For example, a software program with three calculations can be executed at least partially in parallel: x = 5 + 6, y = 1 + 2, and z = x + y. Because x = 5 + 6 and y = 1 + 2 are independent, the x and y calculations can be executed in parallel. However, the z calculation must wait for the first two equations to complete before it can be executed. The instruction pipeline can schedule all three instructions and complete them in as few as two cycles. Control component 608 can manage the scheduling of instructions to maximize the number of functional units in the operation at once, thereby minimizing the total time required to produce the final result.

[0095] In some examples, the general flow of control component 608 may include fetching one or more instructions from memory, decoding and parsing one or more instructions, developing an instruction schedule (e.g., via an instruction pipeline), passing one or more instructions to one or more functional units 604 for processing according to the instruction schedule along with one or more timing signals, receiving results from one or more functional units 604, and storing the results back to memory.

[0096] Before processing subsequent instructions, control component 608 may store intermediate results and one or more decoded instructions in cache 612 or RAM 616. However, control component 608 may store data according to a memory hierarchy, prioritizing one form of memory over another based on the type of data to be stored. In some examples, cache 612 may have lower cost (e.g., latency and power) due to its proximity to control component 608 and one or more functional units 604. In such examples, control component 608 may prioritize cache 612 over RAM 616 when data is urgently or frequently loaded. Because cache 612 can be finite and valuable, data not immediately needed may be stored in RAM 616 or in external non-volatile memory (not shown). For example, consider the calculation example above with only a single ALU. Control component 608 may take a first calculation (e.g., x = 5 + 6) and pass it to the ALU for processing and return the result to cache 612. Then, control component 608 can pass a second calculation (e.g., y = 1 + 2) to the ALU and store the result of the second sub-calculation in cache 612, and then retrieve the final calculation (e.g., z = x + y) along with the results from the first and second calculations. The result of the final instruction represents the completed instruction, which may not need to be executed immediately, and therefore can be stored in RAM 616 or externally in non-volatile memory.

[0097] Cache 612 can be any random access memory that provides reduced latency for one or more load and store operations. For example, cache 612 can be static random access memory (SRAM). In some examples, CPU 600 may include multiple cores (e.g., two cores, four cores, eight cores, or more), where each core may include a control component (e.g., control component 608), one or more functional units (e.g., functional unit 604), and a cache (e.g., cache 612). In such examples, CPU 600 may also include a monolithic cache (not shown) that can be shared by multiple cores.

[0098] In some examples, cache 612 may include one or more levels (e.g., L1, L2, L3, and L4), where L1 cache has the lowest access latency and L4 cache has the highest access latency. In some examples, L1 cache may represent cache 612, which is not shared by other cores of CPU 600 but is dedicated to a specific CPU core. In such examples, L2-L4 caches are part of a larger cache shared among multiple cores of CPU 600. L4 cache may be internal or external to CPU 600 and may include SRAM or any other type of RAM, such as dynamic RAM (DRAM) or embedded dynamic RAM (eDRAM). CPU 600 may include one or any number of cores and any number or type of memory, including cache memory with various levels.

[0099] Figure 7 A CPU 700 according to one embodiment is shown (e.g., Figure 6 An example block diagram of the internal components of CPU 700. As described above, the general flow of CPU 700 can be fetch, decode, and execute. The process of executing instructions can begin with program counter 708, which can control the execution order of one or more instructions in the program. The current program counter value of program counter 708 can be the address location of the instruction to be executed. The current program counter value can be passed to memory interface 716, which can access the internal and / or external memory of CPU 700. Memory interface 716 can access the address associated with the program counter value in memory 728, and pass instructions to instruction register 704. Memory 728 can be volatile and / or non-volatile memory. Instruction register 704 can be short-term memory with low access latency (e.g., ...). Figure 6 (Cache 612). If instruction decoding is required, instruction register 704 can pass the instruction to function unit 720 or instruction decoder 712.

[0100] Instruction decoder 712 manages the execution of functional unit 720 according to the instruction set architecture of CPU 700. Instruction decoder 712 receives a binary number of one or more bits from instruction register 704 and identifies the bits corresponding to the first operand, second operand, operator, and storage location. Different types of instruction set architectures can exist, each following different conventions. For example, for an instruction to add a constant to a variable in a Reduced Instruction Set Computing (RISC) architecture (where the instruction length is 32 bits), the first 6 bits may correspond to the operator (e.g., adding in this example), the next 5 bits may correspond to the storage address of the result, the next 5 bits may correspond to the first operand, and the last 16 bits may correspond to the constant. Instruction decoder 712 may expect the operator, one or more operands, and other aspects of the instruction to appear in a specific order. Instruction decoder 712 can operate on any instruction set architecture, as long as it knows the instruction set architecture. Once the operator and operands are decoded, they can be passed to functional unit 720, which can then execute the operation.

[0101] Functional unit 720 (e.g., ALU, FPU, etc.) can receive input from instruction decoder 712 and generate output. If the output represents the conclusion of an instruction, it can be passed to memory interface 716 (which stores the output in memory 728) or to instruction register 704 if the output is used by a subsequent instruction. If the instruction is incomplete, the output of functional unit 720 can be passed to accumulator 724. Accumulator 724 can be a register storing intermediate arithmetic and logic. Accumulator 724 can accumulate the output of functional unit 720 for one or more operations that require multiple passes through functional unit 720. Each iteration of functional unit 720 is passed to accumulator 724 so that it can be reused by functional unit 720 in subsequent iterations or passed to memory interface 716 for storage once completed. For example, if the ALU is executing the instruction y = 5^2, the ALU can process the instruction as y = 5 + 5 + 5 + 5 + 5; however, because the ALU can only use two operators per operand, the ALU can perform four addition operations. In such an example, the ALU can process 10 + 5 in the second iteration, 15 + 5 in the third iteration, and 20 + 5 in the last iteration. Once the final result (i.e., 25) is reached, the output from the accumulator can be stored in memory 728. Functional unit 720 and accumulator 724 can also handle multiplication using shift and addition in binary operations. When functional unit 720 processes the last iteration of the operation, the result can be passed to memory 728.

[0102] Once an instruction has been completed, the program counter 708 increments the instruction length according to the instruction set architecture (e.g., 16-bit, 32-bit, etc.) until the program counter value equals the memory address of the next instruction. Then, the program counter 708 can load the next instruction through the memory interface 716 and continue the above process until the program completes.

[0103] Figure 8 An example block diagram of a graphics processing unit (GPU) 800 according to one embodiment is shown. It should be noted that while the GPU is specifically referred to throughout this disclosure, a VPU may be used to replace the functionality of the GPU in some examples. In some examples, the GPU 800 may be a microprocessor. In such examples, the unique architecture of the GPU 800 may provide different processing power than a CPU (e.g., a CPU 600 / 700). The differences between the CPU 600 / 700 and the GPU 800 may involve optimizations for each hardware element. The CPU 600 / 700 may be designed to execute threads (e.g., discrete subroutines of an entire program) with low latency (from fetch to execution) by using a large cache 612. Conversely, the GPU 800 may be designed to execute a large number of similar instructions in parallel. For example, the GPU 800 may include 16 streaming multiprocessors (SMs) 816, where each SM 816 can execute a block of threads in parallel with another SM 816. In some examples, the GPU 800 may execute multiple blocks of threads at a time, equal to the number of SMs 816. In some examples, the GPU800 can perform a large number of repetitive operations more proficiently, maximizing the effectiveness of parallel SM 816. The GPU800 can include many SM 816s. Although Figure 8 The diagram shows 16 SM 816s; the GPU 800 can include any number of SM 816s.

[0104] In some examples, each SM can share a Level 2 (L2) cache 812. By sharing the L2 cache 812, when the first SM 816 and the second SM 816 are executed in parallel, the first SM 816 can use the intermediate results of the second SM 816. The L2 cache 812 can enable low-latency load / store operations to allow data to be transferred between different SMs 816.

[0105] In some examples, one or more memory controllers (e.g., memory controller 804) provide an interface between the SM 816, L2 cache 812, and external devices or systems (not shown) via host interface 808. Memory controller 804 may include random access memory (RAM) with higher latency than L2 cache 812 for storing leased or indefinitely used data. For example, memory controller 804 may include, but is not limited to, DRAM, SRAM, eDRAM, etc., or any combination thereof. In some examples, memory controller 804 may not include any memory, but instead enable GPU 800 to access external volatile and / or non-volatile memory, such as RAM, read-only memory (ROM), HDD, flash memory, etc. Additionally, memory controller 804 facilitates access to internal RAM or L2 cache 812 from hardware or software outside of GPU 800 via host interface 808.

[0106] In some examples, it can be obtained from, such as Figure 6 CPU600 or Figure 7 CPU700, Vision Processing Unit (VPU) (such as Figure 10 The host interface 808 receives instructions executed by the GPU 800 from an external source (such as RAM, ROM, HDD, flash memory, etc.) or external memory. The host interface 808 provides access to the GPU 800's computing resources to the external source by providing an interface. Information received from the host interface 808 can be passed to the memory controller 804, which can then pass relevant data to the L2 cache 812 or a single SM 816 as needed.

[0107] The GPU 800 differs from CPUs (e.g., CPU600 / 700) in its ability to execute a large number of threads concurrently. The memory controller 804 manages and coordinates the concurrent threads executing on the SM816 according to software instructions. For example, the GPU 800 might be responsible for updating a single frame of a 1080p video. In such an example, the GPU 800 might need to calculate the pixel value for each pixel of the frame (which could include 2,073,600 pixels). For such instructions, the GPU 800 must solve the same calculations for over 2 million pixels. The memory controller 804 can leverage the concurrent processing capabilities of the GPU 800 by differentiating thread blocks (e.g., one or more threads) within a single task, where each SM816 executes the thread block concurrently. Utilizing the parallel processing capabilities of multiple SM816s, the GPU 800 can provide computation to seamlessly render video at frame rates between 1 and 240 frames per second (or some other frame rate).

[0108] In some examples, the GPU 800 can save power by updating only a portion of the frame whose values ​​have changed. For example, if a portion of the first frame is black and the same portion of the second frame is the same color, the GPU 800 does not need to expend resources to compute and update the values ​​of the unchanged portion of the frame. In such examples, only the difference (the difference between frames) needs to be processed. The memory controller 804 can coordinate the threads executing on the SM816 to enable the parallel processing capabilities of the GPU 800, allowing the GPU 800 to update and display each frame according to the frame rate. In some examples, the memory controller 804 can operate similarly to gigathread, which represents a single unified thread that manages the individual threads executing on many SM816s. In some examples, the GPU 800 is a CPU dedicated to processing graphics.

[0109] Figure 9 A single streaming multiprocessor 900 (such as...) according to one embodiment is shown. Figure 8 An example block diagram of a streaming multiprocessor (SM 900) is shown. A single SM 900 may include multiple execution blocks (e.g., control components, caches, and one or more functional units may be a single execution block), where each execution block can execute a single thread. Figure 9 Four execution blocks are depicted: the first block includes control component 904, cache 908, and functional unit 912; the second block includes control component 916, cache 920, and functional unit 924; the third block includes control component 928, cache 932, and functional unit 936; and the fourth block includes control component 940, cache 944, and functional unit 948. The SM can include any number of execution blocks. In some examples, a single SM 900 can execute thread blocks by assigning each thread within a thread block to the execution block. In such examples, each thread block can be executed concurrently and completed within the execution time of a single thread. In some examples, the memory capacity of the caches for the execution blocks can be limited (e.g., cache 908 can be smaller than cache 952, which can be smaller than...). Figure 8 The L2 cache (812). The cache of the execution block can store one or more operators, operands, timing and / or status signals, or any combination thereof, passed to each functional unit (e.g., FU 912). The cache of the execution block can also store the results (intermediate or final) from each functional unit. Control components (similar to...) Figure 6The control component 608 can follow an instruction execution loop of fetching, decoding, and executing. Control component 608 can fetch one or more instructions from its caches (i.e., caches 908, 920, 932, 944) and decode one or more instructions according to the instruction set architecture used by a single SM 900. Any instruction set architecture can be adopted by the execution block, provided that the decoder in the control has the corresponding decoding capability to decode the instruction set architecture. The control component can pass the decoded instructions to the functional unit associated with it. For example, control component 904 can pass one or more decoded instructions to functional unit 912, while control component 916 can pass one or more decoded instructions to functional unit 924, and so on.

[0110] The results of a set of instructions (once decoded and executed by one or more functional units) can be stored in the execution unit's local cache before being passed to cache 952. In some examples, caches 908, 920, 932, and 944 can represent level 1 (L1) caches because each cache is not shared between other execution blocks. In such examples, cache 952 can represent a level 2 (L2) cache because it can be shared between many execution blocks. For example, host interface 956 can... Figure 8 The memory controller 804 receives thread blocks and stores the corresponding threads in cache 952 before dispatching them. In some examples, each thread in a thread block can be dispatched to each execution block. In some examples, a thread block can include the same number of threads as or fewer than the number of execution blocks in a single SM 900. The execution time of each thread block can be equal to the execution time of the longest individual thread. In other examples, a thread block can include more threads than a single SM 900 has execution blocks. In such examples, a single SM 900 can schedule threads between execution blocks in a manner that minimizes execution time, for example, by scheduling the shortest and longest threads executing on the same execution block.

[0111] The results of a completed thread can be stored in cache 952 and then passed to host interface 956 for delivery to memory controller 804. In some examples, intermediate results for each thread can be passed to each thread upon completion via host interface 956. In other examples, a single SM900 can wait until the entire thread block completes before passing the results of the entire thread block to memory controller 804.

[0112] Figure 10An example block diagram of a vision processing unit (VPU) 1000 is shown for managing input from one or more sensors, such as normally open sensor 114, vision sensor 116, combined sensor 118, video and still cameras, accelerometers, illumination time, etc. It should be noted that although the VPU is specifically mentioned throughout this disclosure, a GPU may be used instead of the VPU's functionality in some examples. The VPU 1000 may include a processor array to process instructions simultaneously. The processor array can be used to process large amounts of raw data by executing one or more instructions on the entire dataset at a time, rather than a single operand. In some examples, the processor in the processor array may be a vector very long instruction word (VLIW) processor 1004. The vector VLIW processor 1004 may operate on one or more data arrays referred to as vectors, rather than a CPU (e.g., ...). Figure 6 The CPU 600 can process scalars (e.g., single data). Additionally, the vector VLIW processor 1004 can use a VLIW instruction set architecture, where each instruction length can be 64 bits or longer. In some examples, the VLIW instruction set architecture can be a variable VLIW architecture, where instructions can have varying bit lengths. In some examples, the processor array can be a single SM, such as... Figure 9 Those shown and used by GPU 800.

[0113] The processors in the processor array may include internal cache memory, memory array 1008, L2 cache 1036, or any combination thereof. Memory array 1008 may represent a large number of interconnect nodes consisting of RAM (e.g., DRAM, SRAM, etc.), operating as a large shared memory library for each processor in the processor array, one or more vision hardware accelerators 1012, and one or more Reduced Instruction Set Computing (RISC) processors (e.g., RISC 1 1028 and RISC 2 1032). In some examples, the processor array may use tightly coupled address spaces to reduce load latency and increase storage. In some examples, one or more vision hardware accelerators 1012 may be dedicated hardware elements designed specifically for a particular type of computation, thereby reducing the computational load of that particular type on the processors in the processor array. Hardware accelerators (e.g., one or more vision hardware accelerators 1012) can provide fast processing of one or more specific instructions. In some examples, when a processor might be slow (or otherwise unsuitable for a particular instruction type) to execute one or more specific instructions, a hardware accelerator may outperform the processors in the processor array in executing one or more specific instructions. In some examples, the VPU 1000 may include up to 20 hardware accelerators that offload some processing from the processor array. In some examples, one or more vision hardware accelerators may be controlled by a RISC processor via the processor array.

[0114] Bus 1040 can connect to a processor array, L2 cache 1036, RISC 1 1028, RISC 2 1032, and one or more interfaces 1020. L2 cache 1036 can represent shared memory in the processor array, RISC 1 1028, and RISC 2 1032. Furthermore, one or more interfaces 1020 can store external data from input / output (I / O) interfaces 1024 in L2 cache 1036 for processing with one or more processors. In some examples, both RISC 1 1028 and RISC 2 1032 can be reduced instruction set computing processors controlling sub-components of VPU 1000. For example, RISC 1 1028 can manage one or more vision hardware accelerators 1012, while RISC 2 1032 can manage I / O interfaces 1024 and arbitrarily connected sensors (e.g., cameras and sensors).

[0115] In some examples, the control of the VPU 1000 can be distributed, where a RISC 1 1028 and / or a RISC 2 1032 can manage the I / O interface 1024, peripheral devices, and one or more vision hardware accelerators 1012. In some examples, a single RISC processor can manage the VPU 1000 by acting as both a RISC 1 1028 and a RISC 2 1032. One or more external sensors can be managed and controlled by the RISC processor to meet the needs of the VPU 1000. In some examples, the VPU 1000 can modify the rate of sensor input data from one or more external sensors. For example, the VPU 1000 can reduce the power consumption of a camera by changing the rate at which the camera captures video. As another example, the VPU 1000 can manage a camera capable of capturing 60 frames per second, causing it to capture 24 frames per second. In some examples, the VPU 1000 can reduce or increase the frame rate or data capture rate of any connected sensor, including but not limited to cameras, accelerometers, totems, position sensors, gesture sensors, etc.

[0116] In some examples, one or more interfaces 1020 can operate on one or more wired protocols and / or one or more wireless protocols. For example, one or more interfaces 1020 can connect to one or more external sources via: Universal Serial Bus (USB) (e.g., USB 2.0, USB 3.0, USB OTG, etc.), Joint Test Group (JTAG), Ethernet, Universal Asynchronous Receiver / Transmitter (UART), SD Mode (SDIO), etc. Additionally, one or more interfaces 1020 can enable data exchange for various display types, such as, but not limited to, LCD, Common Intermediate Format (CIF), NAL, etc. The VPU 1000 can also use a Mobile Industrial Processor Interface (MIPI) 1016 to provide one or more interfaces for one or more external devices (e.g., CPU 600, GPU 800, and / or sensors). MIPI 1016 can include one or more independent communication channels (e.g., 12). MIPI 1016 can provide connectivity from I / O interface 1024 to one or more vision hardware accelerators 1012. Connected external devices (e.g., sensors) can have low-latency throughput from I / O interface 1024 to one or more vision hardware accelerators. I / O interface 1024 can transfer data to and from one or more external devices (e.g., CPU 600, GPU 800, and / or sensors) and VPU 1000.

[0117] In some examples, the VPU 1000 can be a CPU dedicated to processing visual input. Additionally, although according to... Figure 10 The exemplary hardware components depicted are for the VPU 1000, but the VPU 1000 may include hardware components similar to or the same as those of a GPU (such as those combined with...). Figure 8 and 9 (Depicted and described GPU 800). In some examples, one or more operations performed by VPU 1000 can be performed by GPU 800.

[0118] The described features can be implemented in digital electronic circuits, or in computer hardware, firmware, software, or combinations thereof. The apparatus can be implemented in a computer program product tangibly embodied in an information carrier, for example, in a machine-readable storage device for execution by a programmable processor; and can be executed by a programmable processor executing instruction programs to perform the described implemented functions by manipulating input data and generating output. The described features can advantageously be implemented in one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive and transmit data and instructions from a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform an activity or produce a result. Computer programs can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0119] As an example, suitable processors for executing instruction programs include general-purpose and special-purpose microprocessors, as well as a single processor or one of several processors in any type of computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the processor for executing instructions and one or more memories for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data files; these devices include disks, such as internal hard disks and removable disks, magneto-optical disks, and optical disks. Suitable storage devices for tangibly representing computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented or incorporated into ASICs (Application-Specific Integrated Circuits).

[0120] To provide interaction with the user, features can be implemented on a computer having: a display device for displaying information to the user (such as a CRT (cathode ray tube), LCD (liquid crystal display), LED (light emission diode) monitor), a keyboard, and pointing devices (such as a mouse or trackball), through which the user can provide input to the computer.

[0121] These features can be implemented in computer systems that include back-end components (such as data servers), or in computer systems that include middleware components (such as application servers or internet servers), or in computer systems that include front-end components (such as client computers with graphical user interfaces or internet browsers), or any combination thereof. The components of the system can be connected via digital data communication of any form or medium (such as communication networks). Examples of communication networks include LANs, WANs, and the computers and networks that form the internet.

[0122] Computer systems may include clients and servers. Clients and servers are typically geographically separated and usually interact via a network, such as the network described above. The client-server relationship is established by computer programs running on individual computers and having client-server relationships with each other. Although some implementations have been described in detail above, other modifications are possible.

[0123] Furthermore, the logical flow depicted in the figures does not require the specific order or sequence shown to achieve the desired result. Additionally, other steps can be provided from the described flow, or steps can be eliminated from the described flow, and other components can be added to or removed from the described system. Therefore, other embodiments are within the scope of the following claims.

[0124] When a component is described as being configured to perform certain operations, such configuration can be achieved by, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits), or any combination thereof.

[0125] Many embodiments of this disclosure have been described. However, it should be understood that various modifications can be made without departing from the scope of this disclosure.

Claims

1. A method for presenting mixed reality content using a mixed reality device, the method comprising: The visual processing unit (VPU) of the mixed reality device receives sensor data associated with the physical environment, wherein the sensor data is received from one or more sensors; The VPU of the mixed reality device uses the sensor data to identify the posture of the mixed reality device; The gesture of the mixed reality device is sent from the VPU to the central processing unit (CPU); The VPU identifies one or more features of the physical environment based on the sensor data; Send one or more of the features to the CPU; The VPU uses one or more features of the physical environment to generate a representation of the physical environment; The representation of the physical environment is sent to the CPU; The CPU determines the content to be presented along with the physical environment; A world view is generated using the representation of the physical environment and one or more features of the physical environment; Use the content to update the representation of the physical environment; The CPU sends the content, the updated representation of the physical environment, and the world view to the graphics processing unit (GPU). The world operator of the GPU requests a world view from the GPU's content cache; The pose and / or one or more features are received from the normally open sensing module at the world manipulator of the GPU; The world manipulator of the GPU updates the world view based on the pose and one or more features to generate an updated world view; It was determined that the updated world view did not include dynamic content; and Switch the CPU from high power mode to low power mode.

2. The method according to claim 1, wherein, The one or more sensors include normally open sensors.

3. The method according to claim 1, wherein, The one or more sensors include vision sensors.

4. The method according to claim 1, wherein, The GPU includes a host interface, a plurality of streaming multiprocessors (SMs) coupled to the host interface, and one or more memory controllers coupled to the host interface and the plurality of SMs, wherein the CPU includes memory components for executing threads with low latency, and each of the plurality of SMs of the GPU is configured as a block for executing threads in parallel.

5. The method according to claim 1, wherein, The posture includes a prediction of the field of vision of the mixed reality device at a future time.

6. The method according to claim 1, wherein, The one or more features include the corners or edges of objects in the physical environment.

7. The method according to claim 1, further comprising: Before generating the representation of the physical environment, the CPU is switched from the low-power mode to the high-power mode.

8. The method according to claim 1, wherein, The representation includes a computer simulation of the physical environment in three dimensions.

9. The method according to claim 1, wherein, The content to be presented in conjunction with the physical environment includes virtual content.

10. The method of claim 9, further comprising: The virtual content is overlaid on the view of the physical environment.

11. The method according to claim 1, wherein, The CPU generates the content based on the representation of the physical environment.

12. The method according to claim 1, wherein, Updating the representation includes inserting the content into the representation.

13. The method according to claim 1, wherein, Updating the representation includes associating location information with the content, wherein the location information corresponds to the representation.

14. The method according to claim 1, wherein, The representation is not stored in the content cache before the content is determined.

15. The method according to claim 1, wherein, The posture of the mixed reality device includes the position and orientation of the mixed reality device.

16. The method according to claim 1, wherein, The GPU is located away from the mixed reality device.

17. The method according to claim 1, wherein, The sensor in the one or more sensors is one of the following: a camera, an accelerometer, a gyroscope, a totem, or a global positioning system.

18. The method according to claim 1, wherein, The GPU and the CPU are located in the mixed reality device.

19. The method according to claim 1, wherein, The GPU and the CPU are located in the battery pack.

20. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations: Obtain sensor data associated with the physical environment, wherein, The sensor data is obtained using one or more sensors; The vision processing unit (VPU) uses the sensor data to determine the device's posture. The gesture of the device is sent from the VPU to at least one graphics processing unit (GPU); The at least one GPU identifies the mixed reality content corresponding to the pose; The identified mixed reality content was determined to be dynamic content; A message is sent from the at least one GPU to a central processing unit (CPU) operating in a low-power mode, wherein the message causes the CPU to switch from the low-power mode to a high-power mode. The posture is sent from the VPU to the CPU; The dynamic content is received from the CPU at at least one GPU; The dynamic content is aligned with the physical environment by the at least one GPU; Sending the targeted dynamic content from the at least one GPU to the device; and Use the monitor to display the dynamic content being pointed at.

21. The computer program product according to claim 20, wherein, The posture is in the configuration space of the device, and the posture defines the position and orientation of the device.

22. The computer program product according to claim 20, wherein, The sensor in the one or more sensors is one of the following: a camera, an accelerometer, a gyroscope, a totem, or a GPS device.

23. The computer program product according to claim 20, wherein, The at least one GPU is located away from the device.

24. The computer program product according to claim 23, wherein, The at least one GPU and the CPU are located in the battery pack.

25. The computer program product according to claim 20, wherein, The at least one GPU and the CPU are located in the device.

Images