Mixed reality split-chip power loop control flow
The power control process in mixed reality split-chip architectures addresses thermal challenges by estimating chip power states and adjusting processing parameters, effectively managing heat and power consumption for improved performance in mixed reality devices.
Patent Information
- Application Number
- PCT/US2023/083887
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2023-12-13
- Publication Date
- 2024-11-21
AI Technical Summary
Mixed reality devices with split-chip architectures face thermal challenges due to increasing render resolutions, requiring effective power loop control to manage heat while preserving low latency in video capture and content display.
A power control process that estimates the power states of both front-end and back-end chips, adjusts the fovea region of interest, and updates sensor capture resolution and frame capture rate to reduce thermal throttling, thereby optimizing processing and reducing power consumption.
This solution enables thermal management in split-chip architectures by dynamically adjusting processing demands based on power states, reducing thermal issues and power consumption while maintaining low latency in mixed reality applications.
Smart Images

Figure US2023083887_21112024_PF_FP_ABST
Abstract
Description
MIXED REALITY SPLIT-CHIP POWER LOOP CONTROL FLOWCROSS-REFERENCE TO RELATED APPLICATION
[0001] The present application claims the benefit of India Patent Application No. 202341034479, filed on May 17, 2023, and titled ‘‘MIXED REALITY SPLIT-CHIP POWER LOOP CONTROL FLOW,” the disclosure of which is expressly incorporated by reference in its entirety.BACKGROUNDField
[0002] Aspects of the present disclosure relate to computing devices, and more specifically, to power loop control in a mixed real i ty split-chip architecture.Background
[0003] Mobile or portable computing devices include mobile phones, laptops, palmtop and tablet computers, portable digital assistants (PDAs), portable game consoles, and other portable electronic devices. Mobile computing devices are comprised of many electrical components that consume power and generate heat. The components (or compute devices) may include system-on-a-chip (SoC) devices, graphics processing unit (GPU) devices, neural processing unit (NPU) devices, digital signal processors (DSPs), and modems, among others.
[0004] Extended reality (XR) includes virtual reality (VR), mixed reality (MR), and augmented reality (AR). Augmented reality is the augmentation of the real (e.g., physical) world with virtual content. Mixed reality is an extension of augmented reality that allows real and virtual elements to interact in an environment, enabling users to see though the headset into a blend of the real physical world and the digital world. Mixed realiN may use one of two methods to display virtual content onto the real physical world: video see-through systems present video feeds from cameras, and optical see- through systems project virtual content onto glasses worn by the user. The augmentation can be accomplished with a wearable AR deuce capable of mapping the physical yvorld, localizing itself in that physical world, and positioning and rendering virtual content on a near-eye display visible to the user. Many such devices utilize handand / or fingertip tracking to allow users to control interfaces in augmented reality. It w ould be desirable to improve processing in MR devices.SUMMARY
[0005] In aspects of the present disclosure, a method of processing with a mixed reality split-chip architecture includes estimating a first power state of a front-end chip of the mixed reality split-chip architecture. The method also includes estimating a second power state of a back-end chip of the mixed reality split-chip architecture. The back-end chip is located remotely from the front-end chip. The method further includes adjusting a fovea region of interest based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
[0006] Other aspects of the present disclosure are directed to an apparatus. The apparatus has at least one memory and one or more processors coupled to the at least one memory. The processor(s) is configured to estimate a first power state of a frontend chip of a mixed reality split-chip architecture. The processor(s) is also configured to estimate a second power state of a back-end chip of the mixed reality split-chip architecture. The back-end chip is located remotely from the front-end chip. The processor(s) is further configured to adjust a fovea region of interest based on the first power state and the second power state, in response to detecting operation of the frontend chip in a low power mode and operation of the back-end chip in the low power mode.
[0007] Other aspects of the present disclosure are directed to an apparatus. The apparatus includes means for estimating a first power state of a front-end chip of a mixed reality split-chip architecture. The apparatus also includes means for estimating a second power state of a back-end chip of the mixed reality split-chip architecture. The back-end chip is located remotely from the front-end chip. The apparatus further includes means for adjusting a fovea region of interest based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
[0008] In other aspects of the present disclosure, a non-transitory computer-readable medium with program code recorded thereon is disclosed. The program code isexecuted by a processor and includes program code to estimate a first power state of a front-end chip of a mixed reality split-chip architecture. The program code also includes program code to estimate a second power state of a back-end chip of the mixed reality split-chip architecture, the back-end chip located remotely from the front-end chip. The program code further includes program code to adjust a fovea region of interest based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
[0009] This has outlined, rather broadly, the features and technical advantages of the present disclosure in order that the detailed description that follows may be better understood. Additional features and advantages of the present disclosure will be described below. It should be appreciated by those skilled in the art that this present disclosure may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. It should also be realized by those skilled in the art that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features, which are believed to be characteristic of the present disclosure, both as to its organization and method of operation, together with further objects and advantages, will be better understood from the following description when considered in connection with the accompanying figures. It is to be expressly understood, how ever, that each of the figures is provided for the purpose of illustration and description only and is not intended as a definition of the limits of the present disclosure.BRIEF DESCRIPTION OF THE DRAWINGS
[0010] For a more complete understanding of the present disclosure, reference is now made to the following description taken in conjunction with the accompanying drawings.
[0011] FIGURE 1 illustrates an example implementation of a host system-on-a-chip (SoC), in a mixed reality' split-chip architecture, in accordance with certain aspects of the present disclosure.
[0012] FIGURE 2 is a block diagram illustrating a mixed reality split-chip architecture, in accordance w ith various aspects of the present disclosure.
[0013] FIGURE 3 is a flow diagram illustrating a power control process for thermal throttling in a mixed reality split-chip architecture, in accordance with various aspects of the present disclosure.
[0014] FIGURE 4 is a flow diagram illustrating a process for adjusting a fovea region of interest based on a power state of a mixed reality split-chip architecture, in accordance with various aspects of the present disclosure.
[0015] FIGURE 5 is a block diagram showing an exemplary wireless communications system in which a configuration of the present disclosure may be advantageously employed.
[0016] FIGURE 6 is a block diagram illustrating a design workstation used for circuit, layout, and logic design of components, in accordance with various aspects of the present disclosure.DETAILED DESCRIPTION
[0017] The detailed description set forth below; in connection with the appended drawings, is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. It will be apparent, however, to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form in order to avoid obscuring such concepts.
[0018] As described, the use of the term “and / or” is intended to represent an “inclusive OR,” and the use of the term “or” is intended to represent an “exclusive OR.” As described, the term “exemplary” used throughout this description means “serving as an example, instance, or illustration,” and should not necessarily be construed as preferred or advantageous over other exemplary configurations. As described, the term “coupled” used throughout this description means “connected, whether directly or indirectly through intervening connections (e.g., a switch), electrical, mechanical, or otherwise,” and is not necessarily limited to physical connections. Additionally, the connections can be such that the obj ects are permanently connected or releasablyconnected. The connections can be through switches. As described, the term “proximate” used throughout this description means “adjacent, very near, next to, or close to.” As described, the term “on” used throughout this description means “directly on” in some configurations, and “indirectly on” in other configurations.
[0019] With increasing render resolutions of virtual reality / mixed reality head mounted displays (HMDs), thermal challenges arise. Split-chip solutions enable distribution of heat across the devices. According to aspects of the present disclosure, power loop control flow is employed for split-chip architectures. The power loop control flow may be employed, for example, in video see-through (VST) mixed reality scenarios. In some aspects, a power state is used for controlling resolution and regions of interest (ROI) for fovea processing. The resolution control and fovea control help with thermal throttling in mixed reality platforms while also preserving a low latency between video capture and content display.
[0020] A front-end chip includes an image signal processor (ISP) front-end, a video see-through (VST) sensor controller, and a power loop controller. The image signal processor front-end enhances images captured by a camera. To enhance these images, the image signal processor front-end performs, for example, demosaicing, noise reduction, red-green-blue (RGB) to YUV (where Y is luma, and U and V are chroma components) conversion, and exposure control. The video see-through sensor controller is a runtime component that inputs environment variables and outputs one or more virtual channels. The power loop controller communicates with the video see-through controller and the image signal processor front-end to modify their functionality based on inputs received from a back-end chip. The front-end chip may incorporate a camera chip in some implementations. The split-chip architecture may perform perception via, in part, the camera, indicating where a user is located in an environment. A RGB camera may be provided for streaming images of the real world to various components in the split-chip architecture. The split-chip architecture processes this stream to overlay virtual objects onto real objects.
[0021] The front-end chip performs camera processing (e.g., resolution and capture processing based on sensors) and transmits an image buffer to the back-end chip for rendering. The image buffer may include two channels: a fovea channel, and anon- fovea channel. The fovea region corresponds to where a user's eyes are looking, andthus, specifies a higher resolution than non-fovea regions, such as regions in the peripheral vision of the user. After rendering a scene, the back-end chip transmits the rendering to the front-end chip for display to the user.
[0022] The back-end chip includes an ISP post-processing block that performs functions such as resolution clean up as well as noise clean up. According to aspects of the present disclosure, based on an estimate of a current power state, the ISP postprocessing block and a perception block run a power control process to adjust a fovea region of interest (RO I). The perception block, which may be located on the front-end chip or the back-end chip, tracks the user’s position and head orientation. The render block produces a virtual content overlay for display to the user.
[0023] Aspects of the present disclosure introduce a power control process for thermal throttling in a mixed reality split-chip architecture, in accordance with various aspects of the present disclosure. The power control process runs in the background to perform thermal throttling for the mixed reality split-chip architecture. The power control process communicates a region of interest, a resolution, a bandwidth compression factor, a frame capture rate (e.g.. frame per second (FPS)), binning parameters, and / or a degree of ISP processing (denoise / sharpening, etc.) to the front-end chip and to the ISP post-processing block on the back-end chip. The power control process accesses the power state for both the front-end and back-end chips. The scene content texture / resolution may be inferred from ISP statistics estimated on the video see-through sensor streams, and head movement / speed estimated based of the perception head tracking process. Using the inferred information and power state, the region of interest, resolution, bandwidth compression factor, frame capture rate, binning parameters, and / or degree of ISP processing may be determined for the VST sensor output. For example, if the head movement is slow and a scene is not very detailed, then the fovea region of interest may be reduced.
[0024] If the front-end and back-end chips are in or near low power mode, the back- end chip reduces the fovea region of interest, which reduces ISP traffic and processing requirements. In one example, the fovea region of interest reduces from 40% to 20% when the low power mode is detected, such that the highest resolution is only used in 20% of the rendering. The frame capture rate may also be reduced.
[0025] In some aspects, the back-end chip updates the sensor capture resolution. For example, binning may be employed such that a 36 megapixel resolution is reduced to a nine megapixel resolution with 2 x 2 binning in both X and Y directions. In other aspects, a number of cameras may be reduced in response to detecting or estimating a low power mode. For example, the device may switch from four cameras to two cameras for perception.
[0026] After the adjustments are made, the power control process communicates to a render engine the fovea / non-fovea region coordinates to update blending for a final render buffer. The fovea region of interest and bandwidth compression factor are indicated to the front-end chip to adjust processing at the front-end. An example of a bandwidth compression factor is lossless compression. The compression may be adjusted independently from the fovea region of interest.
[0027] To estimate the power state, the front-end chip may inform the back-end chip of a battery state and a thermal state. The front-end chip may also inform the back- end chip of a type of scene being rendered and a user state. A scene type may be a static scene or a dynamic scene, for example. A static scene type may not experience significant degradation when reducing a frame rate. Examples of user state include whether a user is playing a mobile game (specifying fast activity in the virtual environment) or whether the user is web browsing (specifying slow activity ). The user state may be detected based on head movement and user eye movement. The power, thermal, and user states, along with the type of scene, help the split-chip architecture select the change (e.g., fovea region of interest or sensor capture resolution) that will have the least impact on the user. The transfer of information from the front-end chip to the back-end chip can occur based on the current frame rate such that information for the next frame is communicated in time for processing of that frame.
[0028] Particular aspects of the subject matter described in this disclosure can be implemented to realize one or more of the following potential advantages. In some examples, the described techniques, such as adjusting a fovea region of interest based on power state, enable thermal management in split-chip architectures.
[0029] FIGURE 1 illustrates an example implementation of a host system-on-a-chip (SoC) 100, which may operate as part of a split-chip architecture, in accordance withaspects of the present disclosure. The host SoC 100 includes processing blocks tailored to specific functions, such as a connectivity block 110. The connectivity block 110 may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, universal serial bus (USB) connectivity, Bluetooth® connectivity. Secure Digital (SD) connectivity, and the like.
[0030] In this configuration, the host SoC 100 includes various processing units that support multi -threaded operation. For the configuration shown in FIGURE 1, the host SoC 100 includes a multi-core central processing unit (CPU) 102. a graphics processor unit (GPU) 104, a digital signal processor (DSP) 106, and a neural processor unit (NPU) 108. The host SoC 100 may also include a sensor processor 114, image signal processors (ISPs) 116, a navigation module 120, which may include a global positioning system (GPS), and a memory 118. The multi -core CPU 102, the GPU 104. the DSP 106, the NPU 108. and the multi-media engine 112 support various functions such as video, audio, graphics, gaming, artificial networks, and the like. Each processor core of the multi-core CPU 102 may be a reduced instruction set computing (RISC) machine, an advanced RISC machine (ARM), a microprocessor, or some other type of processor. The NPU 108 may be based on an ARM instruction set.
[0031] According to aspects of the present disclosure, a virtual reality device includes a split-chip power controller. The device may include means for estimating, means for adjusting, means for updating, means for indicating, means for reducing, means for communicating, and means for receiving. In one configuration, the means for estimating, means for adjusting, means for updating, means for indicating, means for reducing, means for communicating, and means for receiving may be the CPU, GPU, DSP, NPU ISPs, multimedia block and / or memory’, as shown in FIGURE 1. In other aspects, the aforementioned means may be any structure or any material configured to perform the functions recited by the aforementioned means.
[0032] FIGURE 2 is a block diagram illustrating a mixed reality split-chip architecture 200. in accordance with various aspects of the present disclosure. As shown in FIGURE 2, the split-chip architecture 200 may include a front-end chip 220 and a back-end chip 240. The front-end chip 220 may include an image signal processor (ISP) front-end 222, a video see-through (VST) sensor control block 224, a power loop control block 226, and a perception block 228. In some implementations,(not shown) the perception block 228 is located on the back-end chip 240. The image signal processor front-end 222 enhances images captured by one or more cameras integrated with or in communication with the split-chip architecture 200. For example, a mixed reality device may use two red-green-blue (RGB) cameras to emulate the parallax between a user’s eyes. To enhance the images captured by a camera, the image signal processor front-end 222 performs, for example, demosaicing, noise reduction, RGB to YUV conversion, and exposure control. The video see-through sensor control block 224 is a runtime component of the split-chip architecture 200 that inputs environment variables and outputs one or more virtual channels. The power loop control block 226 communicates with the video see-through sensor control block 224 and the image signal processor front-end 222 to modify their functionality based on inputs received from the back-end chip 240. The perception block 228 inputs a scene and performs user head tracking and user positional tracking.
[0033] The back-end chip 240 may include an image signal processor (ISP) postprocessing block 242, a render block 244, and a power manager 246. The image signal processor post-processing block 242 performs post-processing functions such as resolution clean up, noise clean up, and other image-processing functions not otherwise performed on the front-end chip 220. In some examples, one or more of these postprocessing functions may instead be performed on a graphics processing unit (GPU). The render block 244 produces virtual content over a scene for display to a user. The power manager 246 adjusts one or more of a fovea region of interest, an image signal processing traffic bandwidth compression factor, a sensor capture resolution, binning parameters, and a frame capture rate based on power levels indicated by the front-end chip 220 and / or the back-end chip 240. In some aspects, the power manager 246 may be a video see-through (VST) runtime or an application layer. The power manager 246 may adjust the fovea region of interest by sending an indication of a new fovea region of interest to the front-end chip 220. The front-end chip 220 may then send an image buffer to the back-end chip 240 using the new fovea region of interest. The front-end chip 220 may send a transmission to the back-end chip 240 indicating the fovea region of interest used to capture and / or crop the image buffer.
[0034] In the example split-chip architecture 200 illustrated in FIGURE 2, the frontend chip 220 communicates to the back-end chip 240 a current power state, thermalstate, user state, and type of scene. Based on one or more of the power state, thermal state, user state, and type of scene, the back-end chip 240 communicates to the front-end chip 220 a fovea region of interest (ROI) and a bandwidth compression factor. The fovea region of interest is the part of an image or scene that a viewer is most likely to visually focus on. The bandwidth compression factor indicates a measurement of the relative reduction in size of data transmitted between devices, such as the front-end chip 220 and back-end chip 240. The term “bandwidth” may refer to image or video data bandwidth that may be measured by bytes-per-second, for example. The type of scene refers to where rendered content is being placed as well as what is included in a scene. Based on the fovea region of interest and / or bandwidth compression factor, the frontend chip 220 communicates to the back-end chip 240 an image buffer containing fovea and non-fovea regions. In other aspects, the split-chip architecture 200 may instead use a single chip, where the components of the front-end chip 220 and the back-end chip 240 operate on the single chip. In still other aspects, the split-chip architecture 200 may instead use more than two chips, where the components of the front-end chip 220 and back-end chip 240 operate on the more than two chips.
[0035] FIGURE 3 is a flow diagram illustrating a power control process 300 for thermal throttling in a mixed reality split-chip architecture, in accordance with various aspects of the present disclosure. As show n in FIGURE 3, in some aspects, the process 300 includes estimating a current power state at block 302. For example, a device incorporating a split-chip architecture may estimate the power state of the front-end chip and the back-end chip.
[0036] At block 304, the process 300 performs one or more actions if the front-end chip and the back-end chip are in or near a low power mode, as indicated by the frontend chip and back-end chip operating below a power threshold. The power threshold may comprise one or more indicators of each chip’s computational load, including voltage, clock speed, and / or temperature. In some examples, the power threshold may be an indicator-specific variable. As an illustration, the front-end chip and back-end chip may be in a low power mode if the front-end chip and back-end chip are each operating below one volt. In other examples, the power threshold may be a percentage of a variable. As an illustration, the front-end chip and back-end chip may be in a low power mode if the front-end chip and back-end chip are each operating below twentypercent of their maximum clock speed. The device may also be in a low power mode depending on a user state or type of scene. For example, the device may be in a low power mode if the user generates a small amount of head movement. As another example, the device may be in a low power mode if the scene being rendered for display contains little virtual content, virtual content that does not require intensive rendering, and / or virtual content of a predefined classification. As an illustration, the device may be in a low power mode if the scene being rendered for display contains a static menu screen in a majority of the user’s field of view. Similarly, the device may be in a low power state if the content displayed is relatively static, e.g., non-changing. By contrast, the device may not be in a low power state if the content displayed includes a salient object moving in the scene. In still other examples, the device may be in a low power mode if virtual content overlaps the user’s fovea. As an illustration, the device may be in a low power mode if the device is rendering a video panel in the center of the user’s field of view. In still other examples, the power threshold may comprise a load budget, w here the load budget is an expression of two or more indicators. As an illustration, the device may be in a low power mode if the front-end chip and back-end chip are both operating below' one volt and the scene being rendered for display contains little virtual content. In some aspects, the device may utilize multiple power thresholds, where the device may perform one or more actions if the device is at or near one threshold, and one or more actions if the device is at or near another threshold.
[0037] If the front-end chip and back-end chip are in or near a low power mode, the device may take one or more actions, such as: reducing the fovea region of interest (ROI), reducing image signal processing (ISP) traffic and processing requirement, updating sensor capture resolution, and enabling binning. For example, a number of perception cameras may be reduced from two cameras to a single camera if low power mode is detected for the front-end and back-end chips. In another example, YUV color processing is reduced to Y only (e.g., luma) processing, in other words, color processing swatches to gray scale processing in response to detecting low' po 'er mode. In still other examples, detected head movement may trigger a frame rate change and / or certain types of scene content may trigger a sensor capture resolution update. These operations may occur if the front-end chip and back-end chip are detected to be near (as opposed to in) low' power mode, in some aspects.
[0038] Each of these operations allows the device to preserve power, reduce thermal issues, and maximize longevity of the device. To decrease a fovea region of interest, the device may decrease the portion of a user’s field of view that is captured in a high resolution, and increase the non-fovea region. The non-fovea region is the region on the periphery of the fovea region of interest that is captured in a low resolution. In other examples, the device may crop the fovea region of interest of a captured image, decreasing the fovea region of interest in proportion to the non-fovea region. As an illustration, the device may decrease the fovea region of interest of a captured image such that the image size goes from twelve megapixels to two megapixels. By reducing the fovea region of interest, the device reduces the area in the user’s field of view in which images are rendered at a higher resolution. Reducing the fovea region of interest therefore reduces the aggregate resolution of the rendered image, reducing the computational work required for the front-end chip and back-end chip to render and display the image.
[0039] The device may also reduce the image signal processing (ISP) traffic and processing requirement. By reducing the image signal processing (ISP) traffic and processing requirement, the device reduces the bandwidth and / or transmission time for data transmitted between the front-end chip, back-end chip, and / or individual entities within each chip. For example, the device may use a higher bandwidth compression factor, which reduces the size of data transmitting in the device. To update a sensor capture resolution, the device may reduce camera capture resolution from, for example, 704 x 480 pixels to 352 x 240 pixels. This reduced sensor capture resolution reduces the sensor power consumption and the power needed to process the image into rendered content. The device may also enable binning by combining clusters of adjacent pixels, throughout an image, into single pixels. For example, the device may use 2 x 2 binning to bin groups of four pixels into a single pixel. The effect of binning is a lower- resolution image requiring less power for the front-end chip and back-end chip to process into rendered content.
[0040] The device may also reduce a sensor capture rate if the front-end chip and back-end chip are in a low power mode. For example, the device may reduce a camera capture rate from ninety frames-per-second to sixty frames-per-second. A lower sensor capture rate also allows the front-end chip and back-end chip to use less power toprocess an image for display. The device may also decrease the field of view of an image by, for example, cropping the image. By reducing the field of view of the image, the device reduces the size of the image, which also allows the front-end chip and back- end chip to use less power to process the image for display. The device may also reduce image capture resolutions in the regions of a user’s field of view that overlap with virtual content. As an illustration, the device may stop capturing the center of a user’s field of view if the device is displaying virtual content at the center of the user’s field of view. At block 306, the process communicates to a render engine to update the blending for a final render buffer. This communication may include fovea and nonfovea region coordinates.
[0041] FIGURE 4 is a flow' diagram illustrating an example process 400 performed, for example, by in a mixed reality split-chip architecture, in accordance w ith various aspects of the present disclosure. The example process 400 is an example of split-chip power control. As shown in FIGURE 4, in some aspects, the process 400 may include estimating a first pow er state of a front-end chip of the mixed reality split-chip architecture (block 402). For example, the power state may represent the voltage, clock speed, or temperature of the front-end chip. In other examples, the pow er state may represent a user head state or scene. In some aspects, the process 400 may include estimating a second pow er state of a back-end chip of the mixed reality split-chip architecture. The back-end chip is located remotely from the front-end chip (block 404). In other aspects, the process 400 may use only a single chip, where the process 400 estimates the power state of that single chip. In still other aspects, the process 400 may use more than two chips, where the process 400 estimates the pow er state of one or more of those chips. In these aspects, the process 400 may include taking one or more actions based on the power state in the one or more chips.
[0042] In some aspects, the process 400 may include adjusting a fovea region of interest based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low pow er mode and operation of the back-end chip in the low power mode (block 406). In some aspects, the mixed reality split-chip architecture may also update sensor capture resolution and / or a frame capture rate based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low pow er mode and operation of the back-end chipin the low power mode. The updating of the sensor capture resolution may comprise binning of pixels. The mixed reality split-chip architecture may also indicate from the back-end chip, an indication of a bandwidth compression factor. In some aspects, the mixed reality split-chip architecture may also reduce a quantity of cameras employed for perception based on the first power state and the second power state, in response to detecting operation of the front-end chip in the low power mode and operation of the back-end chip in the low power mode. The mixed reality split-chip architecture may also communicate to a render engine the fovea region of interest that was used when capturing images. The mixed reality split-chip architecture may also: receive from the front-end chip, at the back-end chip, an indication of a type of scene being processed and a user state comprising a head position of the user; update the fovea region of interest based on the type of scene and the user state; update a sensor capture resolution based on the type of scene; and / or update a frame rate based on the user state.
[0043] FIGURE 5 is a block diagram showing an exemplary wireless communications system 500, in which an aspect of the present disclosure may be advantageously employed. For purposes of illustration, FIGURE 5 shows three remote units 520, 530, and 550, and two base stations 540. It will be recognized that wireless communications systems may have many more remote units and base stations. Remote units 520, 530. and 550 include integrated circuit (1C) devices 525 A, 525B. and 525C that include the disclosed split-chip power controller. It will be recognized that other devices may also include the disclosed split-chip power controller, such as the base stations, switching devices, and network equipment. FIGURE 5 shows forward link signals 580 from the base stations 540 to the remote units 520, 530. and 550, and reverse link signals 590 from the remote units 520, 530, and 550 to the base stations 540.
[0044] In FIGURE 5, remote unit 520 is shown as a mobile telephone, remote unit 530 is shown as a portable computer, and remote unit 550 is shown as a fixed location remote unit in a wireless local loop system. For example, the remote units may be a mobile phone, a hand-held personal communication systems (PCS) unit, a portable data unit, such as a personal data assistant, a GPS enabled device, a navigation device, a set top box, a music player, a video player, an entertainment unit, a fixed location data unit, such as meter reading equipment, or other device that stores or retrieves data orcomputer instructions, or combinations thereof. Although FIGURE 5 illustrates remote units according to the aspects of the present disclosure, the disclosure is not limited to these exemplary illustrated units. Aspects of the present disclosure may be suitably employed in many devices, which include the disclosed split-chip power controller.
[0045] FIGURE 6 is a block diagram illustrating a design workstation 600 used for circuit, layout, and logic design of a semiconductor component, such as the split-chip power controller disclosed above. The design workstation 600 includes a hard disk 601 containing operating system software, support files, and design software such as Cadence or OrCAD. The design workstation 600 also includes a display 602 to facilitate design of a circuit 610 or a semiconductor component 612, such as the splitchip power controller. A storage medium 604 is provided for tangibly storing the design of the circuit 610 or the semiconductor component 612 (e.g., the PLD). The design of the circuit 610 or the semiconductor component 612 may be stored on the storage medium 604 in a file format such as GDSII or GERBER. The storage medium 604 may be a CD-ROM, DVD, hard disk, flash memory', or other appropriate device. Furthermore, the design workstation 600 includes a drive apparatus 603 for accepting input from or writing output to the storage medium 604.
[0046] Data recorded on the storage medium 604 may specify logic circuit configurations, pattern data for photolithography masks, or mask pattern data for serial write tools such as electron beam lithography. The data may further include logic verification data such as timing diagrams or net circuits associated with logic simulations. Providing data on the storage medium 604 facilitates the design of the circuit 510 or the semiconductor component 612 by decreasing the number of processes for designing semiconductor wafers.Example Aspects
[0047] Aspect 1 : A method of processing in a mixed reality split-chip architecture, comprising: estimating a first power state of a front-end chip of the mixed reality splitchip architecture; estimating a second power state of a back-end chip of the mixed reality split-chip architecture, the back-end chip located remotely from the front-end chip; and adjusting a fovea region of interest based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
[0048] Aspect 2: The method of Aspect 1, further comprising updating at least one of a sensor capture resolution and a frame capture rate based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
[0049] Aspect 3: The method of Aspect 1 or 2, in which the updating sensor capture resolution comprises binning of pixels.
[0050] Aspect 4: The method of any of the preceding Aspects, further comprising indicating from the back-end chip, an indication of a bandwidth compression factor.
[0051] Aspect 5: The method of any of the preceding Aspects, further comprising reducing a quantity of cameras employed for perception based on the first power state and the second power state, in response to detecting operation of the front-end chip in the low power mode and operation of the back-end chip in the low power mode.
[0052] Aspect 6: The method of any of the preceding Aspects, further comprising communicating to a render engine the fovea region of interest that was used when capturing images.
[0053] Aspect 7 : The method of any of the preceding Aspects, further comprising receiving from the front-end chip, at the back-end chip, an indication of a type of scene being processed and a user state comprising a head position of a user; updating the fovea region of interest based on the type of scene and the user state; updating a sensor capture resolution based on the type of scene; and updating a frame rate based on the user state.
[0054] Aspect 8: An apparatus for processing in a mixed reality split-chip architecture, comprising: at least one memory; and at least one processor coupled to the memory, the at least one processor configured: to estimate a first power state of a frontend chip of the mixed reality split-chip architecture; to estimate a second power state of a back-end chip of the mixed reality split-chip architecture, the back-end chip located remotely from the front-end chip; and to adjust a fovea region of interest based on the first power state and the second power state, in response to detecting operation of thefront-end chip in a low power mode and operation of the back-end chip in the low power mode.
[0055] Aspect 9: The apparatus of Aspect 8, in which the at least one processor is further configured to update at least one of a sensor capture resolution and a frame capture rate based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
[0056] Aspect 10: The apparatus of Aspect 8 or 9, in which the updating sensor capture resolution comprises binning of pixels.
[0057] Aspect 11: The apparatus of any of the Aspects 8-10, in which the at least one processor is further configured to indicate from the back-end chip, an indication of a bandwidth compression factor.
[0058] Aspect 12: The apparatus of any of the Aspects 8-11, in which the at least one processor is further configured to reduce a quantity of cameras employed for perception based on the first power state and the second power state, in response to detecting operation of the front-end chip in the low power mode and operation of the back-end chip in the low power mode.
[0059] Aspect 13: The apparatus of any of the Aspects 8-12, in which the at least one processor is further configured to communicate to a render engine the fovea region of interest that was used when capturing images.
[0060] Aspect 14: The apparatus of any of the Aspects 8-13, in which the at least one processor is further configured to receive from the front-end chip, at the back-end chip, an indication of a type of scene being processed and a user state comprising a head position of a user; to update the fovea region of interest based on the type of scene and the user state; to update a sensor capture resolution based on the type of scene; and to update a frame rate based on the user state.
[0061] Aspect 15: An apparatus for processing in a mixed reality7split-chip architecture, comprising: means for estimating a first power state of a front-end chip ofthe mixed reality split-chip architecture; means for estimating a second power state of a back-end chip of the mixed reality split-chip architecture, the back-end chip located remotely from the front-end chip; and means for adjusting a fovea region of interest based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
[0062] Aspect 16: The apparatus of Aspect 15, further comprising means for updating at least one of a sensor capture resolution and a frame capture rate based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
[0063] Aspect 17: The apparatus of Aspect 15 or 16, further comprising in which the updating sensor capture resolution comprises binning of pixels.
[0064] Aspect 18: The apparatus of any of the Aspects 15-17, further comprising means for indicating from the back-end chip, an indication of a bandwidth compression factor.
[0065] Aspect 19: The apparatus of any of the Aspects 15-18, further comprising means for reducing a quantity7of cameras employed for perception based on the first power state and the second power state, in response to detecting operation of the frontend chip in the low power mode and operation of the back-end chip in the low power mode.
[0066] Aspect 20: The apparatus of any of the Aspects 15-19, further comprising means for communicating to a render engine the fovea region of interest that was used when capturing images.
[0067] Aspect 21: The apparatus of any of the Aspects 15-20, further comprising means for receiving from the front-end chip, at the back-end chip, an indication of a type of scene being processed and a user state comprising ahead position of a user; means for updating the fovea region of interest based on the type of scene and the userstate; means for updating a sensor capture resolution based on the type of scene; and means for updating a frame rate based on the user state.
[0068] Aspect 22: A non-transitory computer-readable medium having program code recorded thereon, the program code executed by a processor and comprising: program code to estimate a first power state of a front-end chip of the mixed reality split-chip architecture; program code to estimate a second power state of a back-end chip of the mixed reality split-chip architecture, the back-end chip located remotely from the front-end chip; and program code to adjust a fovea region of interest based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
[0069] Aspect 23: The non-transitory computer-readable medium of Aspect 22, in which the program code comprises program code to update at least one of a sensor capture resolution and a frame capture rate based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
[0070] Aspect 24: The non-transitory computer-readable medium of Aspect 22 or 23, in which the program code to update sensor capture resolution comprises program code to bin pixels.
[0071] Aspect 25: The non-transitory computer-readable medium of any of the Aspects 22-24, in which the program code comprises program code to indicate from the back-end chip, an indication of a bandw idth compression factor.
[0072] Aspect 26: The non-transitory computer-readable medium of any of the Aspects 22-25, in which the program code comprises program code to reduce a quantity of cameras employed for perception based on the first power state and the second power state, in response to detecting operation of the front-end chip in the low power mode and operation of the back-end chip in the low power mode.
[0073] Aspect 27: The non-transitory computer-readable medium of any of the Aspects 22-26, in which the program code comprises program code to communicate to a render engine the fovea region of interest that was used when capturing images.
[0074] Aspect 28: The non-transitory computer-readable medium of any of the Aspects 22-27, in which the program code comprises program code to receive from the front-end chip, at the back-end chip, an indication of a type of scene being processed and a user state comprising a head position of a user; program code to update the fovea region of interest based on the type of scene and the user state; program code to update a sensor capture resolution based on the type of scene; and program code to update a frame rate based on the user state.
[0075] Aspect 29: The method of any of the Aspects 1-7, further comprising adjusting the fovea region of interest based on the inferred scene content texture / resolution.
[0076] For a firmware and / or software implementation, the methodologies may be implemented with modules (e.g.. procedures, functions, and so on) that perform the functions described. A machine-readable medium tangibly embodying instructions may be used in implementing the methodologies described. For example, software codes may be stored in a memory and executed by a processor unit. Memory may be implemented within the processor unit or external to the processor unit. As used, the term ‘'memory” refers to types of long term, short term, volatile, nonvolatile, or other memory and is not limited to a particular type of memory or number of memories, or type of media upon which memory' is stored.
[0077] If implemented in firmware and / or software, the functions may be stored as one or more instructions or code on a computer-readable medium. Examples include computer-readable media encoded with a data structure and computer-readable media encoded with a computer program. Computer-readable media includes physical computer storage media. A storage medium may be an available medium that can be accessed by a computer. By way of example, and not limitation, such computer- readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), compact disc read-only memory'(CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray® disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0078] In addition to storage on computer-readable medium, instructions and / or data may be provided as signals on transmission media included in a communications apparatus. For example, a communications apparatus may include a transceiver having signals indicative of instructions and data. The instructions and data are configured to cause one or more processors to implement the functions outlined in the claims.
[0079] Although the present disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made without departing from the technology of the disclosure as defined by the appended claims. For example, relational terms, such as ‘‘above” and “below” are used with respect to a substrate or electronic device. Of course, if the substrate or electronic device is inverted, above becomes below, and vice versa. Additionally, if oriented sideways, above and below may refer to sides of a substrate or electronic device. Moreover, the scope of the present disclosure is not intended to be limited to the particular configurations of the process, machine, manufacture, composition of matter, means, methods, and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the present disclosure, processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding configurations described may be utilized according to the present disclosure. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.
[0080] Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the present disclosure may be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality' is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
[0081] The various illustrative logical blocks, modules, and circuits described in connection with the disclosure may be implemented or performed with a general- purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described. A general-purpose processor may be a microprocessor, but, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g.. a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
[0082] The steps of a method or algorithm described in connection with the present disclosure may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM, flash memory, ROM, erasable programmable read-only memory' (EPROM), EEPROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
[0083] The previous description of the present disclosure is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications tothe disclosure will be readily apparent to those skilled in the art, and the generic principles defined may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the examples and designs described, but is to be accorded the widest scope consistent with the principles and novel features disclosed.
Claims
CLAIMSWHAT IS CLAIMED IS:
1. A method of processing in a mixed reality split-chip architecture, comprising: estimating a first power state of a front-end chip of the mixed reality split-chip architecture; estimating a second power state of a back-end chip of the mixed reality splitchip architecture, the back-end chip located remotely from the front-end chip; and adjusting a fovea region of interest based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
2. The method of claim 1. further comprising updating at least one of a sensor capture resolution and a frame capture rate based on the first power state and the second power state, in response to detecting operation of the front-end chip in the low power mode and operation of the back-end chip in the low power mode.
3. The method of claim 2, in which the updating sensor capture resolution comprises binning of pixels.
4. The method of claim 1 , further comprising indicating from the back-end chip, an indication of a bandwidth compression factor.
5. The method of claim 1, further comprising reducing a quantity of cameras employed for perception based on the first power state and the second power state, in response to detecting operation of the front-end chip in the low power mode and operation of the back-end chip in the low power mode.
6. The method of claim 1 , further comprising communicating to a render engine the fovea region of interest that was used when capturing images.
7. The method of claim 1, further comprising: receiving from the front-end chip, at the back-end chip, an indication of a ty pe of scene being processed and a user state comprising a head position of a user;updating the fovea region of interest based on the type of scene and the user state; updating a sensor capture resolution based on the type of scene; and updating a frame rate based on the user state.
8. The method of claim 1. further comprising adjusting the fovea region of interest based on inferred scene content texture / resolution.
9. An apparatus for processing in a mixed reality split-chip architecture, comprising: at least one memory'; and at least one processor coupled to the memory, the at least one processor configured: to estimate a first power state of a front-end chip of the mixed reality split-chip architecture; to estimate a second power state of a back-end chip of the mixed reality split-chip architecture, the back-end chip located remotely from the front-end chip; and to adjust a fovea region of interest based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
10. The apparatus of claim 9, in which the at least one processor is further configured to update at least one of a sensor capture resolution and a frame capture rate based on the first power state and the second power state, in response to detecting operation of the front-end chip in the low power mode and operation of the back-end chip in the low power mode.11 . The apparatus of claim 10, in which the at least one processor is further configured to update sensor capture resolution by binning pixels.
12. The apparatus of claim 9, in which the at least one processor is further configured to indicate from the back-end chip, an indication of a bandwidth compression factor.
13. The apparatus of claim 9. in which the at least one processor is further configured to reduce a quantity of cameras employed for perception based on the first power state and the second power state, in response to detecting operation of the frontend chip in the low power mode and operation of the back-end chip in the low power mode.
14. The apparatus of claim 9, in which the at least one processor is further configured to communicate to a render engine the fovea region of interest that was used when capturing images.
15. The apparatus of claim 9, in which the at least one processor is further configured: to receive from the front-end chip, at the back-end chip, an indication of a type of scene being processed and a user state comprising a head position of a user; to update the fovea region of interest based on the type of scene and the user state; to update a sensor capture resolution based on the type of scene; and to update a frame rate based on the user state.
16. An apparatus for processing in a mixed reality split-chip architecture, comprising: means for estimating a first power state of a front-end chip of the mixed reality split-chip architecture; means for estimating a second power state of a back-end chip of the mixed reality split-chip architecture, the back-end chip located remotely from the front-end chip; and means for adjusting a fovea region of interest based on the first power state and the second power state, in response to detecting operation of the front-end chip in a low power mode and operation of the back-end chip in the low power mode.
17. The apparatus of claim 16, further comprising means for updating at least one of a sensor capture resolution and a frame capture rate based on the first power state and the second power state, in response to detecting operation of the front-end chip in the low power mode and operation of the back-end chip in the low power mode.
18. The apparatus of claim 17. in which the means for updating sensor capture resolution comprises means for binning of pixels.
19. The apparatus of claim 16, further comprising means for indicating from the back-end chip, an indication of a bandwidth compression factor.
20. The apparatus of claim 16, further comprising means for reducing a quantity of cameras employed for perception based on the first power state and the second power state, in response to detecting operation of the front-end chip in the low power mode and operation of the back-end chip in the low pow er mode.
21. The apparatus of claim 16, further comprising means for communicating to a render engine the fovea region of interest that was used when capturing images.
22. The apparatus of claim 16, further comprising: means for receiving from the front-end chip, at the back-end chip, an indication of a t pe of scene being processed and a user state comprising a head position of a user; means for updating the fovea region of interest based on the type of scene and the user state; means for updating a sensor capture resolution based on the type of scene; and means for updating a frame rate based on the user state.
Citation Information
Patent Citations
Split hierarchy graphics processing system
US20220188964A1
Display peak power management for artificial reality systems
US20230113746A1
