Apparatus, system and method for accelerating position tracking of head mounted displays
By accelerating image processing with dedicated hardware devices and cache in head-mounted displays, the delay and power consumption problems of traditional head-mounted displays during position tracking is solved, achieving more efficient and accurate position tracking, and improving user experience .
Patent Information
- Application Number
- CN201880093718.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-05-22
- Filing Date
- 2018-06-11
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2038-06-11
AI Technical Summary
Traditional head-mounted displays have visual delays in position and orientation changes and require a large amount of power and computing resources to accurately track these changes, resulting in increased latency and power consumption, and potentially sacrificing the accuracy of the position tracking system.
Adopt special hardware equipment, including image signal processors and position tracking components, uses cache to accelerate image processing, reduce the number of DRAM accesses, and accelerate feature detection and image processing components to improve the efficiency of position tracking algorithms and achieve more accurate position tracking. .
Reduces processing time and power consumption of head-mounted displays, improves accuracy and accuracy of position tracking, and improves user experience.
Smart Images

Figure CN112166397B_ABST
Abstract
Description
[0001] background
[0002] Over the past few years, head-mounted displays have revolutionized the way people experience all kinds of digital media. For example, head-mounted displays can allow virtual reality users to experience realistic, immersive virtual environments while playing video games, during flight simulation training, or even while interacting with colleagues around the world. In addition to such virtual reality applications, head-mounted displays can also enable users to augment reality and / or combine certain aspects of reality with those of the virtual world.
[0003] Despite the incredible advancements in this technology, conventional head-mounted displays still have certain deficiencies that negatively impact the overall user experience. For example, some conventional head-mounted displays may exhibit noticeable visual delays after certain changes in position and / or orientation. More specifically, because these head-mounted displays often account for changes in position when incorporating virtual content into images and / or videos, any delay in obtaining information used to determine such changes in position may ultimately slow down the entire image processing cycle.
[0004] Further complicating the problem, some conventional head mounted displays may require significant power and / or computing resources to accurately determine changes in position and / or orientation. In order to reduce latency and power consumption, these conventional head mounted displays may sacrifice the accuracy of the position tracking system by determining certain changes in position based on low resolution image data.
[0005] Accordingly, the present disclosure identifies and addresses the need for additional apparatus, systems, and methods for accelerating position tracking of head mounted displays.
[0006] Overview
[0007] As will be described in more detail below, the present disclosure generally relates to apparatus, systems, and methods for accelerating position tracking in a head-mounted display. In some embodiments, a dedicated hardware device can achieve this goal and / or can provide various other features. For example, the dedicated hardware device may include an image signal processor that receives an image frame of the physical environment surrounding the head-mounted display system from a camera device of the head-mounted display system. The dedicated hardware device may also include a position tracking component that stores at least a portion of the image frame in a cache of the dedicated hardware device. The cache may have a faster access speed than the main memory of the dedicated hardware device. The position tracking component may also track changes in the position of the head-mounted display system within the physical environment based at least in part on the portion of the image frame stored in the cache of the dedicated hardware device. The position tracking component may then store the change in the position of the head-mounted display system in the main memory for generating one or more augmented reality frames to be presented to a user wearing the head-mounted display system. In addition, the dedicated hardware device may include a frame output interface that feeds the augmented reality frame to a display device of the head-mounted display system.
[0008] In some examples, the location tracking component can identify one or more features of the physical environment that represent at least a portion of a physical object located within the physical environment. The location tracking component can also maintain an image map in the main memory of the dedicated hardware device that identifies one or more physical locations of the features within the physical environment.
[0009] In some embodiments, the location tracking component may include a hardware-accelerated feature detection component. When the input image frame is received and stored in the cache of the dedicated hardware device, the feature detection component may detect at least one of the features of the physical environment within the input image frame. The feature detection component may then pass the detected features to the main memory of the dedicated hardware device to facilitate incorporation of the detected features into the image map.
[0010] The position tracking component may also include a hardware accelerated image processing component. When the input image frame is received and stored in the cache of the dedicated hardware device, the image processing component may create at least one low-resolution version of the input image frame. The image processing component may then transfer the low-resolution version of the input image frame to the main memory of the dedicated hardware device to facilitate tracking changes in the position of the head mounted display system.
[0011] In some examples, the position tracking component can identify a set of key features within the physical environment that facilitate tracking changes in the position of the head mounted display system. In these examples, the position tracking component can (1) identify an initial position of a representation of at least one key feature of the physical environment within an initial image frame captured by the camera device, (2) identify a subsequent position of the representation of the key feature of the physical environment within a subsequent image frame captured by the camera device, and then (3) compare the initial position of the representation of the key feature to the subsequent position of the representation of the key feature.
[0012] In one embodiment, the position tracking component can identify the subsequent position of the key feature representation within the subsequent image frame by: (1) determining an expected area of the subsequent image frame in which the key feature of the physical environment may be represented based on the initial position of the key feature representation within the initial image frame, and then (2) identifying the specific position of the key feature representation within the subsequent image frame by searching the expected area of the subsequent image frame.
[0013] In some examples, the location tracking component can generate one or more image patches that represent the physical appearance of key features within the physical environment from different perspectives. In one embodiment, the location tracking component can store these image patches in the main memory of a dedicated hardware device.
[0014] In some embodiments, the position tracking component may include an image search unit. The image search unit may identify the specific location of the representation of the key feature within the subsequent image frame by: (1) distorting (warping) at least one image block so that the image block resembles the current physical appearance of the key feature from the perspective of the camera device capturing the subsequent image frame, and then (2) identifying the location within the subsequent image frame where the pixel value of the distorted image block most closely matches the corresponding pixel value of the subsequent image frame. In one example, the image search unit may identify the sub-pixel location of the representation of the key feature within the subsequent image frame by: (1) performing sub-pixel interpolation analysis on pixel values within an expected area of the subsequent image frame, and then (2) determining, based on the sub-pixel interpolation analysis, the sub-pixel location where the pixel value of the distorted image block most closely matches the corresponding pixel value of the subsequent image frame.
[0015] In some examples, the image search unit may determine that additional key features of the physical environment are expected to be represented within an expected region of a subsequent image frame. In these examples, the image search unit may load the expected region of the subsequent image frame into a cache of the dedicated hardware device. The image search unit may then identify both the specific location of the representation of the key features within the subsequent image frame and the specific location of the representation of the additional key features within the subsequent image frame by searching the expected region of the subsequent image frame within the cache of the dedicated hardware device.
[0016] In some embodiments, the dedicated hardware device may include a hardware-accelerated search component. The search component may be dedicated to identifying the specific location of the representation of the key feature within a subsequent image frame, and identifying the physical location of the key feature within the physical environment. In one example, the search component may identify the physical location of the key feature within the physical environment by: (1) identifying the specific location of the representation of the key feature within a previous image frame captured by the camera device, and then (2) performing an epipolar search based on the specific location of the representation of the key feature within the previous image frame to find the three-dimensional location of the key feature within the physical environment.
[0017] In some examples, the dedicated hardware device may include a frame generation unit. The frame generation unit may retrieve the position of the head mounted display system within a main memory of the dedicated hardware device. While generating an augmented reality frame to be presented to the user, the frame generation unit may adjust at least one aspect of the augmented reality frame to account for the change in the position of the head mounted display system.
[0018] The corresponding head-mounted display system may include at least one camera device that captures an image frame of a physical environment surrounding the head-mounted display system. The head-mounted display system may also include a dedicated hardware device communicatively coupled to the camera device. When receiving an image frame from the camera device, the dedicated hardware device may store at least a portion of the image frame in a cache within the dedicated hardware device. In one embodiment, the cache may have a faster access speed than a main memory of the dedicated hardware device. The dedicated hardware device may also track changes in the position of the head-mounted display system within the physical environment based at least in part on the portion of the image frame stored in the cache. The dedicated hardware device may also store the change in the position of the head-mounted display system in the main memory for generating at least one augmented reality frame to be presented to a user wearing the head-mounted display system. In addition, the head-mounted display system may include a display device communicatively coupled to the dedicated hardware device. The display device may receive the augmented reality frame from the dedicated hardware device and then present the augmented reality frame to the user wearing the head-mounted display system.
[0019] In some examples, the head-mounted display system may include a communication interface communicatively coupled to an independent host device that feeds virtual image content to the head-mounted display system. The head-mounted display system may also include at least one additional camera device that captures one or more image frames from the perspective of a user wearing the head-mounted display system. In some embodiments, from the perspective of a user wearing the head-mounted display system, a dedicated hardware device may generate an augmented reality frame to be presented to the user by overlaying virtual image content received from the independent host device on the image frame. In one example, the head-mounted display system may also include a hardware-accelerated image correction component that applies an asynchronous time warp or an asynchronous space warp to the virtual image content based at least in part on a change in the position of the head-mounted display system.
[0020] In some embodiments, the head mounted display system may include an inertial measurement unit communicatively coupled to the dedicated hardware device. In such embodiments, the dedicated hardware device may also track changes in the position of the head mounted display system based on data provided by the inertial measurement unit.
[0021] The corresponding method may include: (1) receiving, by a dedicated hardware device, an image frame of a physical environment surrounding the head-mounted display system from a camera device of the head-mounted display system, (2) when receiving the image frame from the camera device, storing, by the dedicated hardware device, at least a portion of the image frame in a cache of the dedicated hardware device, the cache having an access speed faster than a main memory of the dedicated hardware device, (3) tracking, by the dedicated hardware device, changes in the position of the head-mounted display system within the physical environment based at least in part on the portion of the image frame stored in the cache, (4) storing, by the dedicated hardware device, the changes in the position of the head-mounted display system in a main memory for use in generating at least one augmented reality frame to be presented to a user wearing the head-mounted display system, and then (5) feeding, by the dedicated hardware device, the augmented reality frame to a display device of the head-mounted display system so that the display device presents the augmented reality frame to the user wearing the head-mounted display system.
[0022] Features from any of the above-mentioned embodiments may be used in combination with each other according to the general principles described herein.These and other embodiments, features and advantages will be more fully understood by reading the following detailed description in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The accompanying drawings illustrate several exemplary embodiments and are a part of the specification. Together with the following description, these drawings demonstrate and explain various principles of the present disclosure.
[0025] Figure 1is a block diagram of an exemplary dedicated hardware device for accelerating position tracking of a head mounted display in accordance with some embodiments.
[0026] Figure 2 is an illustration of an exemplary head-mounted display incorporating specialized hardware devices for accelerating positional tracking of the head-mounted display, in accordance with some embodiments.
[0027] Figure 3 is a block diagram of an exemplary head-mounted display that incorporates dedicated hardware devices for accelerating position tracking of the head-mounted display, according to some embodiments.
[0028] Figure 4 is a block diagram of an exemplary implementation for accelerating position tracking of a head mounted display in accordance with some embodiments.
[0029] Figure 5 is a block diagram of an exemplary implementation for accelerating position tracking of a head mounted display in accordance with some embodiments.
[0030] Figure 6 is a block diagram of an exemplary implementation for accelerating position tracking of a head mounted display in accordance with some embodiments.
[0031] Figure 7 is a block diagram of an exemplary implementation for accelerating position tracking of a head mounted display in accordance with some embodiments.
[0032] Figure 8 is an illustration of exemplary image frames used by a dedicated hardware device to detect changes in the position of a head mounted display, according to some embodiments.
[0033] Fig. 9 is a block diagram of an exemplary implementation for accelerating position tracking of a head mounted display in accordance with some embodiments.
[0034] Fig.10 is a flow chart of an exemplary method for accelerating position tracking of a head mounted display in accordance with some embodiments.
[0035] In all drawings, the same reference numerals and descriptions represent similar but not necessarily identical elements. Although the exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, the exemplary embodiments described herein are not intended to be limited to the specific forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives that fall within the scope of the appended claims.
[0036] Detailed Description of Exemplary Embodiments
[0037] The present disclosure generally relates to devices, systems, and methods for accelerating position tracking of head-mounted displays. As will be explained in more detail below, the devices, systems, and methods disclosed herein can reduce the number of dynamic random access memory (DRAM) accesses involved in tracking changes in the position of a head-mounted display. Specifically, these devices, systems, and methods can cache input image data instead of buffering the data in DRAM. By executing at least a portion of the position tracking algorithm based on the cached data, the disclosed devices, systems, and methods can reduce or minimize the number of DRAM accesses required. Because DRAM accesses are typically more resource-intensive than performing in-line operations in the cache, these devices, systems, and methods can reduce processing time and power consumption of the head-mounted display.
[0038] Embodiments of the present disclosure may further improve the efficiency of executing position tracking algorithms by caching and analyzing only relevant portions of input image frames. More specifically, the disclosed apparatus, systems, and methods may divide input image frames into tiles and then cache tiles containing data relevant to performing computer vision operations such as feature tracking. By improving the efficiency of position tracking algorithms, these apparatus, systems, and methods may enable head-mounted displays to process high-resolution image data, thereby producing more precise and accurate position tracking data than conventional position tracking systems.
[0039] Additionally or alternatively, the devices, systems, and methods may implement a hardware-accelerated computer vision unit that efficiently tracks the position of a head-mounted display via one or more integrated circuits or hardware blocks. In particular, the hardware-accelerated unit may reuse certain hardware blocks for different processing steps (e.g., a single hardware block may be configured to perform two or more steps of a position tracking algorithm). By doing so, the disclosed devices, systems, and methods may reduce the power consumed when executing the position tracking algorithm and the on-chip presence of the computer vision unit.
[0040] Reference will now be made to various exemplary embodiments, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide an understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be implemented without these specific details. In other cases, well-known systems, methods, processes, components, circuits, and networks are not described in detail in order to avoid unnecessarily obscuring aspects of the embodiments.
[0041] refer to Figure 1-9 , a detailed description of exemplary devices, systems and / or corresponding implementations for accelerating position tracking of a head-mounted display will be provided below. Fig.10 A detailed description of a method for accelerating position tracking of a head mounted display is provided.
[0042] Figure 1 An exemplary dedicated hardware device 100 for accelerating position tracking according to some embodiments is shown. In some examples, the dedicated hardware device 100 may be incorporated into a head-mounted display and / or represent a portion of a head-mounted display. In such examples, the dedicated hardware device 100 may be responsible for tracking changes in position, orientation, and / or inertia of the head-mounted display.
[0043] The dedicated hardware device 100 generally represents a circuit, system, and / or hardware accelerator designed to perform various image processing operations and / or position tracking algorithms. In some examples, the dedicated hardware device 100 may include and / or represent a system on a chip (SoC) and / or an application-specific integrated circuit (ASIC). For example, the dedicated hardware device 100 may implement a computer vision (CV) accelerator by way of a SoC and / or an ASIC. In some embodiments, the term "CV accelerator" may refer to any software and / or hardware-based component or device that performs a position tracking algorithm.
[0044] In addition to the CV accelerator, the SoC and / or ASIC may also include a variety of other computing components involved in processing images for display by the head-mounted display. Examples of such computing components include, but are not limited to, advanced reduced instruction set computer (RISC) machine (ARM) processors, central processing units (CPUs), image signal processors (ISPs), digital signal processors (DSPs), display imaging pipelines, video encoders, video decoders, hardware accelerated processing components, combinations or variations of one or more of them, and / or any other suitable components.
[0045] like Figure 1 As shown, the dedicated hardware device 100 may include an image signal processor 102, a position tracking component 104, and a frame output interface 106. In some examples, the image signal processor 102 may receive an image frame from a camera mounted on a head-mounted display. The image signal processor 102 may optionally perform one or more image processing operations on the image frame (e.g., reformatting and / or rescaling the image), and then pass the image frame to the position tracking component 104. The position tracking component 104 may detect a change in the position of the head-mounted display based on the received image. For example, the position tracking component 104 may detect various features (e.g., physical objects or portions of physical objects) within the physical environment surrounding the head-mounted display based on the received image. The position tracking component 104 may then calculate and / or infer changes in the position of the head-mounted display system based on representations of features within the image frame tracked over time.
[0046] In some embodiments, one or more additional components of dedicated hardware device 100 may use the position tracking information generated by position tracking component 104 to create image frames for presentation to a user of a head mounted display. Figure 1 The frame output interface 106 may then send, transmit and / or feed these image frames to a display device that is integrated and / or incorporated into the head-mounted display. By doing so, the frame output interface 106 may enable the display device to display the generated frames for presentation to a user wearing the head-mounted display. In one example, the frame output interface 106 may include and / or represent a mobile industry processor interface (MIPI) display serial interface (DSI).
[0047] Figure 2 2 shows an exemplary head mounted display system 200 worn by a user 204 according to some embodiments. Figure 2 As shown, the head mounted display system 200 may include a strap 206 coupled to a head mounted display 202. In this example, the head mounted display 202 may include one or more cameras 210 that capture images and / or video of the surroundings of the user 204. In some embodiments, the cameras 210 may be positioned and / or angled to capture a wide field of view (e.g., a 360° field of view) around the head mounted display 202. In this manner, the cameras 210 may facilitate tracking movement of the head mounted display 202 in any and / or all directions.
[0048] In some embodiments, the cameras 210 may also capture images and / or video from the perspective of the user 204. For example, at least one of the cameras 210 may be aligned with the line of sight of the user 204. Thus, the cameras 210 may facilitate see-through functionality or features, and / or enable the head mounted display 202 to act as a mixed reality portal, where the real world environment of the user 204 is merged with virtual world content. Examples of the cameras 210 include, but are not limited to, digital cameras, RGB cameras, complementary metal oxide semiconductor (CMOS) image sensors, charge coupled device (CCD) image sensors, imaging sensor arrays, variations or combinations of one or more thereof, combinations of one or more thereof, and / or any other suitable camera.
[0049] like Figure 2As shown, the head-mounted display system 200 may include one or more inertial measurement units (IMUs), such as IMU 208. IMU 208 generally represents any type or form of sensor, instrument, and / or device that measures force (e.g., angular velocity and specific force of a subject). For example, when incorporated into and / or attached to the head-mounted display 202, IMU 208 can detect changes in the inertia of the head-mounted display 202. IMU 208 can use various mechanisms and / or tools (e.g., accelerometers, gyroscopes, magnetometers, and / or electronic compasses) to detect these changes. In some embodiments, the inertial measurement data generated by IMU 208 can indicate and / or be used to determine changes in the position and / or orientation of the head-mounted display 202.
[0050] In some examples, the term "head mounted display" may refer to any type or form of display device or system that is worn on or about a user's face and displays visual content to the user. The head mounted display may display content in any suitable manner, including via a screen (e.g., an LCD or LED screen), a projector, a cathode ray tube, an optical mixer, etc. The head mounted display may display content in one or more media formats. For example, the head mounted display 202 may display video, photos, computer generated imagery (CGI), and / or combinations thereof.
[0051] Head-mounted displays can provide diverse and unique user experiences. Some head-mounted displays can provide virtual reality experiences (i.e., they can display computer-generated or pre-recorded content), while other head-mounted displays can provide real-world experiences (i.e., they can display live images from the physical world). Head-mounted displays can also provide any mix of live content and virtual content. For example, virtual content can be projected onto the physical world (e.g., through optical or video perspective) to provide and / or facilitate an augmented reality or mixed reality experience. In some embodiments, the terms "augmented reality" and "mixed reality" can both refer to any type or form of content and / or experience that combines and / or combines virtual content with real-world content. Therefore, for the purpose of readability of this disclosure, the terms "augmented reality" and "mixed reality" can be used interchangeably and / or synonymously.
[0052] A head mounted display can be configured to be mounted to a user's head in a variety of ways. Some head mounted displays can be incorporated into glasses or visors. Other head mounted displays can be incorporated into helmets, hats, or other head mounted equipment.
[0053] like Figure 2As shown, user 204 can place head-mounted display system 200 on the user's head so that head-mounted display 202 is located and / or positioned on the user's face. By placing head-mounted display system 200 on his or her face in this manner, user 204 can place head-mounted display 202 over his or her eyes to experience and / or view virtual content presented on head-mounted display 202. User 204 can also secure head-mounted display 202 in the correct position by wrapping strap 206 around the back of the user's head.
[0054] Figure 3 2 shows exemplary components that may be incorporated into the head mounted display 202 for accelerated position tracking according to some embodiments. Figure 3 As shown, head mounted display 202 may include dedicated hardware device 100, camera 210, and display device 318. In some examples, display device 318 may include a screen that renders image frames (e.g., virtual reality, augmented reality, and / or mixed reality frames) for presentation to a user wearing head mounted display 202. Examples of display device 318 include, but are not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, a retinal projection display, variations or combinations of one or more thereof, and / or any other suitable display device.
[0055] like Figure 3 As shown, the dedicated hardware device 100 may include various components and / or hardware accelerated devices. For example, the dedicated hardware device 100 may include an image signal processor 102, a camera interface 302, a CV accelerator 304, a video encoder 306, a video decoder 308, a central processing unit 312, a digital signal processor 314, a display imaging pipeline 310, a communication path 320, and a frame output interface 106. All or part of these components may operate individually or in conjunction with each other when performing tasks such as tracking the position of the head mounted display 202 and / or generating image frames to be presented to a user of the head mounted display 202.
[0056] In some embodiments, the CV accelerator 304 may include various hardware-accelerated position tracking components responsible for performing various image processing operations. For example, the CV accelerator 304 may be specifically designed to perform position tracking of the head-mounted display 202. In some embodiments, the phrase "position tracking" may refer to any technology or algorithm that detects changes in the position, orientation, and / or inertia of a device relative to the physical environment in which the device is located. For example, position tracking may be used to detect when a user wearing a head-mounted display turns his or her head (thereby changing the orientation and / or inertia of the head-mounted display). Additionally or alternatively, position tracking may be used to detect when a user moves to a new location (e.g., by taking a step), thereby changing the distance between the user and one or more physical objects near the user.
[0057] Position tracking information can often be used to provide a more realistic and / or immersive virtual reality experience to a user of a head-mounted display. For example, the CV accelerator 304 can pass position information about the head-mounted display 202 to one or more other components of the dedicated hardware device 100 (e.g., the display imaging pipeline 310). When generating an augmented reality frame to be presented to the user, these components can correct and / or update the augmented reality frame to more accurately reflect and / or correspond to the current position of the head-mounted display 202.
[0058] Many conventional head-mounted display systems utilize graphics processing units (GPUs) to perform position tracking, as opposed to utilizing SoCs and / or ASICs. However, because GPUs are essentially general-purpose computing components, they are not specifically designed to perform such operations in virtual reality, augmented reality, and / or mixed reality environments. Therefore, when a GPU is deployed in a virtual reality, augmented reality, and / or mixed reality environment, the GPU may incorporate and / or implement certain features and / or functions that slow down such operations, thereby potentially causing or contributing to a lag and / or inaccurate fusion of virtual content and real content. For example, the latency exhibited by the position tracking system may cause the head-mounted display to lag behind the user's actual physical movement. This lag (sometimes referred to as motion-to-photon latency) can be a performance issue that affects the user experience of many head-mounted displays that use GPU-driven position tracking systems.
[0059] As a specific example of this deficiency, a conventional head-mounted display may include a GPU that is programmed to perform position tracking based on image frames captured by one or more cameras of the head-mounted display. The GPU may be designed so that after receiving the image frames from the camera, the GPU temporarily stores and / or buffers the image frames in DRAM. The GPU may then execute a position tracking algorithm for the head-mounted display based on the image frames stored in DRAM. In some examples, after and / or between various steps of the position tracking algorithm, the GPU may access the image frames in DRAM and / or save information extracted from the image frames to DRAM.
[0060] As mentioned above, many head-mounted display systems may utilize position tracking information while generating augmented reality frames to be presented to the user. Therefore, delays in detecting the movement of the head-mounted display may cause motion delays and / or inaccuracies within the augmented reality frames, especially relative to the position and / or orientation of the head-mounted display. Unfortunately, conventional GPU-driven position tracking systems may incur a delay or time penalty on each DRAM access (whether reading or writing). Therefore, in this GPU-driven embodiment, the amount of time required to generate an augmented reality frame may be extended and / or stretched, at least in part due to the general architecture of the GPU and / or its heavy reliance on DRAM buffers.
[0061] In some applications (e.g., a GPU-driven television or computer monitor), this delay may have little, if any, impact on the overall user experience. In fact, this delay may be completely unnoticeable. However, in virtual reality, augmented reality, and / or mixed reality environments, this delay may be noticeable or even obvious to some users, potentially resulting in a disjointed user experience that makes them feel uncomfortable.
[0062] To address this shortcoming, the head mounted display 202 may include and / or incorporate a dedicated hardware device 100 (e.g., a SoC and / or ASIC) that performs many of the position tracking operations that are traditionally handled by a GPU. As will be explained in more detail below, a SoC and / or ASIC driven CV system may outperform and / or improve upon a traditional GPU driven CV system for a variety of reasons.
[0063] Figure 4 An exemplary implementation 400 of the position tracking component 104 for accelerating position tracking of a head mounted display is shown in accordance with some embodiments. Figure 4As shown, implementation 400 may include CV accelerator 304, image frame 402, and memory device 418. Memory device 418 generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, memory device 418 may include and / or represent a DRAM device and / or a buffer. Other examples of memory device 418 include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD), optical drive, cache, a variation or combination of one or more of them, and / or any other suitable storage memory.
[0064] like Figure 4 As shown, the CV accelerator 304 may include an image processing component 404, an image reduction component 406, an oriented FAST rotation BRIEF (ORB, oriented FAST and rotated BRIEF) descriptor unit 408, a feature detection component 410, a search component 412, and a bundle adjustment unit 414. All or part of these units and components may be dedicated to performing one or more steps of the position tracking algorithm. In addition, these components and units may be implemented in hardware and / or software. For example, the image processing component 404, the image reduction component 406, the feature detection component 410, and / or the search component 412 may be implemented in hardware alone or primarily. This hardware-based implementation may accelerate the operations or steps that the components are assigned to perform. In contrast, the ORB descriptor unit 408 and / or the bundle adjustment unit 414 may be implemented in software alone or primarily. This software-based implementation may enable these units to be programmably updated.
[0065] In implementation 400, image frames 402 generally represent image frames captured by one or more of the cameras 210 of the head mounted display 202. When image frames 402 are received by the dedicated hardware device 100 (e.g., received by the image signal processor 102 within the dedicated hardware device 100), the dedicated hardware device 100 may pass the image frames 402 to the CV accelerator 304. Components and units within the CV accelerator 304 may then perform a position tracking algorithm based on one or more of the image frames 402. The position tracking algorithm may detect changes in the position of the head mounted display 202. As part of and / or during the algorithm, components and / or units of the CV accelerator 304 may read data extracted from the image frames 402 and / or write data extracted from the image frames 402 to a memory device 418. In addition, the CV accelerator 304 may store the detected changes in the position of the head mounted display 202 within the memory device 418. These changes may later be accessed by one or more additional components of the dedicated hardware device 100 (e.g., the display imaging pipeline 310).
[0066] Figure 5 An exemplary implementation 500 of the CV accelerator 304 for accelerating position tracking of a head mounted display is shown in accordance with some embodiments. Specifically, Figure 5 A front-end processing component 502 is shown. The front-end processing component 502 generally represents any type or form of processing unit or device that analyzes all or a portion of the image frame 402 as the image frame 402 is received by the CV accelerator 304. Specifically, the front-end processing component 502 can perform one or more online operations on the image frame 402 before the image frame 402 is stored in the main memory (e.g., DRAM 510) of the dedicated hardware device 100. In contrast, many conventional position tracking systems can first store and / or buffer received image frames in a portion of DRAM or a similar type of memory before processing the image frames. By avoiding this initial storage step, the dedicated hardware device 100 can reduce the time and / or computing resources involved in tracking changes in the position of the head-mounted display 202.
[0067] In one embodiment, the front-end processing component 502 may represent a hardware accelerated component. For example, the front-end processing component 502 may represent an integrated device or circuit that is specifically designed and manufactured to perform initial image processing operations on input image frames.
[0068] In implementation 500, front-end processing component 502 may include image processing component 404, image reduction component 406, feature detection component 410, and cache 508. Front-end processing component 502 may include any additional or alternative processing units, components, and / or portions of memory for any suitable purpose. For example, front-end processing component 502 may include Figure 4 An ORB descriptor unit 408, a search component 412 and / or a bundle adjustment unit 414 are shown.
[0069] Cache 508 generally represents any portion of a fast access memory or storage device. For example, cache 508 may represent and / or include a portion of a static random access memory (SRAM). In some embodiments, cache 508 may have a faster access time than DRAM 510. In other words, the dedicated hardware device 100 may read data stored in cache 508 more quickly than data stored in DRAM 510. In some examples, when an image frame 402 is received from camera 210, front-end processing component 502 may load image frame 402 into cache 508. Then, one or more components of front-end processing component 502 may perform one or more online image processing operations on image frame 402 in cache 508.
[0070] In one embodiment, the image processing component 404 may represent and / or include a lightweight image signal processor. In this embodiment, the image processing component 404 may use image signal processing such as gamma correction, Gaussian blur and / or acoustic emission signal processing to process the image frame 402. Such processing may enable additional components of the front-end processing component 502 to more effectively and / or accurately analyze and / or further process the image frame 402.
[0071] In one example, the image processing component 404 may pass the processed version of the image frame 402 to the image reduction component 406. In other examples, the image reduction component 406 may receive the image frame 402 directly from the image signal processor 102 (thereby bypassing the processing operations performed by the image processing component 404). After receiving the image frame 402, the image reduction component 406 may generate one or more low-resolution versions of the image frame 402. For example, the image reduction component 406 may sample the image frame 402 and / or otherwise reduce the resolution of the image frame 402.
[0072] In some embodiments, the image reduction component 406 can generate multiple reduced versions of the image frame 402. Each of these reduced versions of the image frame 402 can have different resolutions. As an example, if one of the image frames 402 is initially composed of 256 pixels, the image processing component 404 can reduce the image frame to 128 pixels, 64 pixels, and 32 pixels (creating a total of four versions of the image frame). This process of generating multiple reduced versions of the image frame can be referred to as generating a pyramid of the image frame. After performing the pyramid generation process on one of the image frames 402, the image reduction component 406 can (e.g., via DRAM access 504) write the resulting image frame to DRAM 510.
[0073] Feature detection component 410 within front-end processing component 502 generally represents any hardware-accelerated component, integrated circuit, and / or device dedicated to detecting one or more features within image frame 402. In some embodiments, the term "feature" can refer to a physical object and / or a portion of a physical object within the physical environment surrounding the head-mounted display. For example, a feature can represent a point or corner of a physical object that is visually distinguishable within the object and / or visually distinguishable from adjacent objects.
[0074] Feature detection component 410 can detect features within an image frame in a variety of ways. In one example, feature detection component 410 can identify one or more features within an image frame by performing a non-maxima suppression analysis of an image frame. The analysis can involve calculating the gradient of pixel values within the image frame, and then identifying positions within the image frame where the pixel value gradient is higher (e.g., steeper) than a certain threshold and / or higher than the pixel value gradient of surrounding positions. Feature detection component 410 can use any additional or alternative image processing techniques to detect features within image frame 402.
[0075] In some examples, feature detection component 410 can identify a set of key features within image frame 402. According to certain embodiments, the term "key features" can generally refer to features selected for tracking and / or analysis as part of a position tracking algorithm. In some embodiments, feature detection component 410 can identify and then track a certain number of key features (e.g., 20 key features). In these embodiments, after compiling a set of key features, CV accelerator 304 can attempt to identify a representation of each key feature within the input image frame.
[0076] The feature detection component 410 can detect key features in a variety of ways. In one embodiment, the feature detection component 410 can identify key features by detecting particularly different features (e.g., compared to other features within the image frame). Additionally or alternatively, the feature detection component 410 can identify key features by detecting new features within one or more image frames 402. For example, the feature detection component 410 can search for features in the input image frame that have not yet been identified and stored in the DRAM 510. In one embodiment, the feature detection component 410 can search for new features within each image frame received by the front-end processing component 502. In other embodiments, the feature detection component 410 can search for new features within certain key image frames (e.g., every three image frames received by the front-end processing component 502).
[0077] In one example, the search component 412 can incorporate one or more key features into the image map, such as Figure 5 Image map 514 shown. In some embodiments, the term "image map" may refer to a representation or description of the physical location (e.g., three-dimensional location) of one or more key features within a physical environment. In one example, image map 514 may indicate the absolute physical location of the key features. For example, image map 514 may indicate the location of the key features within a room or in other areas around head mounted display 202. Additionally or alternatively, image map 514 may indicate the relative physical location of the key features. For example, image map 514 may indicate the distance and / or orientation of the key features relative to head mounted display 202.
[0078] Search component 412 can calculate the physical location of the key feature in a variety of ways. In one embodiment, search component 412 can perform an epipolar search to find the three-dimensional location of the key feature. In some examples, the term "episodic search" can refer to any type or form of analysis that identifies the three-dimensional location of an object within a physical environment based on multiple two-dimensional representations (e.g., images) of the physical environment.
[0079] exist Figure 5 In the example of , search component 412 can include and / or represent a hardware-accelerated component dedicated at least in part to performing an epipolar search. In this example, search component 412 can identify the physical location of the key feature by performing an epipolar search based on two or more image frames containing representations of the key feature in image frame 402. Search component 412 can then incorporate the location of the key feature into image map 514.
[0080] In some embodiments, when feature detection component 410 detects a key feature within image frame 402, search component 412 can merge the key feature into image map 514. In addition, search component 412 can remove a key feature from image map 514 for various reasons, such as determining that a new key feature has been identified, determining that the key feature is no longer represented within the input image frame, and / or after a certain amount of time has passed since the key feature was merged into image map 514. As will be explained in more detail below, by maintaining an image map of the physical locations of features within the physical environment surrounding head mounted display 202, CV accelerator 304 can track the position and / or orientation of head mounted display 202 within the physical environment.
[0081] In implementation 500, search component 412 can read data from and write data to DRAM 510. In one embodiment, search component 412 can read (e.g., via DRAM access 506) a reduced version of image frame 402 generated and stored by image reduction component 406 and / or key features detected and stored by feature detection component 410. In addition, search component 412 can write (e.g., via DRAM access 512) the location of the key features to image map 514. In contrast, because front-end processing component 502 performs initial online processing operations on image frame 402, front-end processing component 502 can avoid and / or reduce reading from DRAM 510.
[0082] Figure 6 An exemplary implementation 600 of the front-end processing component 502 according to some embodiments is shown. Specifically, Figure 6 6 shows the flow of image frame 402 as it is processed by front-end processing component 502. In this example, front-end processing component 502 may include two different processing paths along which image frame 402 may be directed. In one embodiment, front-end processing component 502 may direct one or more of image frames 402 along processing path 602. Processing path 602 may involve writing image frame 402 directly to main memory (e.g., DRAM 510). In some examples, processing path 602 may represent and / or be similar to a processing path of a conventional position tracking system that buffers input image frames within DRAM prior to processing the image frames. Alternatively, front-end processing component 502 may direct all or a portion of image frame 402 along processing path 604. Processing path 604 may involve performing one or more online operations (as described above in conjunction with processing) on image frame 402 before image frame 402 is written to DRAM 510. Figure 5 discussed).
[0083] like Figure 6As shown, the processing path 604 can pass the image frame 402 to the image processing component 404. After the image processing component 404 performs one or more image signal processing operations (e.g., gamma correction and / or Gaussian blur) on the image frame 402, the image processing component 404 can pass the processed image frame to the image reduction component 406, the feature detection component 410, and / or the transformation unit 606. In some embodiments, the image reduction component 406, the feature detection component 410, and the transformation unit 606 can process and / or analyze the image frame 402 simultaneously. For example, these components and units can work independently and / or in parallel.
[0084] As combined Figure 5 As discussed, the image reduction component 406 can generate one or more low-resolution versions of the input image frame. Figure 6 In the example of , the image reduction component 406 can generate a low-resolution image 608 and then write the low-resolution image 608 to the DRAM 510. In addition, the feature detection component 410 can detect one or more key features within the input image frame. Figure 6 In the example of , feature detection component 410 may detect key feature 610 and then write an indication of key feature 610 to DRAM 510 (eg, by storing pixel values of pixels within key feature 610).
[0085] In some examples, processing path 604 may optionally include passing image frame 402 from image processing component 404 to transform unit 606. Transform unit 606 generally represents any hardware accelerated component dedicated to transforming and / or processing a full resolution version of an image frame before writing it to DRAM 510. In one embodiment, transform unit 606 may perform a finite Legendre transform (FLT) on an input image frame. Transform unit 606 may then write the resulting image frame (i.e., full resolution image 612) to DRAM 510. Transform unit 606 may perform any additional or alternative image processing and / or image transformation algorithms on image frame 402.
[0086] In some embodiments, the dedicated hardware device 100 may pass one or more of the image frames 402 through the front-end processing component 502 multiple times. For example, after the image reduction component 406 generates the low-resolution image 608, the dedicated hardware device 100 may forward the low-resolution image 608 along the data path 614. The data path 614 may direct the low-resolution image 608 through the image processing component 404 and / or the feature detection component 410. In some examples, the feature detection component 410 may recalculate the presence and / or location of the key feature 610 within the low-resolution image 608. The feature detection component 410 may then store the resulting low-resolution version of the key feature 610 within the DRAM 510. In some embodiments, the front-end processing component 502 may pass each low-resolution version of the image frame generated by the image reduction component 406 along the data path 614.
[0087] In some examples, the dedicated hardware device 100 may include multiple instances of the front-end processing component 502. For example, the dedicated hardware device 100 may include one instance of the front-end processing component 502 for each camera in the cameras 210. In this way, the CV accelerator 304 may simultaneously process and / or analyze image frames received from each camera. In these examples, each instance of the front-end processing component 502 may write data about the input image frame to the same portion of the DRAM (i.e., DRAM 510).
[0088] Figure 7 An exemplary implementation 700 of a search component 412 is shown in accordance with some embodiments. In this example, the search component 412 may represent a software and / or hardware-based component of the CV accelerator 304 that is dedicated to determining the location of a representation of a key feature within the image frame 402. In some embodiments, the term "representation of a key feature" may refer to a depiction and / or image of a key feature within an image frame. Thus, the location of a representation of a key feature may refer to the location of a pixel within an image frame that depicts the key feature (rather than the location of the key feature within the physical environment).
[0089] In implementation 700, search component 412 can identify the location of a representation of key feature 610 within image frame 702. Image frame 702 generally represents any image frame received and / or processed by front-end processing component 502. For example, image frame 702 can represent Figure 6 608 in the full resolution image 612 and / or the low resolution image 608. In some embodiments, searching for a representation of the key feature 610 within the high resolution image frame and one or more low resolution image frames can help track the position of the head mounted display 202 more accurately and / or more precisely.
[0090] like Figure 7As shown, the search component 412 can access the image frame 702 within the DRAM 510. The search component 412 can also access one or more image blocks within the DRAM 510, such as the image block 704. In some embodiments, the term "image block" can refer to a representation, depiction, and / or image of a key feature. For example, an image block can include and / or represent a portion of an image frame depicting a particular key feature. In one example, an image block can represent the physical appearance of a key feature from a particular viewing angle and / or viewing angle. For example, the image block 704 can represent and / or indicate the physical appearance of the key feature 610 at a particular angle and / or a particular distance relative to the head mounted display 202.
[0091] In some embodiments, the CV accelerator 304 may generate and / or store multiple image tiles for each key feature incorporated into the image map 514. Each of these image tiles may represent the key feature at a different perspective. In addition, the CV accelerator 304 may generate and / or store image tiles of various resolutions.
[0092] like Figure 7 As shown, the search component 412 may include a block search unit 708 and a block warping unit 710. In one example, the block warping unit 710 may load and / or access an image block 704 within the DRAM 510. The block warping unit 710 may then warp the image block 704 so that the image block 704 is similar to and / or matches the current physical appearance of the key feature 610. In implementation 700, the warping process may generate a warped image block 712.
[0093] In one embodiment, block warping unit 710 may generate warped image block 712 by adjusting and / or changing one or more pixels within image block 704 so that image block 704 more closely resembles a current physical appearance of key feature 610 (e.g., the appearance of key feature 610 from a current viewing angle of head mounted display 202) than the appearance of key feature 610 depicted within image block 704. Block warping unit 710 may perform such warping using any type or form of image processing technique, such as anti-aliasing filtering and / or pixel interpolation.
[0094] After the block warping unit 710 generates the warped image block 712, the block search unit 708 can identify the location of the representation of the key feature 610 within the image frame 702 based on the warped image block 712. In some examples, the block search unit 708 can identify the location while analyzing the image frame 702 within a cache, buffer, and / or other portion of a fast access memory. In implementation 700, the block search unit 708 can load all or a portion of the image frame 702 from the DRAM 510 into the cache 714. In this example, the cache 714 can represent a cache dedicated to and / or accessible by the search component 412 (as opposed to Figure 5 In contrast to the cache 508 that may be contained within the front-end processing component 502).
[0095] The block search unit 708 may search for the location of the representation of the key feature 610 within the image frame 702 in a variety of ways. In some examples, the block search unit 708 may identify the location within the image frame 702 where the pixel value of the distorted image block 712 most closely matches the corresponding pixel value of the image frame 702. In some embodiments, the term "pixel value" may refer to a numerical value representing and / or indicating the brightness and / or color of a pixel.
[0096] The block search unit 708 may implement various algorithms and / or image processing techniques to accurately and / or efficiently compare the pixel values of the distorted image block 712 with the pixel values of the image frame 702. In some examples, the block search unit 708 may implement a sum of squared differences (SSD) algorithm or a similar type of algorithm. In one embodiment, performing the SSD algorithm may represent an initial analysis. The block search unit 708 may use techniques such as pixel interpolation, parabola fitting, and / or a Kanade-Lucas-Tomasi (KLT) feature tracking algorithm to improve the initial analysis. In some embodiments, performing these subsequent analyses may enable the block search unit 708 to identify the sub-pixel location of the representation of the key feature 610 within the image frame 702. In addition, in some examples, the block search unit 708 may perform multiple iterations (e.g., four iterations) of one or more of the above-mentioned analyses to improve the precision and / or accuracy of the location of the representation of the key feature 610.
[0097] Although Figure 7 Only the distorted image block 712 is shown, but any additional or alternative image blocks can be generated and / or utilized by the search component 412. For example, the block distortion unit 710 can generate a plurality of distorted image blocks, each distorted image block representing the physical appearance of the key feature 610 from a different perspective and / or at a different resolution. The block search unit 708 can then search the image frame 702 for representations of the key feature 610 based on all or a portion of these distorted image blocks.
[0098] like Figure 7 As shown, block search unit 708 can output motion vector 716. Motion vector 716 generally represents and / or indicates any type or form of difference between a determined position of a representation of key feature 610 within image frame 702 and an expected position of the representation. For example, motion vector 716 can indicate an offset between a position of a representation of key feature 610 within image frame 702 and a position of a representation of key feature 610 within a previously received image frame. In some embodiments, search component 412 can use motion vector 716 to predict and / or estimate a position of a representation of key feature 610 within a subsequent image frame.
[0099] After determining the position of the representation of the key feature 610 within the image frame 702, the search component 412 can determine the current physical position of the head mounted display 202. For example, the search component 412 can infer and / or calculate the position of the head mounted display 202 within the image map 514. The position can correspond to and / or indicate the latest physical position of the head mounted display 202 within the physical environment described by the image map 514. The search component 412 can determine the position in a variety of ways. In one embodiment, based on tracking the representation of the key feature 610 (or a set of key features) across multiple image frames, the search component 412 can determine that the head mounted display 202 has physically changed position relative to the key feature 610. The search component 412 can determine the amount and / or direction that the head mounted display 202 has moved based on the motion vector 716, the physical position of the key feature 610 indicated by the image map 514, and / or any additional or alternative position data.
[0100] In some examples, the search component 412 can communicate the current physical position and / or changes in the physical position of the head mounted display 202 to one or more additional components of the dedicated hardware device 100 (e.g., the display imaging pipeline 310). For example, the search component 412 can store an indication of the current physical position of the head mounted display 202 within the DRAM 510. Additionally or alternatively, the search component 412 can directly forward the current physical position of the head mounted display 202 to the display imaging pipeline 310. In some examples, the search component 412 can continue to detect changes in the position of the head mounted display 202 and / or update the current position of the head mounted display 202 based on new image frames received by the CV accelerator 304.
[0101] In some embodiments, one or more components of the search component 412 may be dedicated to performing multiple steps of the position tracking algorithm. For example, as discussed above, the search component 412 may perform an epipolar search to identify the physical location of key features within the physical environment surrounding the head mounted display 202. In one embodiment, a block search unit 708 within the search component 412 may perform the search. Thus, the block search unit 708 may be dedicated to performing image block searches and epipolar searches. For example, the block search unit 708 may represent and / or include a hardware accelerated component that is designed to efficiently perform the mathematical operations involved in both types of searches. By dedicating and / or reusing certain components of the dedicated hardware device 100 for multiple functions, the disclosed position tracking system may reduce power consumption and on-chip space of the dedicated hardware device 100.
[0102] As mentioned above, the block search unit 708 may load all or a portion of the image frame 702 into the cache 714. In one embodiment, the block search unit 708 may load a specific portion of the image frame 702 into the cache 714 (and then search it). This portion of the image frame 702 may be referred to as a tile. In some embodiments, the term "tile" may refer to an area of an image frame that is expected to contain a representation of one or more key features.
[0103] Figure 8 An exemplary tile 802 within the image frame 702 is shown. Figure 8 As shown, tile 802 can include search area 804 and search area 806. In this example, search area 804 can represent an area in image frame 702 where a representation of key feature 610 is expected to be located. For example, search component 412 can determine that a representation of key feature 610 is found in a corresponding and / or similar area in a previously received image frame. In this way, search component 412 can determine that the current representation of key feature 610 is likely within and / or near the previously identified area.
[0104] exist Figure 8In the example of , the search component 412 can determine that the representation of the additional key feature may be located within the search area 806. In addition, the search component 412 can determine that the search area 804 and the search area 806 at least partially overlap and / or are within a certain proximity (proximity) of each other. Therefore, the search component 412 can select and / or allocate the tile 802 so that the tile 802 contains the search area 804 and the search area 806. Then, the search component 412 can load the tile 802 into the cache 714. In this way, the search component 412 can identify the location of the representation of the key feature 610 and the location of the representation of the additional key feature based on the data loaded into the cache. By dividing the image frame 702 into tiles, the search component 412 can ensure that each portion of the image frame 702 is only loaded into the cache once, thereby reducing the time and computing resources involved in tracking the key features within the image frame.
[0105] As above combined Figure 2 As discussed, the head mounted display 202 may include and / or be equipped with an IMU 208. In some examples, the IMU 208 may communicate inertial measurement data to the dedicated hardware device 100. For example, as the user operates the head mounted display 202, the IMU 208 may periodically or continuously monitor changes in the inertia of the head mounted display 202. The IMU 208 may then forward this data in real time to one or more components of the dedicated hardware device 100, such as the CV accelerator 304 and / or the display imaging pipeline 310.
[0106] In some embodiments, the dedicated hardware device 100 can utilize inertial measurement data generated by the IMU 208 to detect changes in the position and / or orientation of the head-mounted display 202. For example, the inertial measurement data provided by the IMU 208 can indicate the speed and / or acceleration at which the head-mounted display 202 moves and / or changes direction. By processing (e.g., integrating) the data provided by the IMU 208, the CV accelerator 304 and / or the display imaging pipeline 310 can determine the position of the head-mounted display 202 relative to the initial position. In some embodiments, the dedicated hardware device 100 can combine and / or compare the inertial measurement data provided by the IMU 208 with the feature tracking data generated by the search component 412. In this way, the dedicated hardware device 100 can more accurately and / or more efficiently determine the physical position of the head-mounted display 202.
[0107] Fig. 9 An exemplary implementation 900 of a dedicated hardware device 100 for accelerating position tracking of a head mounted display according to some embodiments is shown. Fig. 9As shown, implementation 900 may include and / or involve head mounted display 202 and standalone host device 906 communicating with each other via communication link 908. In one example, standalone host device 906 may represent an external device that provides computer generated imagery (e.g., virtual image content 904) to be overlaid on image frames captured by head mounted display 202. For example, standalone host device 906 may provide imagery used by dedicated hardware device 100 to generate mixed reality and / or augmented reality frames. In implementation 900, standalone host device 906 may represent a personal computer (PC) and / or a game console. In this example, standalone host device 906 may include a physical processor 930, a communication interface 932, and a memory 940.
[0108] like Fig. 9 As shown, the head mounted display 202 may include a dedicated hardware device 100, a camera 210, a communication interface 934, and a memory device 418. In one example, the independent host device 906 and the head mounted display 202 may establish and / or form a communication link 908 between each other through the communication interfaces 932 and 934, respectively. In this example, one or more of the cameras 210 on the head mounted display 202 may capture an image frame 902. The image frame 902 may be captured from the perspective of a user wearing the head mounted display 202. In one embodiment, the image frame 902 may be different from the image frame 402 captured for tracking the position of the head mounted display 202.
[0109] In one example, the standalone host device 906 can feed the computer-generated imagery to the head mounted display 202 via the communication link 908 so that the head mounted display 202 can fuse the imagery with the image frame 902. The resulting experience can appear to the user as a mix of reality and virtual reality. Thus, the head mounted display 202 can effectively function as a perspective camera that enables the user to view his or her surroundings modified by the virtual objects fed from the standalone host device 906.
[0110] While generating the mixed reality and / or augmented reality frames, the dedicated hardware device 100 may identify the current position and / or orientation of the head mounted display 202 based on information generated and / or output by the CV accelerator 304. For example, the display imaging pipeline 310 within the dedicated hardware device 100 may access the current position information of the head mounted display 202 within the DRAM 510. In some embodiments, this information may enable the display imaging pipeline 310 to more efficiently and / or more accurately overlay the computer generated imagery provided by the independent host device 906 onto the image frame 902.
[0111] As a specific use case example, a user may wear and / or operate a head mounted display 202 in his or her living room. In this example, the head mounted display 202 may capture a video of his or her surroundings (including, for example, living room furniture) and then fuse the video with virtual objects fed from a nearby PC. After completing the corresponding processing and / or fusion, the video displayed to the user within the head mounted display 202 may display and / or render a virtual covering (or so-called skin) on his or her couch in the living room, thereby manipulating the appearance of the couch from the user's perspective. Based on the precise and real-time position information of the head mounted display 202 provided by the CV accelerator 304, the dedicated hardware device 100 may accurately and efficiently adjust and / or display virtual content, such as the size and / or position of a virtual covering within the video. Thus, the disclosed apparatus and system may improve the overall experience of the user wearing the head mounted display 202.
[0112] Fig.10 is a flow chart of an exemplary method 1000 for accelerating position tracking of a head mounted display according to any embodiments disclosed herein. Fig.10 The steps shown in can be performed by a dedicated hardware device (such as an ASIC or SoC) incorporated into the head-mounted display. In addition, Fig.10 The steps shown in the above may also be combined and / or involve the above Figure 1-Figure 9 Various sub-steps and / or variations consistent with the description are described.
[0113] like Fig.10 As shown, the dedicated hardware device may receive an image frame of the physical environment surrounding the head mounted display system from a camera device of the head mounted display system at step 1010. For example, the camera 210 may capture the image frame 402 and then forward it to the dedicated hardware device 100. When the image frame 402 arrives, the dedicated hardware device 100 may receive and / or process the image frame 402 as part of and / or in preparation for a position tracking algorithm.
[0114] exist Fig.10 In step 1020 of the embodiment, when an image frame is received from the camera device, the dedicated hardware device may store at least a portion of the image frame in a cache of the dedicated hardware device. For example, the dedicated hardware device 100 may store the image frame 702 in the cache 508 of the front-end processing component 502. In this example, the cache 508 may have a faster access time than the DRAM 510.
[0115] exist Fig.10In step 1030 of , the dedicated hardware device can track changes in the position of the head mounted display system within the physical environment based at least in part on the portion of the image frames stored in the cache. For example, the front-end processing component 502 of the dedicated hardware device 100 can identify a set of key features (including the key features 610) based on the image frames 402 stored in the cache 508. Then, the search component 412 can track the position of the head mounted display 202 based at least in part on tracking the positions of the representations of the key features 610 across multiple image frames.
[0116] exist Fig.10 At step 1040 in the dedicated hardware device 100, the change in the position of the head mounted display system can be stored in the main memory for use in generating at least one augmented reality frame to be presented to the user wearing the head mounted display system. For example, the search component 412 can write the current position tracking information about the head mounted display 202 to the DRAM 510. Then, the display imaging pipeline 310 and / or additional components of the dedicated hardware device 100 can use the information stored in the DRAM 510 to generate the augmented reality frame.
[0117] exist Fig.10 At step 1050 in the embodiment, the dedicated hardware device may feed the augmented reality frame to the display device of the head mounted display system so that the display device presents the augmented reality frame to the user wearing the head mounted display system. For example, the dedicated hardware device 100 may pass and / or forward the augmented reality frame to the display device 318 of the head mounted display 202. Once the augmented reality frame is received, the display device 318 may display the augmented reality frame to the user 204.
[0118] Embodiments of the present disclosure may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or content generated in combination with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or to be used in other ways in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.
[0119] The order of process parameters and steps described and / or illustrated herein are given as examples only and may vary as desired. For example, although the steps illustrated and / or described herein may be shown or discussed in a particular order, the steps do not necessarily need to be performed in the order illustrated or discussed. The various exemplary methods described and / or illustrated herein may also omit one or more steps described or illustrated herein, or include additional steps in addition to those disclosed.
[0120] The foregoing description is provided to enable others skilled in the art to best utilize the various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the present disclosure. The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. When determining the scope of the present disclosure, reference should be made to the attached claims and their equivalents.
[0121] Unless otherwise indicated, the terms "connected to" and "coupled to" (and their derivatives) as used in the specification and claims should be interpreted as allowing both direct and indirect (i.e., via other elements or components) connections. Furthermore, the terms "a" or "an" as used in the specification and claims should be interpreted as meaning "at least one of." Finally, for ease of use, the terms "including" and "having" (and their derivatives) as used in the specification and claims are interchangeable with and have the same meaning as the word "comprising."
Claims
1. A dedicated hardware device, comprising: a camera interface that receives image frames of a physical environment surrounding the head mounted display system from a camera device of the head mounted display system; a position tracking component communicatively coupled to the camera interface, wherein the position tracking component: processing at least a portion of the image frame in a hardware acceleration block of the dedicated hardware device; maintaining an image map within a main memory of the dedicated hardware device, the image map identifying one or more physical locations of features within the physical environment; detecting at least one of the features of the physical environment within an input image frame as the input image frame is received; as well as After detecting the features within the input image frame, transferring the features to a main memory of the dedicated hardware device to facilitate incorporation of the features into an image map identifying physical locations of the features within the physical environment; tracking changes in a position of the head mounted display system within the physical environment based at least in part on a portion of the image map; and storing the change in position of the head mounted display system for use in generating at least one augmented reality frame to be presented to a user wearing the head mounted display system; and A frame output interface is communicatively coupled to the position tracking component, wherein the frame output interface feeds the augmented reality frame to a display device of the head mounted display system for presentation to a user.
2. The dedicated hardware device according to claim 1, wherein: The feature of the physical environment represents at least a portion of a physical object located within the physical environment.
3. The dedicated hardware device according to claim 1, wherein: The hardware acceleration block of the dedicated hardware device includes a hardware accelerated image processing component, wherein the hardware accelerated image processing component: When an input image frame is received at the hardware accelerated image processing component, creating at least one low resolution version of the input image frame; and A low resolution version of the input image frame is transferred to a main memory of the dedicated hardware device to facilitate tracking changes in the position of the head mounted display system.
4. The dedicated hardware device according to claim 1, wherein: The position tracking component identifies a set of key features within the physical environment that facilitate tracking changes in the position of the head mounted display system.
5. The dedicated hardware device according to claim 4, wherein: The position tracking component tracks the change of the position of the head mounted display system by: Identifying an initial position of a representation of at least one key feature of the physical environment within at least one of: an initial image frame captured by the camera device; or an image map identifying physical locations of the set of key features within the physical environment; identifying a subsequent location of a representation of the key feature of the physical environment within a subsequent image frame captured by the camera device; and An initial position of the representation of the key feature is compared to a subsequent position of the representation of the key feature.
6. The dedicated hardware device according to claim 5, wherein: The position tracking component identifies a subsequent position of the representation of the key feature within the subsequent image frame by: Determining an expected area of the subsequent image frame in which the key feature of the physical environment may be represented is based on at least one of: the initial position of the representation of the key feature within the initial image frame; or the image map identifying the physical locations of the set of key features within the physical environment; and The specific location of the representation of the key feature within the subsequent image frame is identified by searching the expected area of the subsequent image frame.
7. The dedicated hardware device according to claim 6, wherein: The position tracking component: generating one or more image patches, the one or more image patches representing the physical appearance of the key features of the physical environment from different perspectives; and The image blocks are stored in a main memory of the dedicated hardware device.
8. The dedicated hardware device according to claim 7, wherein: The position tracking component includes an image search unit, which identifies the specific position of the representation of the key feature in the subsequent image frame by: warping at least one image patch so that the image patch resembles a current physical appearance of the key feature from a perspective at which the camera device captures the subsequent image frame; and Locations within the subsequent image frame where pixel values of the warped image block most closely match corresponding pixel values of the subsequent image frame are identified.
9. The dedicated hardware device according to claim 8, wherein: The image search unit identifies the specific location of the representation of the key feature in the subsequent image frame by: performing sub-pixel interpolation analysis on pixel values within the expected area of the subsequent image frame; and Based on the sub-pixel interpolation analysis, sub-pixel locations where pixel values of the warped image block most closely match corresponding pixel values of the subsequent image frame are determined.
10. The dedicated hardware device according to claim 9, wherein: The image search unit: determining that additional key features of the physical environment are expected to be represented within the expected area of the subsequent image frame; loading the expected region of the subsequent image frame into a cache of the dedicated hardware device; and The specific location of the representation of the key feature within the subsequent image frame and the specific location of the representation of the additional key feature within the subsequent image frame are identified by searching the expected area of the subsequent image frame within the cache of the dedicated hardware device.
11. The dedicated hardware device according to claim 6, wherein: The hardware acceleration block of the dedicated hardware device includes a hardware accelerated search component dedicated to the following two: identifying a specific location of a representation of the key feature within the subsequent image frame; and A physical location of the key feature within the physical environment is identified.
12. The dedicated hardware device according to claim 11, wherein: The hardware-accelerated search component identifies the physical location of the key feature within the physical environment by: identifying a specific location of a representation of the key feature within a previous image frame captured by the camera device; and Based on the specific location of the representation of the key feature within the previous image frame, an epipolar search is performed to find the three-dimensional location of the key feature within the physical environment.
13. The dedicated hardware device according to claim 1, further comprising a frame generation unit, wherein: Retrieving a change in position of the head mounted display system; and While generating the augmented reality frame, at least one aspect of the augmented reality frame is adjusted to account for a change in the position of the head mounted display system.
14. The dedicated hardware device according to claim 1, wherein: The position tracking component processes the portion of the image frame in a hardware acceleration block of the dedicated hardware device when the image frame is received from the camera interface and before the image frame is stored in a main memory of the dedicated hardware device.
15. The dedicated hardware device according to claim 1, wherein: The position tracking component also: storing at least a portion of the image frame directly in a main memory of the dedicated hardware device before the image frame is processed by a hardware acceleration block of the dedicated hardware device; and Changes in the position of the head mounted display system are tracked based at least in part on the portion of the image frame stored directly in the main memory of the dedicated hardware device.
16. The dedicated hardware device according to claim 1, further comprising an image signal processor, wherein: receiving at least a portion of the image frame from the camera interface; performing at least one image processing operation on the received image frame; and The processed image frames are forwarded to the position tracking component.
17. A head mounted display system, comprising: at least one camera device that captures image frames of the physical environment surrounding the head mounted display system; a dedicated hardware device communicatively coupled to the camera device, wherein the dedicated hardware device: processing at least a portion of the image frame in a hardware acceleration block of the dedicated hardware device; maintaining an image map within a main memory of the dedicated hardware device, the image map identifying one or more physical locations of features within the physical environment; detecting at least one of the features of the physical environment within an input image frame as the input image frame is received; as well as After detecting the features within the input image frame, transferring the features to a main memory of the dedicated hardware device to facilitate incorporation of the features into an image map identifying physical locations of the features within the physical environment; tracking changes in a position of the head mounted display system within the physical environment based at least in part on a portion of the image map; and storing the change in position of the head mounted display system for use in generating at least one augmented reality frame to be presented to a user wearing the head mounted display system; and a display device communicatively coupled to the dedicated hardware device, wherein the display device: receiving the augmented reality frame from the dedicated hardware device; and The augmented reality frame is presented to a user wearing the head mounted display system.
18. The head mounted display system of claim 17, further comprising at least one communication interface communicatively coupled to an independent host device, the independent host device feeding virtual image content from the independent host device to the head mounted display system.
19. A method for accelerating position tracking of a head mounted display, comprising: Receiving, by a dedicated hardware device, an image frame of a physical environment surrounding the head mounted display system from a camera device of the head mounted display system; processing, by the dedicated hardware device, at least a portion of the image frame in a hardware acceleration block of the dedicated hardware device; maintaining an image map within a main memory of the dedicated hardware device, the image map identifying one or more physical locations of features within the physical environment; detecting at least one of the features of the physical environment within an input image frame as the input image frame is received; as well as After detecting the features within the input image frame, transferring the features to a main memory of the dedicated hardware device to facilitate incorporation of the features into an image map identifying physical locations of the features within the physical environment; tracking, by the dedicated hardware device, changes in the position of the head mounted display system within the physical environment based at least in part on a portion of the image map; storing, by the dedicated hardware device, the change in the position of the head mounted display system for use in generating at least one augmented reality frame to be presented to a user wearing the head mounted display system; and The dedicated hardware device feeds the augmented reality frame to a display device of the head-mounted display system, so that the display device presents the augmented reality frame to a user wearing the head-mounted display system.
