Method and device for corner detection

Abstract

A device configured to be worn on the head of a user comprises: a screen configured to present graphics to the user; a camera system configured to view an environment in which the user is located; and a processing unit coupled to the camera system, the processing unit being configured to: obtain a first image having a first resolution, the first image having a first corner, determine a second image having a second resolution, the second image having a second corner corresponding to the first corner in the first image, wherein the second image is based on the first image, the second resolution is smaller than the first resolution, detect the second corner in the second image, determine a position of the second corner in the second image, and determine a position of the first corner in the first image based at least in part on the determined position of the second corner in the second image.

Description

Technical Field

[0001] The present disclosure relates to a head-mounted image display device, and a method and apparatus for detecting a corner in an image generated by the head-mounted image display device. Background Art

[0002] Modern computing and display technologies have facilitated the development of "mixed reality" (MR) systems for so-called "virtual reality" (VR) or "augmented reality" (AR) experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that they appear to be real or can be perceived as real. VR scenarios typically involve presenting digital or virtual image information in a manner that is opaque to actual real-world visual input. AR scenarios typically involve presenting digital or virtual image information as an enhancement to the visualization of the real world around the user (i.e., transparency to real-world visual input). Thus, AR scenarios involve presenting digital or virtual image information in a manner that is transparent to real-world visual input.

[0003] MR systems can generate and display color data, which increases the realism of the MR scene. Many of these MR systems display color data by sequentially projecting sub-images of different (e.g., primary) colors or "fields" (e.g., red, green, and blue) corresponding to a color image in rapid succession. Projecting the color sub-images at a sufficiently high rate (e.g., 60 Hz, 120 Hz, etc.) can deliver a smooth color MR scene in the user's mind.

[0004] Various optical systems generate images, including color images, at various depths for displaying MR (VR and AR) scenes. Some such optical systems are described in U.S. Utility Patent Application No. 14 / 555,585 (Attorney Docket No. ML.20011.00), filed on November 27, 2014, the contents of which are expressly and completely incorporated herein by reference in their entirety, as if fully set forth.

[0005] MR systems can employ wearable display devices (e.g., head-mounted displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to the user's head and thereby move when the user's head moves. If the display system detects movement of the user's head, the data being displayed can be updated (e.g., "warped") to account for changes in head pose (i.e., the orientation and / or position of the user's head).

[0006] As an example, if a user wearing a head mounted display device views a virtual representation of a virtual object on a display and walks around the area where the virtual object appears, the virtual object may be rendered for each viewing point, giving the user the perception that they are walking around the object occupying real space. If the head mounted display device is used to present multiple virtual objects, measurements of head pose may be used to render the scene to match the user's dynamically changing head pose and provide increased immersion.

[0007] A head-mounted display device that enables AR provides simultaneous viewing of both real objects and virtual objects. With an "optical see-through" display, a user can see through a transparent (or translucent) element in the display system to directly view light from real objects in the environment. The transparent element (often called a "combiner") superimposes the light from the display on the user's view of the real world, where the light from the display projects an image of the virtual content on a see-through view of real objects in the environment. A camera can be mounted to the head-mounted display device to capture images or video of the scene viewed by the user.

[0008] Current optical systems, such as those in MR systems, render virtual content optically. The content is "virtual" because it does not correspond to a real physical object located at a corresponding position in space. Instead, the virtual content exists only in the brain (e.g., optical center) of the user of the head-mounted display device when the user is stimulated by a beam of light directed to their eyes.

[0009] In some cases, the head-mounted image display device may display virtual objects relative to the real environment and / or may allow the user to place and / or manipulate virtual objects relative to the real environment. In such a case, the image display device may be configured to position the user relative to the real environment so that the virtual object can be correctly shifted relative to the real environment. The head-mounted image display device may perform positioning using a positioning map. Specifically, when performing positioning, the image display device may obtain a real-time input image from a camera system of the image display device and match features in the input image with features in the positioning map.

[0010] Methods and apparatus for detecting corners in an image are described herein. The detected corners can be used as features for creating a positioning map. Alternatively, corners can be detected as features from an input image for matching with features of a positioning map for positioning of a user. Summary of the invention

[0011] A device configured to be worn on the head of a user comprises: a screen configured to present graphics to the user; a camera system configured to view an environment in which the user is located; and a processing unit coupled to the camera system, the processing unit being configured to: obtain a first image having a first resolution, the first image having a first corner, determine a second image having a second resolution, the second image having a second corner corresponding to the first corner in the first image, wherein the second image is based on the first image, the second resolution is smaller than the first resolution, detect the second corner in the second image, determine a position of the second corner in the second image, and determine a position of the first corner in the first image based at least in part on the determined position of the second corner in the second image.

[0012] Optionally, the processing unit is configured to detect a set of corners in the second image, the set of corners having a second corner.

[0013] Optionally, the processing unit is configured to detect a set of corners in the second image based on a Harris corner detection technique.

[0014] Optionally, the processing unit is configured to detect a set of corners in the second image based on one or more criteria.

[0015] Optionally, the one or more criteria include a first criterion for excluding one or more objects in the second image that are planar and / or have insufficient texture.

[0016] Optionally, the one or more criteria include a second criterion for excluding one or more objects having a linear configuration in the second image.

[0017] Optionally, the processing unit is further configured to perform non-maximum suppression to eliminate repeatedly detected corners in the corner set.

[0018] Optionally, non-maximum suppression includes 3×3 non-maximum suppression.

[0019] Optionally, the processing unit comprises hardware and software; wherein the hardware of the processing unit is configured to detect a set of corners; and wherein the software of the processing unit is configured to perform non-maximum suppression.

[0020] Optionally, the processing unit is configured to perform spatial binning to select a subset of corners from the set of corners.

[0021] Optionally, the processing unit is configured to perform spatial binning by: dividing the second image into a plurality of image portions having a first image portion; determining a score for each corner in a set of corners; and selecting one or more corners from the corners in the first image portion based on the scores of the corners in the first image portion and based on a specified maximum number of corners for the first image portion.

[0022] Optionally, the processing unit comprises hardware and software; wherein the hardware of the processing unit is configured to detect a set of corners; and wherein the software of the processing unit is configured to perform spatial binning.

[0023] Optionally, the processing unit is configured to detect the second corner in the second image by applying Harris corner detection to the second image.

[0024] Optionally, the processing unit is configured to detect the first corner in the first image by applying Harris corner detection to the first image.

[0025] Optionally, the processing unit is configured to detect a set of corners including the first corner in the first image.

[0026] Optionally, the processing unit is further configured to detect the first corner from the set of corners in the first image based at least in part on a position of the second corner in the second image.

[0027] Optionally, the processing unit comprises hardware configured to detect a set of corners in the first image, and software configured to detect the first corner from the set of corners in the first image.

[0028] Optionally, the processing unit is configured to determine the position of the first corner based on a relationship between a first resolution of the first image and a second resolution of the second image.

[0029] Optionally, the position of the second corner corresponds to a pixel position (xr, yr) in the second image; wherein the pixel position (xr, yr) in the second image corresponds to multiple pixel positions in the first image; and wherein the processing unit is configured to determine the position of the first corner by selecting a pixel position from the multiple pixel positions in the first image that has an optimal spatial relationship relative to the pixel position (xr, yr) in the second image.

[0030] Optionally, the processing unit comprises an image adjuster configured to determine the second image by converting the first image into the second image.

[0031] Optionally, the image adjuster is configured to reduce a first resolution of the first image to obtain a second image with a second resolution.

[0032] Optionally, the processing unit comprises hardware configured to detect a set of corners in a second image comprising a second corner, and software configured to determine a position of a first corner in the first image.

[0033] Optionally, the processing unit comprises hardware.

[0034] Optionally, the hardware of the processing unit is configured to detect a second corner in the second image.

[0035] Optionally, the hardware of the processing unit is configured to execute a Harris corner detection algorithm.

[0036] Optionally, the processing unit comprises a turning angle detector configured to operate based on a first criterion.

[0037] Optionally, the first criterion excludes objects in the second image that are planar and / or have insufficient texture.

[0038] Optionally, the first criterion includes: min >threshold, where λ min =min(λ2,λ1), where λ1 is the first eigenvalue and λ2 is the second eigenvalue greater than the first eigenvalue.

[0039] Optionally, the corner detector is further configured to operate based on a second criterion.

[0040] Optionally, the second criterion excludes objects having a linear configuration in the second image.

[0041] Optionally, the second criterion includes: λ2 / λ1<threshold, where λ1 is a first eigenvalue and λ2 is a second eigenvalue greater than the first eigenvalue.

[0042] Optionally, the processing unit is configured to detect a second rotation angle in the second image based on the first constant k1.

[0043] Alternatively, k1=R / (1+R)^2, where R is any number.

[0044] Optionally, the first constant k1 is between 0.1 and 0.2.

[0045] Optionally, the first constant k1 is 0.139.

[0046] Optionally, the processing unit is configured to detect a second rotation angle in the second image based on a second constant k2.

[0047] Optionally, the second constant k2 is less than 0.01.

[0048] Optionally, the second constant k2 is 0.00478125.

[0049] Optionally, the second constant k2 is based on the first constant k1.

[0050] Optionally, k2=(k1-1 / 16) / 16.

[0051] Optionally, the processing unit is configured to detect a second corner in the second image based on Det(H), where H is a matrix.

[0052] Optionally, the processing unit is configured to detect a second corner in the second image based on trace(H), where H is a matrix.

[0053] Optionally, the processing unit is configured to perform the first Harris angle detection based on a first constant and a second constant, wherein the second constant is different from the first constant.

[0054] Optionally, the second constant is based on the first constant.

[0055] Optionally, the processing unit is configured to perform non-maximum suppression.

[0056] Optionally, the processing unit is configured to perform spatial binning.

[0057] Optionally, the processing unit comprises software.

[0058] Optionally, the software of the processing unit is configured to perform non-maximum suppression.

[0059] Optionally, the software of the processing unit is configured to perform spatial binning.

[0060] Optionally, the first resolution comprises a VGA resolution.

[0061] Optionally, the second resolution comprises QVGA resolution.

[0062] Optionally, the apparatus further comprises a non-transitory medium configured to store the position of the first corner in the first image.

[0063] Optionally, the processing unit is configured to create a map based at least in part on the location of the first corner.

[0064] Optionally, the map comprises a map point associated with the location of the first corner.

[0065] Optionally, the processing unit is configured to perform positioning of the user based on the created map.

[0066] Optionally, the processing unit is configured to perform matching between the first image and the map based at least in part on the position of the first corner in the first image.

[0067] A method performed by a head-mounted image display device includes: obtaining a first image with a first resolution, the first image having a first corner; determining a second image with a second resolution, the second image having a second corner corresponding to the first corner in the first image, wherein the second image is based on the first image and the second resolution is less than the first resolution; detecting the second corner in the second image; determining a position of the second corner in the second image; and determining a position of the first corner in the first image based at least in part on the determined position of the second corner in the second image.

[0068] Optionally, the second corner is part of a set of corners detected in the second image.

[0069] Optionally, a set of corners is detected in the second image based on Harris corner detection technology.

[0070] Optionally, a set of corners in the second image is detected based on one or more criteria.

[0071] Optionally, the one or more criteria include a first criterion for excluding one or more objects in the second image that are planar and / or have insufficient texture.

[0072] Optionally, the one or more criteria include a second criterion for excluding one or more objects having a linear configuration in the second image.

[0073] Optionally, the method further includes performing non-maximum suppression to eliminate repeatedly detected corners in the corner set.

[0074] Optionally, non-maximum suppression includes 3×3 non-maximum suppression.

[0075] Optionally, the method is performed by a processing unit comprising hardware and software; wherein the set of corners is detected by hardware in the processing unit; and wherein non-maximum suppression is performed by software in the processing unit.

[0076] Optionally, the method further comprises performing spatial binning to select a subset of corners from the set of corners.

[0077] Optionally, the method further includes performing spatial binning by: dividing the second image into a plurality of image portions having a first image portion; determining a score for each corner in a set of corners; and selecting one or more corners from the corners in the first image portion based on the scores of the corners in the first image portion and based on a specified maximum number of corners for the first image portion.

[0078] Optionally, the method is performed by a processing unit comprising hardware and software; wherein the set of corners is detected by hardware in the processing unit; and wherein the spatial binning is performed by software in the processing unit.

[0079] Optionally, the second corner is detected in the second image by applying Harris corner detection to the second image.

[0080] Optionally, the method further comprises detecting a first corner in the first image by applying Harris corner detection to the first image.

[0081] Optionally, the method further comprises detecting, in the first image, a set of corners including the first corner.

[0082] Optionally, the method further comprises identifying a first corner from a set of corners in the first image based at least in part on a position of the second corner in the second image.

[0083] Optionally, the set of corners in the first image is detected by hardware, and the first corner is identified from the set of corners in the first image by software.

[0084] Optionally, the position of the first corner is determined based on a relationship between a first resolution of the first image and a second resolution of the second image.

[0085] Optionally, the position of the second corner corresponds to a pixel position (xr, yr) in the second image; wherein the pixel position (xr, yr) in the second image corresponds to multiple pixel positions in the first image; and wherein the position of the first corner is determined by selecting a pixel position from the multiple pixel positions in the first image that has an optimal spatial relationship relative to the pixel position (xr, yr) in the second image.

[0086] Optionally, the second image is determined by converting the first image into the second image.

[0087] Optionally, the first image is converted into the second image by reducing a first resolution of the first image to obtain a second image having a second resolution.

[0088] Optionally, the second corner is part of a set of corners detected in the second image, and wherein the position of the first corner in the first image is determined by software.

[0089] Optionally, the method is performed by a processing unit comprising hardware.

[0090] Optionally, the second corner is detected in the second image by hardware of the processing unit.

[0091] Optionally, the hardware of the processing unit is configured to execute a Harris corner detection algorithm.

[0092] Optionally, the method is performed by a processing unit comprising a corner detector, the corner detector being configured to operate based on a first criterion.

[0093] Optionally, the first criterion excludes objects in the second image that are planar and / or have insufficient texture.

[0094] Optionally, the first criterion includes: min >threshold, where λ min =min(λ2,λ1), where λ1 is the first eigenvalue and λ2 is the second eigenvalue greater than the first eigenvalue.

[0095] Optionally, the corner detector is further configured to operate based on a second criterion.

[0096] Optionally, the second criterion excludes objects having a linear configuration in the second image.

[0097] Optionally, the second criterion includes: λ2 / λ1<threshold, where λ1 is a first eigenvalue and λ2 is a second eigenvalue greater than the first eigenvalue.

[0098] Optionally, a second rotation angle is detected in the second image based on the first constant k1.

[0099] Alternatively, k1=R / (1+R)^2, where R is any number.

[0100] Optionally, the first constant k1 is between 0.1 and 0.2.

[0101] Optionally, the first constant k1 is 0.139.

[0102] Optionally, a second rotation angle is detected in the second image based on a second constant k2.

[0103] Optionally, the second constant k2 is less than 0.01.

[0104] Optionally, the second constant k2 is 0.00478125.

[0105] Optionally, the second constant k2 is based on the first constant k1.

[0106] Optionally, k2=(k1-1 / 16) / 16.

[0107] Optionally, a second corner is detected in the second image based on Det(H), where H is a matrix.

[0108] Optionally, a second corner is detected in the second image based on trace(H), where H is a matrix.

[0109] Optionally, the second angle is detected by performing the first Harris angle detection based on a first constant and a second constant, wherein the second constant is different from the first constant.

[0110] Optionally, the second constant is based on the first constant.

[0111] Optionally, the method further comprises performing non-maximum suppression.

[0112] Optionally, the method further comprises performing spatial binning.

[0113] Optionally, the method is performed by a processing unit comprising software.

[0114] Optionally, the method further comprises performing non-maximum suppression by software of the processing unit.

[0115] Optionally, the method further comprises performing spatial binning by software of the processing unit.

[0116] Optionally, the first resolution comprises a VGA resolution.

[0117] Optionally, the second resolution includes QVGA resolution

[0118] Optionally, the method further comprises storing the position of the first corner in a non-transitory medium.

[0119] Optionally, the method further comprises creating a map based at least in part on the location of the first corner.

[0120] Optionally, the map comprises a map point associated with the location of the first corner.

[0121] Optionally, the method further comprises performing positioning of a user of the image display device based on the created map.

[0122] Optionally, the method further comprises performing matching between the first image and a map based at least in part on a position of the first corner in the first image.

[0123] Additional and other objects, features, and advantages of the present disclosure are described in the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] The accompanying drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale and that elements having similar structures or functions are represented by the same reference numerals throughout all drawings. In order to better understand how to obtain the above and other advantages and objectives of various embodiments of the present disclosure, a more detailed description of the present disclosure briefly described above will be presented by reference to specific embodiments of the present disclosure. It should be understood that these drawings illustrate only typical embodiments of the present disclosure and, therefore, should not be considered limiting of its scope, and the present disclosure will be described and explained with the aid of additional features and details through the use of the accompanying drawings, in which:

[0125] Figure 1 Another image display system having an image display device according to some embodiments is shown.

[0126] Figure 2 Another image display system having an image display device according to other embodiments is shown.

[0127] Figure 3 Another image display system having an image display device according to other embodiments is shown.

[0128] Figure 4 Another image display system having an image display device according to other embodiments is shown.

[0129] Figure 5 An image display device that displays frames at multiple depth planes is shown.

[0130] Figure 6 Methods for determining a map for allowing an image display device to locate a user of the image display device and / or perform other functions are shown.

[0131] Figure 7 An example of an environment divided into a plurality of cells is shown.

[0132] Figure 8 A processing unit of an image display device is shown.

[0133] Figures 9A-9B Classification of image points using eigenvalues ​​and / or Harris scores is shown.

[0134] Fig.10 Shown for Figure 8 The signal flow of the processing unit.

[0135] Fig.11 Shown by Figure 8 Example of corner detection performed by the processing unit.

[0136] Fig.12 Shows Figure 8 Variants of the processing unit.

[0137] Fig.13 Shown by Figure 8 A method executed by a processing unit.

[0138] Fig.14 A dedicated processing system is shown in accordance with some embodiments. DETAILED DESCRIPTION

[0139] Various embodiments of the present disclosure are directed to methods, apparatuses, and articles of manufacture for providing input to a head mounted video imaging device. Other objects, features, and advantages of the present disclosure are described in the detailed description, drawings, and claims.

[0140] Various embodiments are described below with reference to the accompanying drawings. It should be noted that the drawings are not drawn to scale, and throughout the drawings, elements with similar structures or functions are represented by the same reference numerals. It should also be noted that the drawings are intended only to facilitate the description of the embodiments. They are not intended to serve as an exhaustive description of the invention or as a limitation on the scope of the invention. In addition, the illustrated embodiments do not need to have all the aspects or advantages shown. The aspects or advantages described in conjunction with a specific embodiment are not necessarily limited to that embodiment, and even if not shown as such or if not explicitly described as such, the above aspects or advantages may also be practiced in any other embodiment.

[0141] The following description relates to an exemplary VR, AR, and / or MR system in which the embodiments described herein may be practiced. However, it should be understood that the embodiments are also applicable to applications in other types of display systems (including other types of VR, AR, and / or MR systems), and thus the embodiments are not limited to only the exemplary examples disclosed herein.

[0142] Overview of Problems and Solutions

[0143] In some cases, in order to locate a user of a head-mounted image display device relative to the user's environment, a positioning map of the environment is obtained. Then, a real-time input image from a camera system of the image display device is matched with the positioning map to locate the user. For example, the corner features of the input image can be detected from the input image and matched with the corner features of the positioning map. In some embodiments, in order to obtain a set of corners as features from an image used for positioning, the image may first need to undergo corner detection to obtain an initial set of detected corners. The initial set of detected corners is then further processed, for example, by non-maximum suppression and spatial binning (spinning), etc., in order to obtain a final set of detected corners for positioning purposes. However, non-maximum suppression, spatial binning, or other types of processing may require significant computing resources and may take too long to complete. Therefore, it would be advantageous to have a novel method and apparatus for obtaining a set of corners for positioning purposes. In some embodiments, the first image is adjusted to obtain a second image with reduced resolution. The second image is processed to detect corners to perform non-maximum suppression and spatial binning to obtain a set of corners for positioning. The set of corners obtained based on the reduced-resolution second image may be refined based on the positions of the first image to obtain a final set of corners with improved positions for positioning purposes.

[0144] Figure 1-4Various components of the image display system 100 in various embodiments are shown. The image display system 100 includes an image display device 101 and an apparatus 200 for providing input to the image display device 101. The apparatus 200 will be described in more detail below. The image display device 101 can be any of a VR device, an AR device, an MR device, or other types of display devices. The image display device 101 includes a frame structure 102 worn by a terminal user 50, a display subsystem 110 carried by the frame structure 102 so that the display subsystem 110 is positioned in front of the eyes of the terminal user 50, and a speaker 106 carried by the frame structure 102 so that the speaker 106 is positioned adjacent to the ear canal of the terminal user 50 (optionally, another speaker (not shown) is positioned adjacent to another ear canal of the terminal user 50 to provide stereo / formable sound control). The display subsystem 110 is designed to present a light pattern that can be comfortably perceived as an enhancement of physical reality to the eyes of the terminal user 50 with a high level of image quality and three-dimensional perception, and can present two-dimensional content. The display subsystem 110 presents the sequence of frames at a high frequency that provides the perception of a single coherent scene.

[0145] In the illustrated embodiment, the display subsystem 110 employs an "optical see-through" display through which a user can directly view light from a real object via a transparent (or translucent) element. The transparent element, which is often referred to as a "combiner," superimposes light from the display on the user's view of the real world. To this end, the display subsystem 110 includes a partially transparent display. The display is positioned in the end user's 50 field of view between the end user's 50 eyes and the surrounding environment so that direct light from the surrounding environment is transmitted to the end user's 50 eyes through the display.

[0146] In the illustrated embodiment, the image projection assembly provides light to a partially transparent display, which is combined with direct light from the surrounding environment and transmitted from the display to the eyes of the user 50. The projection subsystem can be a fiber scanning based projection device, and the display can be a waveguide based display, wherein the scanned light from the projection subsystem is injected into the waveguide based display to produce, for example, an image at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focus planes, and / or image layers stacked at multiple viewing distances or focus planes to represent a volumetric 3D object. The layers in the light field can be stacked closely enough together to appear continuous to the human visual subsystem (i.e., one layer is within the cone of confusion of the adjacent layer). Additionally or alternatively, even if the layers are more sparsely stacked (i.e., one layer is outside the cone of confusion of the adjacent layer), image elements can be blended across two or more layers to increase the perceived continuity of the transition between layers in the light field. The display subsystem 110 can be monocular or binocular.

[0147] The image display device 101 may also include one or more sensors (not shown) mounted to the frame structure 102 for detecting the position and movement of the head 54 of the end user 50 and / or the eye position and inter-eye distance of the end user 50. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, and / or gyroscopes) or any combination of the foregoing. Many of these sensors operate based on the assumption that the frame 102 to which they are fixed is, in turn, substantially fixed to the user's head, eyes, and ears.

[0148] The image display device 101 may also include a user orientation detection module. The user orientation module detects the instantaneous position of the head 54 of the end user 50 (e.g., via a sensor coupled to the frame 102), and may predict the position of the head 54 of the end user 50 based on the position data received from the sensor. Detecting the instantaneous position of the head 54 of the end user 50 helps determine the specific real object that the end user 50 is looking at, thereby providing an indication of a specific virtual object to be generated with respect to the real object, and further providing an indication of the location where the virtual object is to be displayed. The user orientation module may also track the eyes of the end user 50 based on the tracking data received from the sensor.

[0149] The image display device 101 may also include a control subsystem that may take any of a variety of forms. The control subsystem includes a plurality of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs) (e.g., field PGAs (FPGAs)), and / or programmable logic controllers (PLUs).

[0150] The control subsystem of the image display device 101 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a three-dimensional database for storing three-dimensional scene data. The CPU may control the overall operation, while the GPU may render frames (i.e., convert the three-dimensional scene into a two-dimensional image) from the three-dimensional data stored in the three-dimensional database and store the frames in the frame buffer. One or more additional integrated circuits may control the reading and / or reading out of frames from the frame buffer and the operation of the image projection components of the control display subsystem 110.

[0151] The various processing components of the image display device 101 may be physically contained in a distributed subsystem. Figure 1-4 As shown, the image display device 101 includes a local processing and data module 130 that is operably coupled (e.g., via a wired lead or wireless connection 136) to the display subsystem 110 and the sensor. The local processing and data module 130 can be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 ( Figure 1 ), fixedly attached to a helmet or hat 56 ( Figure 2 ), removably attached to the torso 58 of the end user 50 ( Figure 3 ) or is configured to be removably attached to the hip 60 of the end user 50 in a belt coupling manner ( Figure 4 The image display device 101 also includes a remote processing module 132 and a remote data repository 134 operably coupled (e.g., via wired leads or wireless connections 138, 140) to the local processing and data module 130, such that these remote modules 132, 134 are operably coupled to each other and can serve as sources for the local processing and data module 130.

[0152] The local processing and data module 130 may include a power efficient processor or controller and digital memory such as flash memory, both of which may be used to help process, cache and store data captured from the sensor and / or data acquired and / or processed using the remote processing module 132 and / or remote data repository 134, which may be transferred to the display subsystem 110 after such processing or retrieval. The remote processing module 132 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. The remote data repository 134 may include a relatively large-scale digital data storage facility that is available through the Internet or other network configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 130, allowing for full autonomous use from any remote module.

[0153] The couplings 136, 138, 140 between the various components described above may include one or more wired interfaces or ports for providing wired or optical communications or one or more wireless interfaces or ports for providing wireless communications such as through RF, microwave, and IR. In some implementations, all communications may be wired, while in other implementations, all communications may be wireless. In further implementations, the choice of wired and wireless communications may be with Figure 1-4 Therefore, the specific choice of wired or wireless communication should not be considered limiting.

[0154] In some embodiments, the user orientation module is included in the local processing and data module 130, while the CPU and GPU are included in the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be included in the local processing and data module 130. The 3D database may be associated with the remote data repository 134 or may be provided locally.

[0155] Some image display systems (e.g., VR systems, AR systems, MR systems, etc.) use multiple volume phase holograms, surface relief holograms, or light-guiding optical elements embedded with depth plane information to generate images that appear to come from various depth planes. In other words, a diffraction pattern or diffractive optical element ("DOE") can be embedded in a light-guiding optical element ("LOE"; for example, a planar waveguide) or embossed / engraved on the LOE so that collimated light (a light beam with a substantially planar wavefront) is substantially totally internally reflected along the LOE, intersecting the diffraction pattern at multiple locations and exiting toward the user's eyes. The DOE is configured so that light emitted from the LOE through it converges so that they appear to originate from a specific depth plane. Collimated light can be generated using an optical focusing lens ("condenser").

[0156] For example, a first LOE can be configured to transmit collimated light to the eye that appears to originate from an optically infinite depth plane (0 diopters). Another LOE can be configured to transmit collimated light that appears to originate from a distance of 2 meters (1 / 2 diopters). Yet another LOE can be configured to transmit collimated light that appears to originate from a distance of 1 meter (1 diopters). By using stacked LOE assemblies, it will be appreciated that multiple depth planes can be formed, wherein each LOE is configured to display an image that appears to originate from a particular depth plane. It should be appreciated that the stack can include any number of LOEs. However, at least N stacked LOEs are required to generate the N depth planes. In addition, RGB color images can be generated on N depth planes using N, 2N, or 3N stacked LOEs.

[0157] In order to present 3D virtual content to the user, the image display system 100 (e.g., VR system, AR system, MR system, etc.) projects images of the virtual content into the user's eyes so that the images appear to originate from various depth planes in the Z direction (i.e., orthogonally away from the user's eyes). In other words, the virtual content can not only change in the X and Y directions (i.e., in a 2D plane orthogonal to the central visual axis of the user's eyes), but also appear to change in the Z direction, so that the user can perceive that the object is very close or at an infinite distance or at any distance between them. In other embodiments, the user can perceive multiple objects at different depth planes at the same time. For example, the user can see a virtual dragon that appears to be running towards the user from an infinite distance. Alternatively, the user can simultaneously see a virtual bird at a distance of 3 meters from the user and a virtual coffee cup at an arm's length (about 1 meter) from the user.

[0158] A multi-plane focusing system creates a variable depth perception by projecting images on some or all of multiple depth planes at various fixed distances from the user's eyes in the Z direction. Figure 5 It should be understood that the multi-plane focusing system can be used at a fixed depth plane 150 (e.g., Figure 5 Frames are displayed at six depth planes 150 as shown. Although the MR system may include any number of depth planes 150, an exemplary multi-plane focusing system has six fixed depth planes 150 in the Z direction. When virtual content is generated at one or more of the six depth planes 150, a 3D perception is formed so that the user perceives one or more virtual objects at different distances from the user's eyes. Assuming that the human eye is more sensitive to objects that are closer than objects that appear to be farther away, more depth planes 150 that are closer to the eye are generated, such as Figure 5 In other embodiments, the depth planes 150 may be placed at equal distances from each other.

[0159] The depth plane position 150 may be measured in diopters, which is a unit of optical power that is equal to the reciprocal of the focal length measured in meters. For example, in some embodiments, depth plane 1 may be 1 / 3 diopters away, depth plane 2 may be 0.3 diopters away, depth plane 3 may be 0.2 diopters away, depth plane 4 may be 0.15 diopters away, depth plane 5 may be 0.1 diopters away, and depth plane 6 may represent infinity (i.e., 0 diopters away). It should be understood that other embodiments may generate depth planes 150 at other distances / diopters. Thus, when virtual content is generated at strategically placed depth planes 150, the user is able to perceive three-dimensional virtual objects. For example, the user may perceive the first virtual object displayed in depth plane 1 as being close to him, while another virtual object appears to be at depth plane 6 at infinity. Alternatively, the virtual object may be first displayed at depth plane 6, then displayed at depth plane 5, and so on, until the virtual object appears to be very close to the user. It should be understood that the above examples are significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be focused at a specific focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup half a meter away from the user, all six depth planes may be generated at various cross-sections of the coffee cup, giving the user a highly granular 3D view of the coffee cup.

[0160] In some embodiments, the image display system 100 (e.g., a VR system, an AR system, an MR system, etc.) can be used as a multi-plane focusing system. In other words, all six LOEs can be illuminated simultaneously, thereby generating images that appear to originate from six fixed depth planes in rapid succession, where the light source quickly delivers image information to LOE 1, then LOE 2, then LOE 3, and so on. For example, a portion of the desired image can be injected at time 1, which includes an image of the sky at optical infinity, and the LOEs that maintain light collimation (e.g., Figure 5 Then, an image of a closer branch can be injected at time 2, and a depth plane configured to appear to originate from 10 meters away (e.g., Figure 5 Then, the pen image can be injected at time 3, and the LOE configured to generate an image that appears to originate from a depth plane 1 meter away can be utilized. Such examples can be repeated in extremely fast timing (e.g., at 360 Hz) so that the user's eyes and brain (e.g., visual cortex) perceive the inputs as all parts of the same image.

[0161] The image display system 100 can project images (i.e., by diverging or converging light beams) that appear to originate from various locations along the Z axis (i.e., depth planes) to generate images for a 3D experience / scene. As used in this application, a "light beam" includes, but is not limited to, a directional projection of light energy (including visible and invisible light energy) radiated from a light source. Generating images that appear to originate from various depth planes complies with the vergence and accommodation of the user's eyes for the image, and minimizes or eliminates the vergence-accommodation conflict.

[0162] Location Map

[0163] Figure 6 A method for determining a map is shown, which is used to allow an image display device 101 to locate a user 50 of the image display device 101. As shown, when the user 50 is using the image display device 101, the user 50 can move the image display device 101 to achieve different viewing positions and / or directions. For example, the user 50 can turn his / her head, turn his / her body and / or walk to different positions. In some embodiments, the image display device 101 includes a camera facing forward. Therefore, by moving the image display device 101, the field of view of the camera facing forward of the image display device 101 will change accordingly. When the user 50 is in different postures, the camera of the image display device 101 generates corresponding images. In the example shown, the user 50 achieves three different postures by turning his / her head, and the camera facing forward of the image display device 101 generates three images 200a-200c corresponding to the three postures. Each of the images 200a-200c captures certain objects 202 in the environment. For example, image 200b captures objects 202a-202d, and image 200c captures objects 202b-202e. Depending on the gestures implemented by the user 50, a certain object in the environment may be captured in multiple images 202 of the camera, and a certain other object may be captured in only one image 200. In some embodiments, the processing unit 130 of the image display device 101 is configured to obtain the image 200 from the camera of the image display device 101, and perform image processing to extract features from the image 200 to create a map 220. The map 200 may be stored in a non-transitory medium of the image display device 101, and may be used by the processing unit 130 to perform positioning of the user 50. Thus, the map 220 serves as a positioning map. In the illustrated embodiment, the map 220 is a three-dimensional representation of the environment detected by different gestures of the user 50.

[0164] In some embodiments, the environment around the user 50 may be divided into a plurality of cells. In this case, the above-described map creation techniques may be used for different cells of the environment. Figure 7An example of an environment divided into a plurality of cells 300 is shown. Each cell 300 is a defined three-dimensional space representing a portion of the environment. Each cell 300 may have a predetermined size and shape. For example, each cell 300 may have a coverage area of ​​2m×2m and a height of 2m. In other embodiments, each cell 300 may have other coverage dimensions and / or other heights. Furthermore, in other embodiments, each cell 300 may not have the rectangular configuration shown and may have other shapes. In the illustrated embodiment, the cells 300 all have the same shape and size. In other embodiments, at least two of the cells 300 may have different corresponding sizes and / or shapes.

[0165] In some embodiments, the user 50 of the image display device 101 may enter different locations in the environment corresponding to different cells 300, and may use the camera of the image display device 101 to scan the space in the corresponding cell to obtain different maps of the corresponding cell of the environment. The map may be stored in a non-transitory medium of the image display device 101 to allow the processing unit 130 of the image display device 101 to perform positioning of the user 50.

[0166] During the use of a map to locate the user 50, the camera of the image display device 101 obtains an image of the environment based on the specific position and orientation of the user 50. Such camera images are used as tracking images (tracking maps) to allow the processing unit 130 of the image display device 101 to track the position of the user 50. Specifically, the processing unit 130 of the image display device 101 processes the image from the camera to determine whether the features in the image match certain features in the map 220. If a match is found, the processing unit 130 can determine the position and orientation of the user 50 based on the matched features. In some embodiments, the map 220 may contain less information (e.g., features) than the tracking image provided by the camera of the image display device 101. This is advantageous because it allows the processing unit 130 to effectively match the tracking image with the map 220. In addition, in some embodiments, the map 220 may be referred to as a "canonical map". When performing positioning, the processing unit 130 performs feature extraction on additional features from the camera image (tracking image) and matches the features with features in the canonical map. In one embodiment, processing unit 130 is configured to find a six degrees of freedom transformation between the tracking image and the standard map to locate user 50. Once user 50 can use the map to successfully locate relative to his / her environment, processing unit 130 can then allow the user to use the map to place virtual content relative to the environment, retrieve virtual content from a previous session, share virtual content with other users, etc.

[0167] During the use of the image display device 101, the processing unit 130 may need to perform corner detection in the images provided by the camera system of the image display device 101. For example, in some embodiments, when the camera images are used to determine the map 220, the processing unit 130 may need to detect corners in these images. The processing unit 130 can use the detected corners as features for constructing the map 220. In addition, when the map 220 is used to perform positioning, the processing unit 130 may need to match the features in the real-time input image with the features of the map 220. Therefore, the processing unit 130 can be configured to detect corners in the real-time input image. Corners are generally more stable features in the sense that corners can usually be detected from different viewing directions. Therefore, the detectability of corners in the image may not be significantly affected by changes in the viewing point. Therefore, corners are good features for matching between the input image and the map 220. Corners are also good features for use in stereo matching between images generated at different times and in different viewing directions.

[0168] Corner detection

[0169] Figure 8 The processing unit 130 is shown, specifically showing the components for implementing corner detection. As shown, the processing unit 130 includes hardware 410 and software 420. The processing unit 130 also includes a gamma corrector 412, an image adjuster 414, a corner detector 416, a non-maximum suppressor 422, a spatial binning module 424, and a corner position determiner 426. In the illustrated embodiment, the gamma corrector 412, the image adjuster 414, and the corner detector 416 are implemented using the hardware 410 of the processing unit 130. In addition, in the illustrated embodiment, the software 420 of the processing unit 130 is implemented using the non-maximum suppressor 422, the spatial binning module 424, and the corner position determiner 426.

[0170] In other embodiments, the gamma corrector 412, the image adjuster 414, the corner detector 416, or any combination thereof may be implemented using software 420 of the processing unit 130. In addition, in other embodiments, the non-maximum suppressor 422, the spatial binning module 424, the corner position determiner 426, or any combination thereof may be implemented using hardware 410 of the processing unit 130.

[0171] Gamma corrector 412 is configured to adjust the non-linear relationship between pixel value and brightness. For example, in some embodiments, gamma corrector 412 may be configured to adjust the relationship between pixel value and brightness so that the relationship is linear or more linear than the relationship in the original unprocessed image.

[0172] The image adjuster 414 is configured to obtain an image and reduce the resolution of the image to provide an adjusted image. In some embodiments, the image adjuster 414 is configured to obtain a first image having a first resolution and reduce the first image to obtain a second image having a second resolution less than the first resolution. For example, in some cases, the first image may have a VGA resolution and the second image may have a QVGA resolution. In this case, the second image has a plurality of pixels that is one quarter of the pixels used for the first image. In other examples, the second image may have a plurality of pixels that is other fractions of the pixels used for the first image.

[0173] Corner detector 416 is configured to detect corners in an image based on one or more criteria. In some embodiments, corner detector 426 is configured to detect corners in an image using Harris corner detection technology. Harris corner detection is implemented by calculating the gradient of each pixel. If the absolute gradient value is large (e.g., above a certain threshold) in two directions (e.g., orthogonal directions), then corner detector 416 can determine that the pixel is a corner. In one implementation of the Harris corner detection technology, the following algorithm is used to calculate the score for each pixel in the image: (1) Calculate the x and y derivatives of the image

[0174]

[0175] Among them G x and G y is the first-order directional difference. In some cases, G can be calculated by convolving the grayscale value and the difference operator in the direction x, y x and G y .

[0176] (2) Calculate the product of the derivatives at each pixel

[0177] I x2 =I x .I x I y2 =I y .I y I xy =I x .I y

[0178] (3) Calculate the sum of the products of the derivatives at each pixel

[0179] S x2 =G σ′ *I x2 S y2 =G σ′ *I y2 Sxy =G σ′ *I xy

[0180] (4) Define the matrix at each pixel (x, y)

[0181]

[0182] Alternatively, the matrix H can be expressed as:

[0183]

[0184] Among them (I x ,I y ) is the gradient at (x,y).

[0185] (5) Calculate the score for each pixel (Harris response)

[0186] R=Det(H)–k(Trace(H))^2

[0187] Where Det(H)=λ1λ2, Trace(H)=λ1+λ2, λ1 and λ2 are the eigenvalues ​​of H.

[0188] In some embodiments, the Harris response R and / or the eigenvalues ​​λ1 , λ2 may be used by the corner detector 416 in one or more criteria for corner detection. Fig. 9A Classification of image points using the response R and / or eigenvalues ​​according to the Harris scoring function is shown. Fig. 9B The classification of image points using eigenvalues ​​according to the Shi-Tomasi criterion is shown. Figures 9A-9B As shown, if both eigenvalues ​​λ1 and λ2 are small, the pixel may be part of a flat region. Therefore, the corner detector 416 may be configured to detect a corner based at least in part on the criterion that min(λ1,λ2)>threshold. As another example, Figures 9A to 9B As shown in , if the larger eigenvalue λ2 is larger than the smaller eigenvalue λ1 by a threshold value, the pixel may be part of an object having a linear configuration (e.g., a line, an edge, etc.). Therefore, the corner detector 416 may be configured to detect a corner based at least in part on the criterion of λ2 / λ1<threshold value. In some embodiments, the corner detector 416 may be configured to detect a corner based on two criteria:

[0189] (1)min(λ1,λ2)>threshold T1

[0190] (2)λ2 / λ1<Threshold T2

[0191] In some embodiments, threshold T1 may be set to 300. In other embodiments, threshold T1 may have other values ​​other than 300. In some embodiments, threshold T2 may be set to 5. In other embodiments, threshold T2 may have other values ​​other than 5.

[0192] In other embodiments, the above two criteria may be expressed as Det(H) and Trace(H), as follows:

[0193] Trace(H)>2T1 and Det(H)–T1*Trace(H)+T1^2>0

[0194] Det(H)–T2 / ((1+T2)^2)*Trace(H)^2>0

[0195] In this case, the corner detector 416 can be configured to calculate Det and Trace using two Harris responses with different k (k1, k2) as follows:

[0196] (1) Harris response based on k1>0, where when T2=5, k1=T2 / ((1+T2)^2)=0.139

[0197] (2) Trace^2 = (Harris response based on k2 - Harris response based on k1) / (k1 - k2) > 4T1^2

[0198] In some embodiments, k2 can be selected in a manner that simplifies the calculation. For example, the above equation (2) can be rewritten as:

[0199] (2) (Harris response based on k2 - Harris response based on k1) > (4T1^2)*(k1-k2)

[0200] In this case, k2 can be selected so that the term on the right side ((4T1^2)*(k1-k2)) becomes 1, and the evaluation can be simplified based on the expression on the left side being greater than 1. In some embodiments, k2 can be calculated based on the following formula:

[0201] K2=(k1–1 / 16) / 16

[0202] Therefore, if k1 is set to 0.139, k2 is then equal to 0.00478125 based on the above equation.

[0203] It should be noted that k1 and k2 are not limited to having the above values, and may have other values ​​in other embodiments. For example, in other embodiments, k1 may have a value of any value from 0.1 to 0.2. In addition, for example, in other embodiments, k2 may have a value less than 0.001.

[0204] Furthermore, it should be noted that as used in this specification, the term "corner" is not limited to the intersection point between two straight lines, and may also refer to any point along a line of a curve connecting two lines (e.g., straight lines), where the curve line may have a radius of curvature (such that the smaller the radius of curvature, the sharper the "corner").

[0205] In some embodiments, corner detector 416 may be configured to apply Harris corner detection to the first image one or more times to detect a set of corners in the first image based on one or more criteria. Corner detector 416 may also be configured to apply Harris corner detection to the second image one or more times to detect a set of corners in the second image based on one or more criteria. One or more criteria may be selected so that only corners having certain desired characteristics are detected.

[0206] Furthermore, it should be noted that the corner detector 416 is not limited to detecting the corner based on the example of the above formula and value, and the corner detector 416 can be configured to detect the corner based on other formulas that can be derived, varied and / or modified from the above formula. Additionally, in other embodiments, the corner detector 416 can be configured to detect the corner based on other corner detection techniques different from the Harris corner detection technique.

[0207] The non-maximum suppressor 422 is configured to perform non-maximum suppression on the detected corners, thereby preventing repeated detection of the same corner. In some cases, due to the image resolution of the image and / or the way in which corner detection is implemented, the same corner may be detected multiple times. For example, when in fact all three detected corners are for the same corner in the image, the corner detector 416 may detect the same corner three times at three positions close to each other. Therefore, as used in this specification, the term "corner" may refer to an instance of a corner or a corner that is detected multiple times. In some embodiments, the non-maximum suppressor 422 may be configured to perform 3×3 non-maximum suppression. In this case, if multiple corners are detected within a 3×3 pixel area, the non-maximum suppressor 422 will select one of the multiple detected corners in the area as the detected corner. In one implementation, the non-maximum suppressor 422 may be configured to determine the corner scores of the corresponding multiple detected corners within the area, and select the corner with the highest score as the detected corner of the area. The score may indicate the probability that the detected corner is located at the detected position or the accuracy of the position of the detected corner.

[0208] The spatial binning module 424 is configured to select certain of the detected corners for a given image so that different regions of the image will have a somewhat uniform distribution of the detected corners. Specifically, after the non-maximum suppressor 422 removes duplicate detected corners, there may still be a large number of detected corners for a given image, and / or the remaining detected corners may have different distributions in different regions of the image. The spatial binning module 424 is configured to select a subset of the detected corners so that different regions of the image will have a specific maximum number of detected corners that meet certain criteria. For example, in some embodiments, the second image may be divided into four regions. The four regions may have different corresponding numbers of detected corners and corresponding scores, as follows:

[0209] Region 1: (C1, 0.7), (C2, 0.8) (C3, 0.85) (C4, 0.9)

[0210] Region 2: (C5, 0.6), (C6, 0.65)

[0211] Region 3: (C7, 0.66), (C8, 0.82)

[0212] Region 4: (C9, 0.9), (C10, 0.88), (C11, 0.63)

[0213] In the above example, region 1 has 3 detected corners C1-C4, region 2 has 2 detected corners C5-C6, region 3 has 2 detected corners C7-C8, and region 4 has 3 detected corners C9-C11. If the maximum number of corners expected per region is selected to be 3, and if the criterion for selecting corners is that a corner must have a score of 0.7 or higher, then the spatial binning module 424 may select the following corners for different corresponding regions 1-4:

[0214] Region 1: C2, C3, C4 (note that corner C1 has the lowest score in region 1, and therefore the three corners with the highest scores in region 1 are selected; the spatial binning module 424 cannot select all 4 corners C1-C4 because the maximum number of corners in the example is 3).

[0215] Region 2: No corners are selected (because corners C5 and C6 both have scores that do not satisfy the minimum score criterion).

[0216] Region 3: Only corner C8 is selected (because corner C7 has a score that does not meet the minimum score criterion).

[0217] Region 4: Only corners C9 and C10 are selected (because corner C11 has a score that does not satisfy the minimum score criterion).

[0218] Corner position determiner 426 is configured to determine the final position of the corresponding detected corner. Because corner detector 416 detects corners based on an image with reduced resolution, it may be desirable to refine the positions of these detected corners based on an image with a higher resolution. As discussed, in some embodiments, image adjuster 414 is configured to convert a first image with a first resolution into a second image with a second resolution. In addition, as discussed, in some embodiments, the second resolution of the second image used to detect the corner is one-quarter of the first resolution of the first image. Following this example, for each pixel in the second image that has been identified as the position of the detected corner, there are four corresponding pixels in the first image corresponding to the detected corner. Therefore, corner position determiner 426 may be configured to select one of the four pixels in the first image (which corresponds to the pixel of the detected corner in the second image) as the detected corner.

[0219] In one implementation, the corner position determiner 426 is configured to select a pixel (from four pixels in the first image) based on one or more corners detected in the first image. For example, the corner detector 416 may detect that there are two corners in the first image that are in close proximity to the coordinates (xr, yr) of the corners in the second image. For example, the coordinates of the two corners in the first image may be determined by the corner position determiner 426 as (x1, y1), (x2, y2). Next, the corner position determiner 426 then calculates the distance between the corresponding pixel coordinates of the corners in the first image and the pixel coordinates of the corners in the second image as follows:

[0220] D1=((x1-xr)^2+(y1-yr)^2)^1 / 2

[0221] D2=((x2-xr)^2+(y2-yr)^2)^1 / 2

[0222] The corner position determiner 426 then selects the coordinates of the pixel having the lowest corresponding distance D (ie, min(D1, D2)) in the first image as the position of the detected corner.

[0223] In other embodiments, the corner position determiner 426 may use other techniques to refine the positions of the detected corners (detected from the second image). For example, in other embodiments, the positions of the set of corners output by the spatial binning module 424 (which are selected by the non-maximum suppressor 422 and the spatial binning module 424 as a subset of the detected corners in the second image) may be input to the corner detector 416. The corner detector 416 then executes a corner detection algorithm to detect corners at discrete positions in the first image corresponding to the positions of the set of corners from the second image. In one implementation, the corner detector 416 may execute a Harris corner detection algorithm to detect a set of corners at positions in the first image corresponding to the positions of those detected corners in the second image. This technique is advantageous because Harris corner detection does not need to be performed on the entire first image, but only on discrete portions of the first image, thereby saving time and computing resources.

[0224] In some embodiments, the gamma corrector 412, the image adjuster 414, and the corner detector 416 may be implemented using dedicated hardware specifically designed to perform the functions of these components. By way of non-limiting example, the hardware may include one or more FPGA processors, one or more ASIC processors, one or more signal processors, one or more mathematical processors, one or more processors specifically designed to perform matrix operations, or any combination of the foregoing. In addition, in some embodiments, the processing unit 130 may be implemented as separate components that are communicatively coupled together. For example, the processing unit 130 may have a first substrate that carries the gamma corrector 412 and the image adjuster 414 and another substrate that carries the corner detector 416. As another example, the processing unit 130 may have a processor for executing the software 420, wherein the processor may be implemented on the same substrate that supports the corner detector 416, or on a substrate different from the substrate for the corner detector 416. In other embodiments, there may be separate substrates that carry the gamma corrector 412, the image adjuster 414, the corner detector 416, and the processor that runs the software 420, respectively. Additionally, in some embodiments, any, some, or all of the components of the processing unit 130 may be implemented at the head-mounted frame structure 102. In other embodiments, any, some, or all of the components of the processing unit 130 may be implemented at a device remote from the head-mounted frame structure 102, such as at a band-clip module, a neck-worn module, a cellular phone, etc.

[0225] Fig.10 Shows Figure 8 130. Fig.11 The signal flow is described using the graphical examples shown in Figure 1. Fig.10 As shown, the processing unit 130 receives an image from the camera system of the image display device 101. The image may be a real-time image or an image with a time lag. In other embodiments, the image may be received from a non-transitory medium storing the image. The non-transitory medium may be a part of the image display device 101 or an external device, such as another image display device, a server, a cellular phone, a media device, etc. The image is initially processed by the hardware 410 of the processing unit 130. Specifically, the gamma corrector 412 of the processing unit 130 adjusts the nonlinear relationship between the pixel value and the brightness of the pixel in the image. For example, the gamma corrector 412 may adjust the relationship between the pixel value and the brightness so that the relationship is linear or more linear than the relationship in the original unprocessed image. After adjusting the image for gamma correction, the gamma corrector then passes the image 502 to the image adjuster 414.

[0226] The image adjuster 414 is configured to obtain an image (first image) 502 having a first resolution, and adjust the image 502 to obtain a second image 504 having a second resolution lower than the first resolution. For example, in some embodiments, the first image 502 may have a VGA resolution, and the second image 504 may have a QVGA resolution. In this case, the second image 504 has one-fourth the number of pixels in the first image 502. In other embodiments, the first image 502 may have a resolution different from the VGA resolution, and the second image 504 may have a resolution different from the QVGA resolution. After obtaining the second image 504, the image adjuster 414 then passes the second image 504 to the corner detector 416. Fig.11 As shown in the example of , second image 504 includes an image of object 600 and some corners (eg, corner 602 ).

[0227] Corner detector 416 detects corners in second image 504 based on one or more criteria. In some embodiments, corner detector 416 executes the Harris corner detection algorithm one or more times based on the one or more criteria to detect corner set 610 in second image 504. In other embodiments, corner detector 416 may use other techniques and algorithms to perform corner detection. Criteria may be selected so that only corners that meet certain desired characteristics are detected. Fig.11 As shown, the detected corner set includes detected corners 610a-610g. After the corner detector 416 detects the corners 610a-610g in the second image 504, the corner detector 416 then passes the corner detection result 506 to the non-maximum suppressor 422. The corner detection result 506 may include identifiers of the detected corner set 610a-610g and their corresponding coordinates. In some embodiments, the corner detection result 506 may also optionally include a score for the corresponding detected corner 610a-610g. The score may indicate the quality of the detected corner, the confidence level of the detected corner, the accuracy of the detected corner, etc.

[0228] like Fig.11As shown in the example of , the corner detector 416 detects a set of corners 610a-610g in the second image 504. However, some of the detected corners 610 may be different instances of detection for the same corner (e.g., corner 306) in the second image 504. For example, the corner detector 416 may detect two corners 610a, 610b that are close to each other, when in fact they are the same corner 602 in the second image 504. Similarly, the corner detector 416 may detect two corners 610d, 610e that are close to each other, when in fact they are the same corner in the second image 504. Therefore, the non-maximum suppressor 422 is used to perform non-maximum suppression on the detected corners 610, thereby preventing repeated detection of the same corner. In some embodiments, the non-maximum suppressor 422 utilizes a 3×3 window to ensure that there are no repeated detected corners within a grid of 3×3 pixels. In other embodiments, the non-maximum suppressor 422 may utilize other window sizes to perform non-maximum suppression. After non-maximum suppressor 422 removes the repeatedly detected corners, non-maximum suppressor 422 then passes result 508 to spatial binning module 4246. In some embodiments, result 508 is a subset of result 506 output by corner detector 416. Fig.11 As shown in the example of , the result 508 from the non-maximum suppressor 422 includes identifiers of corners 610b, 610c, and 610e-610g, while corners 610a and 610d are excluded as duplicate detected corners. The result 508 may include identifiers of a subset of the set of corners detected by the corner detector 416 and their corresponding coordinates. The result 508 from the non-maximum suppressor 422 may also optionally include a score for the corresponding detected corner. The score may indicate the quality of the detected corner, the confidence level of the detected corner, the accuracy of the detected corner, etc.

[0229] Spatial binning module 424 selects certain of the corners of image 504 so that different areas of image 504 will have a somewhat uniform distribution of detected corners. Fig.11Following the above example, the spatial binning module 424 receives five identified corners 610b, 610c and 610e-610g from the non-maximum suppressor 422b. The spatial binning module 424 then divides the image 504 into a specified number of regions. In the example shown, the spatial binning module 424 divides the image 504 into four regions 620a-620d. The first region 620a has two detected corners 610b, 610c, the second region 620b has zero detected corners, the third region 620c has three detected corners 610e-610g, and the fourth region 620d has zero detected corners. In the example shown, the spatial binning module 424 is configured to select a subset of corners (determined by the non-maximum suppressor 422) in order to improve the distribution of the detected corners. For example, the spatial binning module 424 can be configured to select a subset of corners based on the maximum number of corners expected for each region and / or the minimum required score for each corner. If the maximum number of corners expected in this example is 2, and if the minimum required score for each corner is set to 0.7, the spatial binning module 424 can eliminate or exclude one (some) of the detected corners based on these criteria. In the example shown, assuming that the corners 610b, 610c detected in the first region 620a both have scores higher than 0.7, the spatial binning module 424 will select both of these corners 610b, 610c in its output. In the second region 620b, there are no detected corners, and therefore the number of corners of the second region 620b is zero in the output of the spatial binning module 4246. In the third region 620c, assuming that corner 610g has a score below 0.7, and corners 601e, 610f have scores above 0.7, the spatial binning module 424 accordingly selects corners 610e, 610f to include in its output. Based on the foregoing, the spatial binning module 424 provides a subset of corners (e.g., Fig.11 ) as its output 510. In some embodiments, the output 510 of the spatial binning module 424 may include an identifier of a subset of corners and the positions of the corresponding corners in the subset. The output 510 of the spatial binning module 424 may also optionally include scores for the corresponding corners, as similarly discussed. Therefore, as can be seen from the illustrated example, the spatial binning module 424 is advantageous because it ensures that the detected corners are not concentrated in a certain portion of the image and prevents excessive detected corners for a certain portion of the image.

[0230] As shown in the example, the detection of corners by the corner detector 416, the non-maximum suppression by the non-maximum suppressor 422, and the spatial binning by the spatial binning module 424 are all performed based on the second image 504 having a reduced resolution. Therefore, it may be desirable to refine the position of the detected corner based on the resolution of the first image 502 so as to make the position of the detected corner in the result 510 output by the spatial binning module 424 more accurate. Fig.10 As shown, the corner position determiner 426 is configured to refine the position of the corner (selected by the spatial binning module 424) based on the output 522 provided by the corner detector 416. Specifically, as shown, the first image 502 with a first resolution is also processed by the corner detector 416 to obtain a set of corners. In some embodiments, the first image 502 can be processed by the corner detector 416 based on the Harris corner detection technology. For example, one or more parameters of the corner detector 416 used to detect the corner in the second image 504 can be used by the corner detector 416 to detect the corner in the first image 502. In one implementation, the corner detector 416 uses a constant k3 to detect the corner in the first image 502, where k3 can be equal to k1, or can have other values. In other embodiments, the corner detector 416 can use different parameters and / or criteria to detect the corner in the first image 502. In another embodiment, the corner detector 416 can use different corner detection techniques to detect the corner in the first image 502. Corner position determiner 426 is configured to refine the positions of the detected corners in the set (result 510) based on the detected corners in first image 502 in order to determine final positions of the detected corners for output 5302. In some embodiments, corner position determiner 426 is configured to determine which corner detected based on first image 502 (in output 522) has the best spatial correspondence with each of the corners in the set output by spatial binning module 424 (i.e., in output 510). For example, to improve the position of detected corner 610b, if corner 610b in output 510 (based on second image 504) has a position (x=83, y=97), and if the corner in output 522 (based on first image 502) has a position (x=84, y=95) that is closest to the position of corner 610b, corner position determiner 426 will use position (x=84, y=95) as the final position of detected corner 610b. In some embodiments, corner position determiner 426 may be configured to only examine corners within discrete region 630 of first image 502 to identify a corner corresponding to a corner in output 510. Fig.11As shown, the area 630 may be a small window 630 surrounding the position of the detected corner in the output 510. In the example shown, there are two corners 632a, 632b in the window 630, which are detected in the first image 502b. The corner position determiner 426 determines which of the two corners 632a, 632b is closest in position relative to the corner 610b in the output 510. In one implementation, the corner position determiner 426 may be configured to determine the respective distances between the positions of the corners 632a, 632b in the window 630 and the position of the corner 610b, and select the corner position of the corner 632 that provides the shortest distance from the position of the corner 610b. The above process is repeated for other corners in the output 510 to obtain a refined set of positions for the corners 610 in the output 510. As shown in FIG. Fig.11 As shown in the example of , the output 530 from the corner position determiner 426 includes the set of detected corners from the spatial binning module 424, but with improved positions for the corners (610b', 610c', 610e', and 610f').

[0231] As shown in the example, the detection of corners by the corner detector 416, the non-maximum suppression by the non-maximum suppressor 422, and the spatial binning by the spatial binning module 424 are all performed based on the second image 504 with a reduced resolution. This technique is advantageous because it allows these operations to be performed very quickly without significant computing resources. In the example where the first image 502 has a VGA resolution and the second image 504 has a QVGA resolution, in all three stages processed by the corner detector 416, the non-maximum suppressor 422, and the spatial binning module 424, the processing unit 130 needs to use the second image 504 to process only a quarter of the pixels (compared to the scene using the first image 502). In some embodiments, such a technique allows corner features to be extracted very quickly from the image. For example, using the above technique, a set of corners (after non-maximum suppression and spatial binning) can be obtained from the image in less than 10ms, more preferably less than 6ms (such as 5ms or less). This is advantageous because it allows real-time processing of input camera images.

[0232] The above technique is also advantageous because, although corner detection may be performed faster (compared to techniques that process the entire image at the original resolution to detect the corner), the corner detection is as accurate as techniques that process the entire image at the original resolution. This is because only a subset of the first image (having a higher resolution) is utilized to confirm the position of the corner detected using the second image having a reduced resolution. Therefore, the resulting detected corner has the same accuracy as techniques that process the entire first image to detect the corner.

[0233] In other embodiments, the corner position determiner 426 may use other techniques to refine the position of the detected corner (detected from the second image). Fig.12 As shown in , in other embodiments, the output 510 of the spatial binning module 424 (e.g., the positions of the set of corners selected by the non-maximum suppressor 422 and the spatial binning module 424 as a subset of the detected corners in the second image) can be input to the corner detector 416. The corner detector 416 then performs a corner detection algorithm to detect corners at discrete positions in the first image corresponding to the positions of the set of corners from the second image. In one implementation, the corner detector 416 may perform a Harris corner detection algorithm to detect a set of corners in the first image 502 whose positions correspond to the positions of those detected corners in the second image 504. This technique is advantageous because Harris corner detection does not need to be performed on the entire first image 502, but only on discrete portions of the first image 502, thereby further saving time and computing resources. In this case, the corner detector 416 can provide an output 530, which includes a set of corners with refined positions to be used as features / map points of the image.

[0234] In some embodiments, the detection of the corners may be performed by the processing unit 130 to process a real-time input image from a camera system of the image display device 1011. For example, the input image may be provided by the camera system, and the processing unit 130 determines a set of corners from the input image. The set of corners may be used as a set of features of the input image for matching with corresponding features having a positioning map in order to locate the user of the image display device 1011.

[0235] In other embodiments, the detection of corners may be performed by the processing unit 130 to process images from the camera system of the image display device 101 in order to create a positioning map. For example, the processing unit 130 may obtain a sequence of images from the camera system of the image display device 101, and may determine a set of corners from each of the images in the sequence. Images may be obtained by the camera system when the user of the image display device 101 is performing different head gestures so that the images correspond to different viewing directions of the user. The processing unit 130 may also perform stereo matching, in which it matches a corner in one image in the sequence with a corner in another image in the sequence in order to create a 3D map point for the corner. The 3D map point is then included as part of the positioning map.

[0236] Method performed by a processing unit and / or an application in the processing unit

[0237] Fig.131 shows a method 1100 according to some embodiments. The method 1000 may be performed by a device configured to be worn by a user, the device having a screen configured to present graphics to the user, a camera system configured to view the environment in which the user is located, and a processing unit. In some embodiments, the method 1100 may be performed by Figure 1-4 1100 is performed by any of the image display devices 101 shown in . For example, method 1100 can be performed by the processing unit 130 of the image display device 101. Method 1100 includes: obtaining a first image having a first resolution, the first image having a first corner (item 1102). Method 1100 also includes determining a second image having a second resolution, the second image having a second corner corresponding to the first corner in the first image, wherein the second image is based on the first image and the second resolution is less than the first resolution (item 1104). Method 1100 also includes detecting a second corner in the second image (item 1106), and determining a position of the second corner in the second image (item 1108). Method 1100 also includes determining a position of the first corner in the first image based at least in part on the determined position of the second corner in the second image (item 1110).

[0238] Optionally, in method 1100, the second corner is part of a set of corners detected in the second image.

[0239] Optionally, in method 1100, a set of corners is detected in the second image based on Harris corner detection technology.

[0240] Optionally, in method 1100, a set of corners is detected in the second image based on one or more criteria.

[0241] Optionally, in method 1100, the one or more criteria include a first criterion for excluding one or more objects in the second image that are planar and / or have insufficient texture.

[0242] Optionally, in method 1100, the one or more criteria include a second criterion for excluding one or more objects having a linear configuration in the second image.

[0243] Optionally, method 1100 further includes performing non-maximum suppression to eliminate repeatedly detected corners in the corner set.

[0244] Optionally, in method 1100, non-maximum suppression includes 3×3 non-maximum suppression.

[0245] Optionally, method 1100 is performed by a processing unit comprising hardware and software; wherein the set of corners is detected by hardware in the processing unit; and wherein non-maximum suppression is performed by software in the processing unit.

[0246] Optionally, method 1100 further includes performing spatial binning to select a subset of corners from the set of corners.

[0247] Optionally, method 1100 further includes performing spatial binning by: dividing the second image into a plurality of image portions having a first image portion; determining a score for each corner in a set of corners; and selecting one or more corners from the corners in the first image portion based on the scores of the corners in the first image portion and based on a specified maximum number of corners for the first image portion.

[0248] Optionally, method 1100 is performed by a processing unit comprising hardware and software; wherein the set of corners is detected by hardware in the processing unit; and wherein the spatial binning is performed by software in the processing unit.

[0249] Optionally, in method 1100, the second corner is detected in the second image by applying Harris corner detection to the second image.

[0250] Optionally, the method 1100 further includes detecting a first corner in the first image by applying Harris corner detection to the first image.

[0251] Optionally, the method 1100 further includes detecting a set of corners including the first corner in the first image.

[0252] Optionally, the method 1100 further includes identifying a first corner from a set of corners in the first image based at least in part on a position of the second corner in the second image.

[0253] Optionally, in method 1100, the set of corners in the first image is detected by hardware, and the first corner is identified from the set of corners in the first image by software.

[0254] Optionally, in method 1100 , the position of the first corner is determined based on a relationship between a first resolution of the first image and a second resolution of the second image.

[0255] Optionally, in method 1100, the position of the second corner corresponds to a pixel position (xr, yr) in the second image; wherein the pixel position (xr, yr) in the second image corresponds to a plurality of pixel positions in the first image; and wherein the position of the first corner is determined by selecting a pixel position in the plurality of pixel positions in the first image that has an optimal spatial relationship relative to the pixel position (xr, yr) in the second image. For example, the optimal spatial relationship may be a spatial relationship in which the pixel position in the first image is the same as the pixel position in the second image. Alternatively, the optimal spatial relationship may be a spatial relationship in which the pixel position in the first image is within a specified distance from the pixel position in the second image (e.g., within 2 mm, 1 mm, 2 pixel lengths, 1 pixel length, etc.).

[0256] Optionally, in method 1100 , the second image is determined by converting the first image into the second image.

[0257] Optionally, in method 1100 , the first image is converted into the second image by reducing a first resolution of the first image to obtain a second image having a second resolution.

[0258] Optionally, in method 1100, the second corner is part of a set of corners detected in the second image, and wherein the position of the first corner in the first image is determined by software.

[0259] Optionally, method 1100 is performed by a processing unit including hardware.

[0260] Optionally, in method 1100 , a second corner is detected in the second image by hardware of the processing unit.

[0261] Optionally, in method 1100 , the hardware of the processing unit is configured to execute a Harris corner detection algorithm.

[0262] Optionally, method 1100 is performed by a processing unit comprising a corner detector configured to operate based on a first criterion.

[0263] Optionally, in method 1100, the first criterion excludes objects in the second image that are planar and / or have insufficient texture.

[0264] Optionally, in method 1100, the first criterion includes: min >threshold, where λ min =min(λ2,λ1), where λ1 is the first eigenvalue and λ2 is the second eigenvalue greater than the first eigenvalue.

[0265] Optionally, in method 1100, the corner detector is further configured to operate based on a second criterion.

[0266] Optionally, in method 1100, the second criterion excludes objects having a linear configuration in the second image.

[0267] Optionally, in method 1100 , the second criterion includes: λ2 / λ1<threshold, where λ1 is a first eigenvalue, and λ2 is a second eigenvalue greater than the first eigenvalue.

[0268] Optionally, in method 1100 , a second rotation angle is detected in the second image based on the first constant k1.

[0269] Optionally, in method 1100, k1=R / (1+R)^2, where R is any number.

[0270] Optionally, in method 1100 , the first constant k1 is between 0.1 and 0.2.

[0271] Optionally, in method 1100, the first constant k1 is 0.139.

[0272] Optionally, in method 1100 , a second rotation angle is detected in the second image based on a second constant k2.

[0273] Optionally, in method 1100, the second constant k2 is less than 0.01.

[0274] Optionally, in method 1100, the second constant k2 is 0.00478125.

[0275] Optionally, in method 1100 , the second constant k2 is based on the first constant k1 .

[0276] Optionally, in method 1100, k2=(k1-1 / 16) / 16.

[0277] Optionally, in method 1100, a second corner is detected in the second image based on Det(H), where H is a matrix.

[0278] Optionally, in method 1100, a second corner is detected in the second image based on trace(H), where H is a matrix.

[0279] Optionally, in method 1100, the second angle is detected by performing a first Harris angle detection based on a first constant and a second constant, wherein the second constant is different from the first constant.

[0280] Optionally, in method 1100, the second constant is based on the first constant.

[0281] Optionally, the method 1100 further includes performing non-maximum suppression.

[0282] Optionally, the method 1100 further includes performing spatial binning.

[0283] Optionally, the method 1100 is performed by a processing unit comprising software.

[0284] Optionally, the method 1100 further includes performing non-maximum suppression by software of the processing unit.

[0285] Optionally, the method 1100 further includes performing spatial binning by software of the processing unit.

[0286] Optionally, in method 1100, the first resolution comprises a VGA resolution.

[0287] Optionally, in method 1100, the second resolution includes QVGA resolution.

[0288] Optionally, the method 1100 further includes storing the position of the first corner in a non-transitory medium.

[0289] Optionally, the method 1100 further includes creating a map based at least in part on the location of the first corner.

[0290] Optionally, in method 1100, the map includes a map point associated with the location of the first corner.

[0291] Optionally, the method 1100 further includes performing positioning of a user of the image display device based on the created map.

[0292] Optionally, the method 1100 further includes performing matching between the first image and the map based at least in part on the position of the first corner in the first image.

[0293] Dedicated processing system

[0294] In some embodiments, the method 1100 described herein can be performed by a processing unit 130 executing an application or by an application. An application can include a set of instructions. In one implementation, a dedicated processing system having a non-transitory medium storing an instruction set for an application can be provided. Execution of the instructions by the processing unit 130 of the image display device 101 will cause the processing unit 130 to perform the features described herein.

[0295] In some embodiments, the image display device 101 can also be considered a special processing system. Specifically, the image display device 101 is a special processing system because it contains instructions stored in its non-transitory medium, which are used to be executed by the processing unit 130 to provide unique tangible effects in the real world. The features provided by the image display device 101 (as a result of the processing unit 130 executing the instructions) provide improvements in the technical fields of corner detection, positioning map creation, and image-based positioning.

[0296] Fig.14 1 is a block diagram illustrating an embodiment of a dedicated processing system 1600 that can be used to implement various features described herein. For example, in some embodiments, the processing system 1600 can be used to implement the image display device 1011. In addition, in some embodiments, the processing system 1600 can be used to implement the processing unit 130 or one or more components thereof (e.g., the gamma corrector 412, the image adjuster 414, the corner detector 416, etc.).

[0297] The processing system 1600 includes a bus 1602 or other communication mechanism for communicating information and a processor 1604 coupled to the bus 1602 for processing information. The processor system 1600 also includes a main memory 1606, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1602 for storing information and instructions to be executed by the processor 1604. The main memory 1606 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 1604. The processor system 1600 also includes a read-only memory (ROM) 1608 or other static storage device coupled to the bus 1602 for storing static information and instructions for the processor 1604. A data storage device 1610, such as a magnetic disk, solid state disk, or optical disk, is provided and coupled to the bus 1602 for storing information and instructions.

[0298] The processor system 1600 may be coupled to a display 1612, such as a screen, via the bus 1602 for displaying information to a user. In some cases, if the processing system 1600 is part of an apparatus that includes a touch screen, the display 1612 may be a touch screen. An input device 1614 including alphanumeric and other keys is coupled to the bus 1602 for communicating information and command selections to the processor 1604. Another type of user input device is a cursor control 1616, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 1604 and for controlling cursor movement on the display 1612. The input device typically has two degrees of freedom on two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane. In some cases, if the processing system 1600 is part of an apparatus that includes a touch screen, the input device 1614 and the cursor control may be a touch screen.

[0299] In some embodiments, the processor system 1600 may be used to perform various functions described herein. According to some embodiments, such use is provided by the processor system 1600 in response to the processor 1604 executing one or more sequences of one or more instructions contained in the main memory 1606. Those skilled in the art will know how to prepare such instructions based on the functions and methods described herein. Such instructions may be read from another processor-readable medium (e.g., storage device 1610) into the main memory 1606. The execution of the sequence of instructions contained in the main memory 1606 causes the processor 1604 to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequence of instructions contained in the main memory 1606. In alternative embodiments, hard-wired circuits may be used instead of software instructions or in combination with software instructions to implement the various embodiments described herein. Therefore, the embodiments are not limited to any specific combination of hardware circuits and software.

[0300] As used herein, the term "processor-readable medium" refers to any medium that participates in providing instructions to processor 1604 for execution. Such media can take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical, solid-state, or magnetic disks, such as storage device 1610. Non-volatile media can be considered examples of non-transitory media. Volatile media include dynamic memory, such as main memory 1606. Volatile media can be considered examples of non-transitory media. Transmission media include coaxial cables, copper wiring, and optical fiber, including the wiring that comprises bus 1602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0301] Common forms of processor-readable media include, for example, a flexible disk, a hard disk, a magnetic tape or any other magnetic medium, a CD-ROM, any other optical medium, any other physical medium with a pattern of holes, a RAM, a PROM and EPROM, a FLASH-EPROM, a solid state disk or any other memory chip or cartridge, a carrier wave as described below, or any other medium from which a processor can read.

[0302] Various forms of processor-readable media may be involved in carrying one or more sequences of one or more instructions to the processor 1604 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a network (such as the Internet). The processing system 1600 may receive data over a network line. The bus 1602 carries the data to the main memory 1606, from which the processor 1604 retrieves and executes the instructions. The instructions received by the main memory 1606 may optionally be stored on the storage device 1610 before or after execution by the processor 1604.

[0303] Processing system 1600 also includes a communication interface 1618 coupled to bus 1602. Communication interface 1618 provides bidirectional data communication coupled to network link 1620, which is connected to local network 1622. For example, communication interface 1618 can be a local area network (LAN) card that provides a data communication connection to a compatible local area network (LAN). Wireless connections can also be implemented. In any such implementation, communication interface 1618 sends and receives electrical, electromagnetic or optical signals that carry data streams representing various types of information.

[0304] The network link 1620 typically provides data communication to other devices through one or more networks. For example, the network link 1620 can provide a connection to a host computer 1624 or device 1626 through a local network 1622. The data stream transmitted through the network link 1620 may include electrical, electromagnetic or optical signals. The signals through the various networks and the signals on the network link 1620 and the signals through the communication interface 1618 (which carry data to and from the processing system 1600) are exemplary forms of carrier waves that transport information. The processing system 1600 can send messages and receive data including program code through the network, network link 1620 and communication interface 1618.

[0305] It should be noted that, as used in this specification, the term "image" may refer to a displayed image and / or an image that is not in a displayed form (eg, an image that is stored in a medium or is being processed).

[0306] Exemplary aspects of the present disclosure and details about material selection and manufacturing have been described above. For other details of the present disclosure, these details can be understood in conjunction with the above patents and disclosures and in a manner generally known or understood by those skilled in the art. This is also true for the method-based aspects of the present disclosure in terms of additional actions that are usually or logically employed.

[0307] In addition, although the present disclosure has been described with reference to several examples optionally incorporating various features, the present disclosure is not limited to those described or indicated with respect to each variation of the present disclosure. Various changes may be made to the present disclosure described without departing from the spirit and scope of the present disclosure, and equivalents (whether referenced herein or not included for the sake of brevity) may be substituted. In addition, where a range of values ​​is provided, it is understood that each intermediate value between the upper and lower limits of the range, and any other stated or intermediate values ​​within the stated range, are included in the present disclosure.

[0308] Moreover, it is contemplated that any optional feature of the disclosed variations described may be set forth or claimed independently, or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that there are multiple identical items. More specifically, as used herein and in the claims associated herewith, the singular forms "a," "an," "said," and "the" include plural referents unless specifically stated otherwise. It should also be noted that any claim may be drafted to exclude any optional element. Therefore, this statement is intended to serve as antecedent basis for the use of exclusive terminology such as "solely," "only," or the use of a "negative" limitation in conjunction with the recitation of claim elements.

[0309] Furthermore, as used herein, a phrase referring to "at least one" of a list of items refers to any combination of those items, including single members. As an example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. Conjunctions such as the phrase "at least one of X, Y, and Z" (unless otherwise stated) are to be understood in the context of common usage to convey that an item, term, etc. may be at least one of X, Y, or Z. Thus, such conjunctions are generally not intended to imply that certain embodiments require that each of at least one of X, at least one of Y, and at least one of Z be present.

[0310] The breadth of the present disclosure is not limited to the examples provided and / or subject specification, but is limited only by the scope of the claim language associated with the present disclosure.

[0311] In the foregoing description, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure. For example, the above process flow is described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A device configured to be worn on the head of a user, comprising: a screen configured to present graphics to said user; a camera system configured to view an environment in which the user is located; as well as a processing unit coupled to the camera system, the processing unit being configured to: obtaining a first image having a first resolution, the first image having a first rotation angle, determining a second image having a second resolution, the second image having a second rotation angle corresponding to the first rotation angle in the first image, wherein the second image is based on the first image, the second resolution is smaller than the first resolution, and wherein the first image having the first resolution is scaled down to obtain the second image having the second resolution, detecting the second corner in the second image, determining a position of the second corner in the second image, and determining a position of the first corner in the first image based at least in part on the determined position of the second corner in the second image, The processing unit is configured to detect the second rotation angle in the second image based on a first constant k1, wherein k1=R / (1+R)^2, where R is a number other than -1.

2. The device according to claim 1, wherein: The processing unit is configured to detect a set of corners in the second image, the set of corners having the second corner.

3. The device according to claim 2, wherein: The processing unit is configured to detect the set of corners in the second image based on one or more criteria; as well as Wherein, the one or more criteria include a first criterion for excluding one or more objects in the second image that are planar and / or have insufficient texture, a second criterion for excluding one or more objects in the second image that have a linear configuration, or both the first criterion and the second criterion.

4. The device according to claim 2, wherein: The processing unit is also configured to perform non-maximum suppression to eliminate repeatedly detected corners in the corner set.

5. The device according to claim 4, wherein: The processing unit includes hardware and software; wherein the hardware of the processing unit is configured to detect the set of corners; and Wherein, the software of the processing unit is configured to perform the non-maximum suppression.

6. The device according to claim 2, wherein: The processing unit is configured to perform spatial binning to select a subset of corners from the set of corners.

7. The device according to claim 2, wherein: The processing unit is configured to perform spatial binning by: dividing the second image into a plurality of image portions having a first image portion; Determining a score for each of the corners in the set of corners; as well as One or more corners are selected from the corners in the first image portion based on a score of the corners in the first image portion and based on a specified maximum number of corners for the first image portion.

8. The device according to claim 6, wherein: The processing unit includes hardware and software; wherein the hardware of the processing unit is configured to detect the set of corners; and Wherein, the software of the processing unit is configured to perform the spatial binning.

9. The device according to claim 1, wherein: The processing unit is configured to detect a set of corners including the first corner in the first image; and Wherein, the processing unit is further configured to detect the first corner from the set of corners in the first image based at least in part on the position of the second corner in the second image.

10. The device according to claim 9, wherein: The processing unit includes hardware configured to detect the set of corners in the first image, and software configured to detect the first corner from the set of corners in the first image.

11. The device according to claim 1, wherein: The processing unit is configured to determine a position of the first corner based on a relationship between the first resolution of the first image and the second resolution of the second image.

12. The device according to claim 1, wherein: The position of the second corner corresponds to a pixel position (xr, yr) in the second image; wherein the pixel position (xr, yr) in the second image corresponds to a plurality of pixel positions in the first image; and The processing unit is configured to determine the position of the first corner by selecting a pixel position among the multiple pixel positions in the first image that has an optimal spatial relationship with respect to the pixel position (xr, yr) in the second image.

13. The device according to claim 1, wherein: The processing unit includes an image adjuster configured to determine the second image by converting the first image into the second image; as well as The image adjuster is configured to reduce the first resolution of the first image to obtain the second image with the second resolution.

14. The device according to claim 1, wherein: The processing unit comprises a turning angle detector configured to operate based on a first criterion; and Therein, the first criterion excludes objects in the second image that are planar and / or have insufficient texture.

15. The device according to claim 14, wherein: The corner detector is further configured to operate based on a second criterion; and Therein, the second criterion excludes objects having a linear configuration in the second image.

16. The device according to claim 1, wherein The processing unit is configured to detect the second rotation angle in the second image based on a second constant k2.

17. The device according to claim 16, wherein: The second constant k2 is based on the first constant k1.

18. The device according to claim 17, wherein: k2=(k1-1 / 16) / 16.

19. The device according to claim 1, wherein: The processing unit is configured to perform at least one of non-maxima suppression or spatial binning.

20. The device according to claim 19, wherein The processing unit comprises software configured to perform at least one of non-maxima suppression or spatial binning.

21. The device according to claim 1, wherein The processing unit is configured to create a map based at least in part on the location of the first corner, and to perform positioning of the user based on the created map.

22. A method performed by a head mounted image display device, comprising: Obtaining a first image having a first resolution, wherein the first image has a first rotation angle; determining a second image having a second resolution, the second image having a second rotation angle corresponding to the first rotation angle in the first image, wherein the second image is based on the first image, the second resolution is smaller than the first resolution, and wherein the first image having the first resolution is scaled down to obtain the second image having the second resolution; detecting the second corner in the second image; determining a position of the second corner in the second image; and determining a position of the first corner in the first image based at least in part on the determined position of the second corner in the second image, The second rotation angle is detected in the second image based on a first constant k1, wherein k1=R / (1+R)^2, where R is a number other than -1.

23. The method according to claim 22, wherein: The second corner is part of a set of corners detected in the second image.

24. The method according to claim 23, wherein: detecting the set of corners in the second image based on one or more criteria; and Wherein, the one or more criteria include a first criterion for excluding one or more objects in the second image that are planar and / or have insufficient texture, a second criterion for excluding one or more objects in the second image that have a linear configuration, or both the first criterion and the second criterion.

25. The method of claim 23, further comprising: Non-maximum suppression is performed to eliminate repeatedly detected corners in the corner set.

26. The method according to claim 25, wherein: The method is performed by a processing unit including hardware and software; wherein the set of corners is detected by the hardware in the processing unit; and Wherein, the non-maximum suppression is performed by the software in the processing unit.

27. The method of claim 23, further comprising performing spatial binning to select a subset of corners from the set of corners.

28. The method of claim 23, further comprising: Spatial binning is performed by: dividing the second image into a plurality of image portions having a first image portion; Determining a score for each of the corners in the set of corners; as well as One or more corners are selected from the corners in the first image portion based on a score of the corners in the first image portion and based on a specified maximum number of corners for the first image portion.

29. The method according to claim 27, wherein: The method is performed by a processing unit including hardware and software; wherein the set of corners is detected by the hardware in the processing unit; and Wherein, the spatial binning is performed by the software in the processing unit.

30. The method of claim 22, further comprising: detecting, in the first image, a set of corners including the first corner; as well as The first corner is identified from the set of corners in the first image based at least in part on a position of the second corner in the second image.

31. The method according to claim 30, wherein: The set of corners in the first image are detected by hardware, and the first corner is identified by software from the set of corners in the first image.

32. The method of claim 22, wherein: The position of the first corner is determined based on a relationship between the first resolution of the first image and the second resolution of the second image.

33. The method of claim 22, wherein: The position of the second corner corresponds to a pixel position (xr, yr) in the second image; wherein the pixel position (xr, yr) in the second image corresponds to a plurality of pixel positions in the first image; and The position of the first corner is determined by selecting a pixel position having an optimal spatial relationship with respect to the pixel position (xr, yr) in the second image from among the plurality of pixel positions in the first image.

34. The method of claim 22, wherein: determining the second image by converting the first image into the second image; as well as The first image is converted into the second image by reducing the first resolution of the first image to obtain the second image with the second resolution.

35. The method of claim 22, wherein: The method is performed by a processing unit comprising a corner detector configured to operate based on a first criterion; as well as Therein, the first criterion excludes objects in the second image that are planar and / or have insufficient texture.

36. The method of claim 35, wherein: The corner detector is further configured to operate based on a second criterion; and Therein, the second criterion excludes objects having a linear configuration in the second image.

37. The method of claim 22, wherein: The second rotation angle is detected in the second image based on a second constant k2.

38. The method of claim 37, wherein: The second constant k2 is based on the first constant k1.

39. The method of claim 38, wherein: k2=(k1-1 / 16) / 16.

40. The method of claim 22, further comprising: At least one of non-maximum suppression or spatial binning is performed.

41. The method of claim 22, further comprising: A map is created based at least in part on the location of the first corner.

Images