Efficient multi-scale ORB without image size adjustment

Abstract

A method for detecting indicia in an image of a camera is described. In one aspect, a method includes accessing a camera device image generated by an optical sensor of an augmented reality (AR) device; accessing the query image from a storage device of the AR device; and identifying, using a feature detector program, a query image in the camera image without scaling the camera image.

Description

[0001] This application claims the benefit of priority to Indian patent application serial number 202211034970 filed on June 17, 2022, the entire contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] The subject matter disclosed herein generally relates to computer vision systems. Specifically, the present disclosure proposes systems and methods for detecting and tracking visual markers in images. Background Art

[0003] Augmented reality (AR) devices enable a user to observe a scene while seeing related virtual content that can be anchored to items, images, objects, or environments in the device's field of view. For example, an AR device detects images of known markers in a camera image and displays augmented data (e.g., a 3D model of a virtual object) based on the marker's location in the camera image. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0005] Figure 1 is a block diagram illustrating an environment for operating an AR device according to an example embodiment.

[0006] Figure 2 is a block diagram illustrating an AR device according to an example embodiment.

[0007] Figure 3 is a block diagram illustrating a marker locating system according to one example implementation.

[0008] Figure 4 is a block diagram illustrating the operation of a landmark locating system according to one example implementation.

[0009] Figure 5 is a flow chart illustrating a method for detecting a marker according to an example embodiment.

[0010] Figure 6 Examples of a marker image, a standard camera image, and a wide angle image are shown according to an aspect of the subject matter of one example implementation.

[0011] Figure 7 An example of a detector window for a camera image and a detector window for a marker image is shown according to an example embodiment.

[0012] Figure 8 An example of an image pyramid according to the prior art is shown.

[0013] Fig. 9 Another example of an image pyramid according to the prior art is shown.

[0014] Fig.10 A comparison of marker detection operations according to one example implementation is shown.

[0015] Fig.11 is a block diagram illustrating a software architecture within which the present disclosure may be implemented according to an example implementation.

[0016] Fig.12 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed according to an example embodiment.

[0017] Fig.13 A network environment in which a head wearable device may be implemented according to an example implementation is shown. DETAILED DESCRIPTION

[0018] The following description describes the system, method, technology, instruction sequence and computing machine program product of the example implementation of this theme. In the following description, for the purpose of illustration, many specific details are set forth in order to provide an understanding of the various implementations of this theme. However, it will be apparent to those skilled in the art that the implementation of this theme can be put into practice without some or other details in these specific details. Examples represent possible variations only. Unless otherwise clearly stated, structures (e.g., structural components such as modules) are optional and can be combined or subdivided, and operations (e.g., in processes, algorithms or other functions) can be changed in order or combined or subdivided.

[0019] The term "augmented reality" (AR) is used herein to refer to an interactive experience of a real-world environment in which physical objects residing in the real world are "augmented" or enhanced by computer-generated digital content (also referred to as virtual content or synthetic content). AR may also refer to a system that enables the combination of the real and virtual worlds, real-time interaction, and 3D registration of virtual and real objects. Users of AR systems perceive virtual content that appears to be connected to or interacting with physical objects in the real world.

[0020] The term "virtual reality" (VR) is used herein to refer to a simulated experience of a virtual world environment that is completely different from the real world environment. Computer-generated digital content is displayed in a virtual world environment. VR also refers to a system that enables a user of a VR system to be fully immersed in a virtual world environment and interact with virtual objects presented in the virtual world environment.

[0021] The term "AR application" is used herein to refer to a computer-operated application that implements an AR experience. The term "VR application" is used herein to refer to a computer-operated application that implements a VR experience. The term "AR / VR application" refers to a computer-operated application that implements an AR experience or a combination of a VR experience.

[0022] The term "visual tracking system" is used herein to refer to a computer-operated application or system that enables the system to track visual features identified in images captured by one or more cameras of the visual tracking system. The visual tracking system builds a model of the real-world environment based on the tracked visual features. Non-limiting examples of visual tracking systems include: visual simultaneous localization and mapping systems (VSLAM) and visual odometry inertial (VIO) systems. VSLAM can be used to construct a target from an environment or scene based on one or more cameras of the visual tracking system. VIO (also known as visual-inertial tracking system and visual-inertial odometry system) determines the latest posture (e.g., position and orientation) of the device based on data obtained from multiple sensors of the device (e.g., optical sensors, inertial sensors).

[0023] The term "inertial measurement unit" (IMU) is used herein to refer to a device that can report the inertial state of a mobile subject, including the acceleration, velocity, orientation, and location of the mobile subject. The IMU tracks the movement of the subject by integrating the acceleration and angular velocity measured by the IMU. The IMU may also refer to a combination of an accelerometer and a gyroscope, which can determine and quantify linear acceleration and angular velocity, respectively. The values ​​obtained from the IMU gyroscope can be processed to obtain the pitch, roll, and heading of the IMU, thereby obtaining the pitch, roll, and heading of the subject associated with the IMU. The signal from the accelerometer of the IMU can also be processed to obtain the velocity and displacement of the IMU.

[0024] Both AR applications and VR applications enable users to access information, such as information in the form of virtual content presented in a display of an AR / VR display device (also referred to as a display device). The presentation of virtual content can be based on the positioning of the display device relative to a physical object or relative to a reference system (outside the display device) so that the virtual content appears correctly in the display. For AR, virtual content appears anchored to real-world physical objects as perceived by the user and the camera device of the AR display device. Virtual content appears to be attached to / anchored to the physical world (e.g., a physical object of interest). To do this, the AR display device detects physical objects and tracks the posture of the positioning of the AR device relative to the physical object. The gesture recognizes the positioning and orientation of the display device relative to a reference system or relative to another object. For VR, virtual objects appear at a position based on the posture of the VR display device. Therefore, the virtual content is refreshed based on the latest posture of the device. The visual tracking system at the display device determines the posture of the display device. Examples of visual tracking systems include visual inertial tracking systems (also referred to as visual inertial odometer systems), which rely on data acquired from multiple sensors (e.g., optical sensors, inertial sensors).

[0025] The terms "marker" and "visual marker" are used herein to refer to a predefined visual code or image. For example, some visual markers include graphic symbols designed to be easily recognized by a machine. By scanning a visual marker with a camera phone, a user can retrieve location information and access mobile services (e.g., enhanced content that is displayed and appears anchored to the marker).

[0026] A method for detecting a marker in a camera image is described. In one aspect, the method includes: accessing a camera image generated by an optical sensor of an augmented reality (AR) device; accessing a query image from a storage device of the AR device; and identifying the query image in the camera image using a feature detector program without scaling the camera image. The method also includes: scaling a detector window of the feature detector program; and extracting features from the camera image by scanning the unscaled camera image with a scaled detector (of one or more different sizes).

[0027] Thus, one or more of the methods described herein help solve the technical problem of power saving by avoiding computationally intensive image pyramid processing of camera images. The presently described methods provide improvements in the operation of functions of a computer by providing power consumption reduction. Thus, one or more of the methods described herein can eliminate the need for certain workloads or computing resources. Examples of such computing resources include processor cycles, network traffic, memory usage, data storage capacity, power consumption, network bandwidth, and cooling capabilities.

[0028] Figure 1 1 is a network diagram illustrating an environment 100 suitable for operating an imaging device 106 according to some example embodiments. The environment 100 includes a user 102, an imaging device 106, and a marker 104. The user 102 operates the imaging device 106. The user 102 may be a human user (e.g., a human), a machine user (e.g., a computer configured by a software program to interact with the imaging device 106), or any suitable combination thereof (e.g., a human assisted by a machine or a machine supervised by a human). The user 102 is associated with the imaging device 106.

[0029] The imaging device 106 may be a computing device with a display, such as a smart phone, a tablet computer, or a wearable computing device (e.g., a watch or glasses). The computing device may be handheld or may be removably mounted to the head of the user 102. In one example, the display includes a screen that displays images captured using a camera device of the imaging device 106. In another example, the display of the device may be transparent, such as in a lens of wearable computing glasses. In other examples, the display may be opaque, partially transparent, or partially opaque. In yet other examples, the display may be worn by the user 102 to cover a portion of the user's 102 field of view.

[0030] The imaging device 106 includes an AR application 112 and a marker localization system 110. The marker localization system 110 detects the marker 104 in an image captured by a camera of the imaging device 106. In one example, the marker localization system 110 uses a feature detector program (e.g., Figure 3 ) identifies the query image in the image without scaling the image. Figure 3 The marker positioning system 110 is described.

[0031] The AR application 112 generates virtual content based on the marker 104 detected by the camera of the imaging device 106. For example, the user 102 can cause the optical sensor 212 to capture an image of the marker 104. The AR application 112 generates virtual content corresponding to the marker 104 and displays the virtual content in the display 204 of the imaging device 106.

[0032] The AR application 112 uses, for example, optical sensors (e.g., depth-enabled 3D cameras, image cameras), inertial sensors (e.g., gyroscopes, accelerometers), wireless sensors (Bluetooth, Wi-Fi), GPS sensors, and audio sensors to track the posture (e.g., position and orientation) of the imaging device 106 relative to the real-world environment 108. In one example, the imaging device 106 displays virtual content based on the posture of the imaging device 106 relative to the real-world environment 108 and / or the markers 104.

[0033] Figure 1 Any of the machines, databases, or devices shown may be implemented in a general-purpose computer that is modified (e.g., configured or programmed) by software to be a special-purpose computer to perform one or more of the functions described herein for the machine, database, or device. Figure 5 A computer system capable of implementing any one or more of the methods described herein is discussed. As used herein, a "database" is a data storage resource and may store data structured as a text file, a table, a spreadsheet, a relational database (e.g., an object-relational database), a triple store, a hierarchical data store, or any suitable combination thereof. In addition, Figure 1 Any two or more of the machines, databases, or devices shown in may be combined into a single machine, and the functionality described herein for any single machine, database, or device may be subdivided among multiple machines, databases, or devices.

[0034] The imaging device 106 may operate on a computer network. A computer network may be any network that enables communication between or among machines, databases, and devices. Thus, the computer network may be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. The computer network may include one or more parts that constitute a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0035] Figure 2 2 is a block diagram illustrating modules (e.g., components) of an imaging device 106 according to some example embodiments. The imaging device 106 includes a sensor 202, a display 204, a processor 206, and a storage device 208. Examples of the imaging device 106 include a wearable computing device, a mobile computing device, or a smart phone.

[0036] Sensors 202 include, for example, optical sensors 212 (e.g., wide-angle camera, standard-angle camera, narrow-angle camera, depth sensor) and inertial sensors 210 (e.g., gyroscope, accelerometer, magnetometer). Optical sensor 212 generates an image referred to as a "camera image". Other examples of sensors 202 include proximity sensors or position sensors (e.g., near field communication, GPS, Bluetooth, Wifi), audio sensors (e.g., microphones), thermal sensors, pressure sensors (e.g., barometers), or any suitable combination thereof. Note that sensors 202 described herein are for illustrative purposes, and thus sensors 202 are not limited to the sensors described above.

[0037] The display 204 comprises a screen or monitor configured to display images generated by the processor 206. In one example implementation, the display 204 may be transparent or translucent so that the user 102 can see through the display 204 (in an AR use case). In another example implementation, the display 204 covers the eyes of the user 102 and blocks the entire field of view of the user 102 (in a VR use case). In another example, the display 204 comprises a touch screen display configured to receive user input via contact on the touch screen display.

[0038] The processor 206 includes an AR application 112 and a marker positioning system 110. The AR application 112 uses computer vision to detect and identify the physical environment or marker 104. The AR application 112 retrieves virtual content (e.g., a 3D object model) based on the identified marker 104 or physical environment. The AR application 112 renders the virtual object in the display 204. In an example embodiment, the AR application 112 includes a local rendering engine that generates a visualization of the virtual object superimposed on (e.g., superimposed on or otherwise displayed simultaneously with) the image of the marker 104 captured by the optical sensor 212. The visualization of the virtual content can be manipulated by adjusting the positioning of the marker 104 relative to the imaging device 106 (e.g., its physical position, orientation, or both). Similarly, the visualization of the virtual content can be manipulated by adjusting the posture of the imaging device 106 relative to the marker 104.

[0039] The marker localization system 110 detects and identifies the marker 104. For example, the marker localization system 110 retrieves a marker image and compares the marker image with image data from a camera image to detect the marker 104 and the location of the marker 104 in the camera image. In one example, the marker localization system does not resize or scale the camera image, but rather keeps the size of the camera image fixed (e.g., unscaled) and scales the descriptor / detector window of the feature extraction process of the marker localization system 110. An example of a feature detector algorithm includes ORB (Oriented FAST and Rotated BRIEF). Figure 3 The marker positioning system 110 is described in more detail.

[0040] Storage device 208 stores virtual content 214 and tag data 216. For example, tag data 216 includes a predefined image of tag 104. Virtual content 214 includes, for example, a database of visual references (e.g., images of physical objects) and corresponding experiences (e.g., three-dimensional virtual object models).

[0041] Any one or more modules in the modules described herein can be implemented using hardware (e.g., a processor of a machine) or a combination of hardware and software. For example, any module described herein can configure a processor to perform the operation described herein for the module. In addition, any two or more modules in these modules can be combined into a single module, and the functions described herein for a single module can be subdivided among multiple modules. In addition, according to various example embodiments, the modules described herein as implemented in a single machine, database, or device can be distributed across multiple machines, databases, or devices.

[0042] Figure 3 is a block diagram illustrating a landmark locating system 110 according to one example implementation. The landmark locating system 110 includes a landmark module 308, an optical sensor module 302, and a feature detector program 312.

[0043] The labeling module 308 retrieves a labelled image from the labelled data 216. The labelled image is also referred to as a query image. The optical sensor module 302 retrieves a camera image from the optical sensor 212. The camera image remains unscaled and is at the original captured scale. The camera image is also referred to as an unscaled camera image.

[0044] In computer vision, after key points of interest are detected in an image, each key point is described with a unique feature. This feature (mathematically a 1-D vector) is called a descriptor, and those skilled in the art will recognize that there are several techniques for calculating descriptors.

[0045] The feature detector program 312 includes a query image feature extraction module 304 and a camera image feature extraction module 310 and a feature matching module 306. The query image feature extraction module 304 scans the query image using a fixed size detector window (e.g., an unscaled detector 316). The camera image feature extraction module 310 scans the unscaled camera image with the scaled detector 314. The feature matching module 306 compares and matches the extracted features of the query image from the query image feature extraction module 304 and the extracted features of the unscaled camera image from the camera image feature extraction module 310.

[0046] Figure 4 is a block diagram illustrating the operation of the landmark locating system 110 according to one example implementation. The landmark locating system 110 retrieves image data (eg, camera image) from the optical sensor 212 and landmark data (eg, query image) from the storage device 208 .

[0047] The feature detector program 312 detects the marker image in the image data. The feature detector program 312 confirms the location of the detected marker in the image data to the AR application 112.

[0048] The AR application 112 retrieves the virtual content 214 (corresponding to the detected marker) from the storage 208 and causes the virtual content 214 to be displayed at the location of the marker in the camera image (the location in the display 204 ) based on the location of the marker in the camera image.

[0049] Figure 5 5 is a flow chart illustrating a method 500 for comparing descriptors according to an example embodiment. The operations in the method 500 may be performed by the marker positioning system 110 using the above description of Figure 3 Therefore, method 500 is described by way of example with reference to marker positioning system 110. However, it should be understood that at least some of the operations of method 500 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0050] In block 502, the optical sensor module 302 accesses a camera image from the optical sensor 212. In block 504, the labeling module 308 accesses a query image. In block 506, the camera image feature extraction module 310 scales a detector window of a feature detector program (e.g., the feature detector program 312). In block 508, the camera image feature extraction module 310 scans the unscaled camera image with the scaled detector window. In block 510, the camera image feature extraction module 310 extracts features from the unscaled camera image with the scaled detector window. In block 512, the query image feature extraction module 304 scans the query image with an unscaled fixed size window (e.g., the size of the fixed size window is larger than the size of the scaled detector window). In block 514, the query image feature extraction module 304 extracts features from the query image with the unscaled fixed size window. In block 516, the feature matching module 306 compares the descriptors based on the features from the unscaled camera image with the features from the query image.

[0051] It should be noted that other embodiments can use different orders, additional or less operations and different nomenclature or terminology to complete similar functions. In some embodiments, various operations can be performed in parallel with other operations in a synchronous or asynchronous manner. The operations described herein are selected to illustrate some operating principles in a simplified form.

[0052] Figure 6An example of a marker image, a standard camera image, and a wide angle image according to an aspect of the subject matter of one example implementation is shown. Compared to a "standard" camera such as a phone, markers on a low resolution wide angle camera are small and therefore difficult to locate.

[0053] Figure 7 An example of a detector window for a camera image and a detector window for a marker image according to an example implementation is shown. ORB is a feature extraction pipeline that can be used to extract salient features of an image, which in turn can be used to match (find / locate) certain patterns in the image, such as predefined markers.

[0054] Information is extracted by scanning a window over the image. In ORB, this consists of two stages:

[0055] Keypoint detection: finding interesting descriptive points in an image (ORB uses FAST for this)

[0056] Descriptor calculation: Combine the pixels around the key point and store them in the descriptor. (ORB uses rBRIEF descriptor)

[0057] Example ORB operations include:

[0058] (1) Scan the query image with a fixed-size window (the green square in the right image) to extract and store information.

[0059] (2) Scan the camera image with a fixed size window to extract and store information.

[0060] The information from 1) and 2) is compared / matched to locate the query image in the camera image.

[0061] Figure 8 An example of an image pyramid according to the prior art is shown. To make the algorithm more robust and extend the distance range, ORB uses the same image at different sizes to extract information (eg, ORB computes an image pyramid). However, computing all the different image sizes is expensive.

[0062] Fig. 9 Another example of an image pyramid according to the prior art is shown.

[0063] Fig.10 A comparison of marker detection operations (of conventional image scaling 1010 and detector window scaling 1012) is shown according to an example implementation.

[0064] In conventional image scaling 1010, the process involves upscaling the camera image and running ORB:

[0065] ○ Enlarge the image

[0066] ○Compute image pyramid

[0067] ○ Run ORB with a fixed size window for all image scales

[0068] In detector window scaling 1012, the process does not perform any upscaling, reduces / eliminates the required image pyramid levels, and only runs ORB with the scaled window on the original camera image.

[0069] At 1012, zoom window 1002 may include detectors of different zoom sizes.

[0070] Fig.11 1100 is a block diagram illustrating a software architecture 1104 that may be installed on any one or more of the devices described herein. The software architecture 1104 is supported by hardware such as a machine 1102 including a processor 1120, a memory 1126, and an I / O component 1138. In this example, the software architecture 1104 may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1104 includes layers such as an operating system 1112, a library 1110, a framework 1108, and an application 1106. In operation, the application 1106 invokes an API call 1150 through the software stack, and receives a message 1152 in response to the API call 1150.

[0071] The operating system 1112 manages hardware resources and provides public services. The operating system 1112 includes, for example, a kernel 1114, services 1116, and drivers 1122. The kernel 1114 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1114 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1116 can provide other public services to other software layers. Drivers 1122 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1122 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0072] The libraries 1110 provide a low-level common infrastructure used by the applications 1106. The libraries 1110 may include system libraries 1118 (e.g., C standard libraries), which may provide functions such as storage allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 1110 may include API libraries 1124, such as media libraries (e.g., libraries for supporting presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., OpenGL framework for rendering in two dimensions (2D) and three dimensions (3D) in graphics content on a display), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing functions), etc. The library 1110 may also include various other libraries 1128 to provide many other APIs to the application 1106 .

[0073] The framework 1108 provides a high-level common infrastructure used by the applications 1106. For example, the framework 1108 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 1108 can provide a wide range of other APIs that can be used by the applications 1106, some of which can be specific to a particular operating system or platform.

[0074] In an example implementation, applications 1106 may include a home application 1136, a contacts application 1130, a browser application 1132, a book reader application 1134, a location application 1142, a media application 1144, a messaging application 1146, a game application 1148, and a variety of other applications such as third-party applications 1140. Applications 1106 are programs that perform functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1106 structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1140 (e.g., applications developed by an entity other than the vendor of a particular platform using ANDROID ) may be used to create a third-party application 1140. TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, the third-party application 1140 can call the API call 1150 provided by the operating system 1112 to facilitate the functions described in this article.

[0075] Fig.12 1200 within which instructions 1208 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 1200 to perform any one or more of the methods discussed herein. For example, the instructions 1208 may cause the machine 1200 to perform any one or more of the methods described herein. The instructions 1208 transform a general purpose, unprogrammed machine 1200 into a specific machine 1200 that is programmed to perform the functions described and illustrated in the manner described. The machine 1200 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1200 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1200 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1208, sequentially or otherwise, that specify actions to be taken by the machine 1200. In addition, while only a single machine 1200 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1208 to perform any one or more of the methodologies discussed herein.

[0076] The machine 1200 may include a processor 1202, a memory 1204, and an I / O component 1242 configured to communicate with each other via a bus 1244. In an example embodiment, the processor 1202 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, a processor 1206 that executes instructions 1208 and a processor 1210. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Fig.12 Multiple processors 1202 are shown, but the machine 1200 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0077] The memory 1204 includes a main memory 1212, a static memory 1214, and a storage unit 1216, which are all accessible by the processor 1202 via a bus 1244. The main memory 1204, the static memory 1214, and the storage unit 1216 store instructions 1208 that embody any one or more of the methodologies or functions described herein. The instructions 1208 may also reside, completely or partially, within the main memory 1212, within the static memory 1214, within the machine-readable medium 1218 within the storage unit 1216, within at least one of the processors 1202 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 1200.

[0078] I / O components 1242 may include a variety of components that receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O components 1242 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It should be understood that I / O components 1242 may include Fig.12 12 and 13. In various example embodiments, the I / O components 1242 may include output components 1228 and input components 1230. The output components 1228 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, and the like. The input components 1230 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), pointing-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide location and / or force of touch or touch gestures), audio input components (e.g., microphones), and the like.

[0079] In another example embodiment, the I / O component 1242 may include a biometric component 1232, a motion component 1234, an environmental component 1236, or a positioning component 1238, as well as various other components. For example, the biometric component 1232 includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1234 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1236 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases to ensure safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1238 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0080] A variety of technologies may be used to achieve communication. I / O components 1242 also include communication components 1240, which are operable to couple machine 1200 to network 1220 or device 1222 via coupling 1224 and coupling 1226, respectively. For example, communication components 1240 may include a network interface component or another suitable device to interface with network 1220. In other examples, communication components 1240 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Parts (e.g. Low energy consumption), Components and other communication components for providing communication via other modalities. Device 1222 can be another machine or any peripheral device among various peripheral devices (e.g., a peripheral device coupled via USB).

[0081] In addition, the communication component 1240 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1240 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbols (Dataglyph), MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1240, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... Location of signal triangulation, location via detection of NFC beacon signals that can indicate a specific location, and the like.

[0082] Various memories (e.g., memory 1204, main memory 1212, static memory 1214, and / or memory of processor 1202) and / or storage unit 1216 may store one or more sets of instructions and data structures (e.g., software) that embody or are used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1208) when executed by processor 1202 cause various operations to implement the disclosed embodiments.

[0083] Instructions 1208 may be sent or received via a network interface device (e.g., a network interface component included in communication component 1240) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over network 1220. Similarly, instructions 1208 may be sent or received to device 1222 via coupling 1226 (e.g., a peer-to-peer coupling) using a transmission medium.

[0084] As used herein, the terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. These terms refer to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Thus, these terms should be considered to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage medium, computer storage medium, and / or device storage medium include: non-volatile memory, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field programmable gate arrays (FPGA), and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal media" discussed below.

[0085] The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure. The terms "transmission medium" and "signal medium" should be understood to include any intangible medium that is capable of storing, encoding, or carrying instructions 1416 for execution by the machine 1400, and include digital or analog communication signals or other intangible media that facilitate the communication of such software. Thus, the terms "transmission medium" and "signal medium" should be taken to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

[0086] The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure. These terms are defined to include both machine storage media and transmission media. Thus, these terms include both storage devices / medium and carrier / modulated data signals.

[0087] System with head wearable device

[0088] Fig.13 A network environment 1300 is shown in which a head wearable device 1302 may be implemented according to an example implementation. Fig.13 is a high-level functional block diagram of an example head wearable device 1302 communicatively coupled to a mobile client device 1338 and a server system 1332 via various networks 1340.

[0089] The head wearable device 1302 includes a camera, such as at least one of a visible light camera 1312, an infrared emitter 1314, and an infrared camera 1316. The client device 1338 may be able to connect to the head wearable device 1302 using both communication 1334 and communication 1336. The client device 1338 is connected to the server system 1332 and the network 1340. The network 1340 may include any combination of wired and wireless connections.

[0090] The head wearable device 1302 also includes two image displays in the image display 1304 of the optical assembly. The two image displays include one image display associated with the left lateral side of the head wearable device 1302 and one image display associated with the right lateral side of the head wearable device 1302. The head wearable device 1302 also includes an image display driver 1308, an image processor 1310, low power circuitry 1326 for low power consumption, and high speed circuitry 1318. The image display 1304 of the optical assembly is used to present images and videos to a user of the head wearable device 1302, including images that may include a graphical user interface.

[0091] The image display driver 1308 commands and controls the image display in the image display 1304 of the optical assembly. The image display driver 1308 may deliver the image data directly to the image display of the image display 1304 of the optical assembly for presentation, or may have to convert the image data into a signal or data format suitable for delivery to the image display device. For example, the image data may be video data formatted according to a compression format such as H.264 (MPEG-4), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., and still image data may be formatted according to a compression format such as Portable Network Graphics Group (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or Exchangeable Image File Format (Exif), etc.

[0092] As described above, the head wearable device 1302 includes a frame and a handle (or temple) extending from the side of the frame. The head wearable device 1302 also includes a user input device 1306 (e.g., a touch sensor or a push button) including an input surface on the head wearable device 1302. The user input device 1306 (e.g., a touch sensor or a push button) is used to receive input selections from the user to manipulate the graphical user interface of the presented image.

[0093] Fig.13The components for the head wearable device 1302 are shown to be located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the frame or temples. Alternatively or additionally, the depicted components can be located in a block, frame, hinge, or nose bridge of the head wearable device 1302. The left and right sides can include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, camera lenses, or any other corresponding visible or light capturing elements that can be used to capture data (including images of scenes with unknown objects).

[0094] The head wearable device 1302 includes a memory 1322 that stores instructions for performing a subset or all of the functions described herein. The memory 1322 may also include a storage device.

[0095] like Fig.13 As shown, the high-speed circuit system 1318 includes a high-speed processor 1320, a memory 1322, and a high-speed wireless circuit system 1324. In this example, the image display driver 1308 is coupled to the high-speed circuit system 1318 and is operated by the high-speed processor 1320 to drive the left image display and the right image display of the image display 1304 of the optical assembly. The high-speed processor 1320 can be any processor capable of managing high-speed communications and operations of any general computing system required by the head wearable device 1302. The high-speed processor 1320 includes the processing resources required to manage high-speed data transmission on the communication 1336 to the wireless local area network (WLAN) using the high-speed wireless circuit system 1324. In some examples, the high-speed processor 1320 executes an operating system such as a LINUX operating system or other such operating system of the head wearable device 1302, and the operating system is stored in the memory 1322 for execution. In addition to any other duties, the high-speed processor 1320 that executes the software architecture of the head wearable device 1302 is used to manage data transmission with the high-speed wireless circuit system 1324. In some examples, high-speed wireless circuit system 1324 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards can be implemented by high-speed wireless circuit system 1324.

[0096] The low-power wireless circuit system 1330 and the high-speed wireless circuit system 1324 of the head wearable device 1302 may include a short-range transceiver (Bluetooth TM) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or WiFi). Client device 1338 (including a transceiver for communicating via communication 1334 and communication 1336) can be implemented using details of the architecture of head wearable device 1302, as can other elements of network 1340.

[0097] The memory 1322 includes any storage device capable of storing various data and applications, including camera data generated by the left and right, infrared cameras 1316 and the image processor 1310, and images generated by the image display driver 1308 for display on the image display in the optical assembly 1304. Although the memory 1322 is shown as being integrated with the high-speed circuit system 1318, in other examples, the memory 1322 can be a separate stand-alone element of the head wearable device 1302. In some such examples, the circuit routing can provide a connection from the image processor 1310 or the low-power processor 1328 to the memory 1322 through a chip including the high-speed processor 1320. In other examples, the high-speed processor 1320 can manage the addressing of the memory 1322 so that the low-power processor 1328 will start the high-speed processor 1320 at any time when a read or write operation involving the memory 1322 is required.

[0098] like Fig.13 As shown, the low-power processor 1328 or the high-speed processor 1320 of the head wearable device 1302 can be coupled to a camera device (a visible light camera device 1312; an infrared emitter 1314 or an infrared camera device 1316), an image display driver 1308, a user input device 1306 (for example, a touch sensor or a push button), and a memory 1322.

[0099] The head wearable device 1302 is connected to a host computer. For example, the head wearable device 1302 is paired with a client device 1338 via communication 1336 or connected to a server system 1332 via a network 1340. The server system 1332 can be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the client device 1338 and the head wearable device 1302 through the network 1340.

[0100] The client device 1338 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication over the network 1340, communication 1334, or communication 1336. The client device 1338 may also store at least some of the instructions for generating two-channel audio content in a memory of the client device 1338 to implement the functions described herein.

[0101] The output components of the head wearable device 1302 include visual components, such as a display, such as a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide. The image display of the optical assembly is driven by an image display driver 1308. The output components of the head wearable device 1302 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components of the head wearable device 1302, the client device 1338, and the server system 1332, such as the user input device 1306, may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), pointing-based input components (e.g., mice, touch pads, tracking balls, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide the location and force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0102] The head wearable device 1302 may optionally include additional peripheral elements. Such peripheral elements may include biometric sensors, additional sensors, or display elements integrated with the head wearable device 1302. For example, the peripheral elements may include any I / O components including output components, motion components, positioning components, or any other such components described herein.

[0103] For example, the biometric component includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biological signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The positioning component includes a position sensor component for generating position coordinates (e.g., a global positioning system (GPS) receiver component), a WiFi or a GPS receiver component for generating positioning system coordinates, etc. transceiver, altitude sensor components (e.g., an altimeter or barometer that detects air pressure, from which altitude can be derived), orientation sensor components (e.g., a magnetometer), etc. Such positioning system coordinates may also be received from client device 1338 via low power wireless circuitry 1330 or high speed wireless circuitry 1324 via communication 1336.

[0104] When phrases similar to "at least one of A, B, or C," "at least one of A, B and C," "one or more A, B, or C," or "one or more A, B, and C" are used, it is intended that the phrase be interpreted to mean that A may be present alone in an embodiment, B may be present alone in an embodiment, C may be present alone in an embodiment, or any combination of elements A, B, and C may be present in a single embodiment; for example, A and B, A and C, B and C, or A and B and C.

[0105] Changes and modifications may be made to the disclosed embodiments without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

[0106] Although the embodiments have been described with reference to specific example embodiments, it will be apparent that various modifications and changes may be made to these embodiments without departing from the broader scope of the present disclosure. Therefore, the description and the drawings should be considered illustrative, rather than restrictive. The drawings forming part of the present invention illustrate specific embodiments in which the subject matter can be practiced by way of illustration and not limitation. The embodiments shown are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be utilized and derived therefrom so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, this specific embodiment should not be understood as limiting, and the scope of the various embodiments is limited only by the appended claims and the full scope of the equivalents given by these claims.

[0107] Such embodiments of the subject matter of the present invention may be referred to herein individually and / or collectively by the term "invention", which is merely for convenience and is not intended to voluntarily limit the scope of the present application to any single invention or inventive concept if, in fact, more than one invention or inventive concept is disclosed. Therefore, although specific embodiments have been shown and described herein, it should be understood that any arrangement calculated to achieve the same purpose may replace the specific embodiments shown. The present disclosure is intended to cover any and all changes or modifications of the various embodiments. Upon reviewing the above description, the combination of the above embodiments and other embodiments not specifically described herein will be obvious to those skilled in the art.

[0108] An abstract of the disclosure is provided to enable the reader to quickly determine the nature of the technical disclosure. The abstract is submitted with the understanding that the abstract will not be used to interpret or limit the scope or meaning of the claims. In addition, in the aforementioned detailed description, it can be seen that various features are combined together in a single embodiment for the purpose of simplifying the disclosure. This method of the disclosure should not be interpreted as reflecting the intention that the claimed embodiment requires more features than the features explicitly recited in each claim. On the contrary, as reflected in the attached claims, the subject matter of the present invention lies in less than all the features of a single disclosed embodiment. Therefore, the attached claims are hereby incorporated into the detailed description, wherein each claim exists independently as a separate embodiment.

Claims

1. A method comprising: accessing a camera device image generated by an optical sensor of an augmented reality (AR) device; accessing a query image from a storage device of the AR device; as well as The query image is identified in the camera image without scaling the camera image using a feature detector procedure.

2. The method according to claim 1, further comprising: scaling a detector window of the feature detector program; as well as Features are extracted from the camera image by scanning the unscaled camera image with a scaled detector.

3. The method according to claim 2, further comprising: Features are extracted from the query image by scanning the query image with an unscaled detector.

4. The method according to claim 3, further comprising: Descriptors based on features extracted from the camera image are compared to descriptors based on features extracted from the query image.

5. The method according to claim 1, further comprising: accessing a virtual content item corresponding to the query image; as well as The virtual content item is displayed on a display of the AR device.

6. The method according to claim 1, further comprising: The camera image is generated using a wide angle lens coupled to the optical sensor.

7. The method according to claim 1, wherein: The camera image comprises a low resolution image.

8. The method according to claim 1, wherein: The feature detector program includes the ORB (Oriented FAST and Rotated BRIEF) local feature detector.

9. The method according to claim 1, wherein: The feature detector program is configured to compare descriptors based on features extracted from the camera image using the scaled detector with descriptors based on features extracted from the query image using the unscaled detector.

10. The method according to claim 1, wherein: The query image includes labeling data indicative of a predefined visual code.

11. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the device to: accessing a camera device image generated by an optical sensor of an augmented reality (AR) device; accessing a query image from a storage device of the AR device; as well as The query image is identified in the camera image without scaling the camera image using a feature detector procedure.

12. The computing device of claim 11, wherein: The instructions also configure the device to: scaling a detector window of the feature detector program; and Features are extracted from the camera image by scanning the unscaled camera image with a scaled detector.

13. The computing device of claim 12, wherein: The instructions also configure the device to: Features are extracted from the query image by scanning the query image with an unscaled detector.

14. The computing device of claim 13, wherein: The instructions also configure the device to: Descriptors based on features extracted from the camera image are compared to descriptors based on features extracted from the query image.

15. The computing device of claim 11, wherein: The instructions also configure the device to: accessing a virtual content item corresponding to the query image; and The virtual content item is displayed on a display of the AR device.

16. The computing device of claim 11, wherein: The instructions also configure the device to: The camera image is generated using a wide angle lens coupled to the optical sensor.

17. The computing device of claim 11, wherein: The camera image comprises a low resolution image.

18. The computing device of claim 11, wherein: The feature detector program includes the ORB (Oriented FAST and Rotated BRIEF) local feature detector.

19. The computing device of claim 11, wherein: The feature detector program is configured to compare descriptors based on features extracted from the camera image using the scaled detector with descriptors based on features extracted from the query image using the unscaled detector.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: accessing a camera device image generated by an optical sensor of an augmented reality (AR) device; accessing a query image from a storage device of the AR device; and The query image is identified in the camera image without scaling the camera image using a feature detector procedure.

Images