Direct Scale Level Selection for Multi-Level Feature Tracking

Abstract

This invention relates to direct scale level selection for multi-level feature tracking. A method for mitigating motion blur in a visual-inertial tracking system is described. In one aspect, the method includes: accessing a first image generated by an optical sensor of the visual tracking system; accessing a second image generated by the optical sensor of the visual tracking system, the second image following the first image; determining a first motion blur level of the first image; determining a second motion blur level of the second image; identifying a scale change between the first and second images; determining a first optimal scale level for the first image based on the first motion blur level and the scale change; and determining a second optimal scale level for the second image based on the second motion blur level and the scale change.

Description

[0001] This application is a divisional application of Chinese patent application No. 202280036061.6, filed on November 17, 2023, entitled "Direct Scale Level Selection for Multi-Level Feature Tracking". The international filing date of the parent application is May 16, 2022, and the international application number is PCT / US2022 / 072340, with the earliest priority date being May 18, 2021.

[0002] Related applications

[0003] This application claims the benefit of priority to U.S. Application Serial No. 17 / 521,109, filed November 8, 2021, which claims priority to U.S. Provisional Patent Application Serial No. 63 / 190,101, filed May 18, 2021, each of which is incorporated herein by reference in its entirety. Technical Field

[0004] The subject matter disclosed herein generally relates to visual tracking systems. Specifically, this disclosure relates to systems and methods for mitigating motion blur in visual-inertial tracking systems. Background Technology

[0005] Augmented reality (AR) devices allow users to observe a scene while seeing related virtual content that can be aligned with items, images, objects, or the environment within the device's field of view. Virtual reality (VR) devices offer a more immersive experience than AR devices. VR devices use virtual content displayed based on the VR device's positioning and orientation to obscure the user's field of view.

[0006] Both AR and VR devices rely on motion tracking systems to track the device's posture (e.g., orientation, orientation, position). Motion tracking systems (also known as visual tracking systems) use images captured by the AR / VR device's optical sensors to track its posture. However, images become blurred when the AR / VR device moves rapidly. Therefore, high motion blur leads to degraded tracking performance. Alternatively, high motion blur results in higher computational operations to maintain sufficient tracking accuracy and image quality under high dynamic conditions. Summary of the Invention

[0007] According to one aspect of the present invention, a method is provided, comprising: accessing a first image and a second image following the first image, the first image and the second image being generated by an optical sensor of a visual tracking system; determining a first motion blur level of the first image and a second motion blur level of the second image; identifying a scale variation between the first image and the second image; calculating a first matching level based on the first motion blur level, and calculating a second matching level based on the second motion blur level; determining a base matching level based on the maximum value of the first matching level and the second matching level; and adjusting the base matching level based on the scale variation.

[0008] According to one aspect of the present invention, a computing device is provided, comprising: one or more processors; and a memory storing instructions, which, when executed by the one or more processors, configure the device to perform operations including: accessing a first image and a second image following the first image, the first image and the second image being generated by an optical sensor of a visual tracking system; determining a first motion blur level of the first image and a second motion blur level of the second image; identifying a scale variation between the first image and the second image; calculating a first matching level based on the first motion blur level and calculating a second matching level based on the second motion blur level; determining a base matching level based on the maximum value of the first matching level and the second matching level; and adjusting the base matching level based on the scale variation.

[0009] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, the computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform operations, the operations including: accessing a first image and a second image following the first image, the first image and the second image being generated by an optical sensor of a vision tracking system; determining a first motion blur level of the first image and a second motion blur level of the second image; identifying a scale variation between the first image and the second image; calculating a first matching level based on the first motion blur level and calculating a second matching level based on the second motion blur level; determining a base matching level based on the maximum value of the first matching level and the second matching level; and adjusting the base matching level based on the scale variation. Attached Figure Description

[0010] To facilitate identification of any particular element or behavior being discussed, one or more of the highest-order digits in the reference numerals indicate the figure number in which the element was first introduced.

[0011] Figure 1This is a block diagram illustrating an environment for operating an AR / VR display device according to an example implementation.

[0012] Figure 2 This is a block diagram illustrating an AR / VR display device according to an example implementation.

[0013] Figure 3 This is a block diagram illustrating a visual tracking system according to an example implementation.

[0014] Figure 4 This is a block diagram illustrating a motion blur mitigation module according to an example implementation.

[0015] Figure 5 This is a block diagram illustrating a process according to an example implementation.

[0016] Figure 6 This is a flowchart illustrating a method for mitigating motion blur according to an example implementation.

[0017] Figure 7 This is a flowchart illustrating a method for mitigating motion blur according to an example implementation.

[0018] Figure 8 An example of a first scenario of the subject matter according to one implementation is shown.

[0019] Figure 9 An example of a second scenario of the subject matter according to one implementation is shown.

[0020] Figure 10 An example of a third scenario of the subject matter according to one implementation is shown.

[0021] Figure 11 An example of a fourth scenario of the subject matter according to one implementation is shown.

[0022] Figure 12 An example of a fifth scenario of the subject matter according to one implementation is shown.

[0023] Figure 13 An example of pseudocode for motion blur mitigation according to one implementation is shown.

[0024] Figure 14 An example of an algorithm for motion blur reduction according to one embodiment is shown.

[0025] Figure 15 A network environment in which a head-mounted device can be implemented is shown according to an example implementation.

[0026] Figure 16This is a block diagram illustrating a software architecture in which the present disclosure can be implemented according to an example embodiment.

[0027] Figure 17 It is a schematic representation of a machine in the form of a computer system according to an example implementation, in which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0028] The following description illustrates systems, methods, techniques, sequences of instructions, and computer program products that demonstrate exemplary embodiments of the subject matter. In this description, numerous specific details are set forth for illustrative purposes to provide an understanding of various embodiments of the subject matter. However, it will be apparent to those skilled in the art that embodiments of the subject matter can be practiced without some or more of these specific details. The examples merely represent possible variations. Unless explicitly stated otherwise, structures (e.g., structural components, such as modules) are optional and can be combined or subdivided, and operations (e.g., in processes, algorithms, or other functions) can vary in order or be combined or subdivided.

[0029] The term "augmented reality" (AR) is used in this article to refer to interactive experiences in real-world environments where physical objects existing in the real world are "enhanced" or strengthened by computer-generated digital content (also known as virtual or synthetic content). AR can also refer to systems that enable the combination of the real and virtual worlds, real-time interaction, and 3D registration of virtual and real objects. Users of AR systems perceive virtual content that appears to be attached to or interact with physical objects in the real world.

[0030] The term "virtual reality" (VR) is used in this article to refer to a simulated experience of a virtual world environment that is completely different from the real world environment. Computer-generated digital content is displayed in the virtual world environment. VR also refers to systems that allow users to be fully immersed in a virtual world environment and interact with virtual objects presented within that environment.

[0031] The term "AR application" is used herein to refer to a computer-operated application that enables an AR experience. The term "VR application" is used herein to refer to a computer-operated application that enables a VR experience. The term "AR / VR application" refers to a computer-operated application capable of enabling either an AR experience or a combination of AR and VR experiences.

[0032] The term "visual tracking system" is used herein to refer to a computer-operated application or system that enables the system to track visual features identified in images captured by one or more camera devices of the visual tracking system. The visual tracking system builds a model of a real-world environment based on the tracked visual features. Non-limiting examples of visual tracking systems include visual simultaneous localization and mapping (VSLAM) systems and visual odometry (VIO) systems. VSLAM can be used to build a target from an environment or scene based on one or more camera devices of the visual tracking system. VIO (also known as visual inertial tracking systems and visual inertial odometry systems) determines the latest posture (e.g., localization and orientation) of the device based on data acquired from multiple sensors (e.g., optical sensors, inertial sensors) of the device.

[0033] The term "Inertial Measurement Unit" (IMU) is used herein to refer to a device capable of reporting the inertial state of a moving object, including its acceleration, velocity, orientation, and position. An IMU tracks the motion of an object by integrating the acceleration and angular velocity measured by the IMU. An IMU can also refer to a combination of accelerometers and gyroscopes that separately determine and quantize linear acceleration and angular velocity. Values ​​obtained from the IMU's gyroscopes can be processed to obtain the IMU's pitch, roll, and heading, and thus the pitch, roll, and heading of the object associated with the IMU. Signals from the IMU's accelerometers can also be processed to obtain the IMU's velocity and displacement.

[0034] Both AR and VR applications allow users to access information, such as in the form of virtual content presented on the display of an AR / VR display device (also referred to as a display device). The presentation of virtual content can be based on the positioning of the display device relative to a physical object or relative to a frame of reference (outside the display device), ensuring that the virtual content appears correctly on the display. For AR, the virtual content appears aligned with the physical object perceived by the user and the camera mechanism of the AR display device. The virtual content appears attached to the physical world (e.g., a physical object of interest). To do this, the AR display device detects the physical object and tracks the pose of the AR display device relative to the physical object. The pose identifies the positioning and orientation of the display device relative to a frame of reference or relative to another object. For VR, the virtual object appears at a location based on the pose of the VR display device. Therefore, the virtual content is refreshed based on the device's latest pose. A visual tracking system at the display device determines the pose of the display device. Examples of visual tracking systems include visual inertial tracking systems (e.g., VIO systems) that rely on data acquired from multiple sensors (e.g., optical sensors, inertial sensors).

[0035] When the camera device moves rapidly (e.g., rotates rapidly), the image captured by the visual tracking system may become blurred. Motion blur in the image can lead to decreased tracking performance (of the visual tracking system). Alternatively, motion blur can also lead to higher computational operations in the visual tracking system to maintain sufficient tracking accuracy and image quality under high dynamic conditions.

[0036] Specifically, visual tracking systems are typically based on image feature matching components. In an incoming video stream, the algorithm detects different 3D points in an image (features) and attempts to find (match) these points again in subsequent images. The first image in this matching process is referred to herein as the "source image." The second image (e.g., the subsequent image in which features are to be matched) is referred to herein as the "target image."

[0037] Reliable feature points are typically detected in high-contrast areas of an image, such as corners or edges. However, for head-mounted devices with built-in cameras, the cameras may move rapidly as the user shakes their head, resulting in severe motion blur in the images captured by the built-in cameras. Such rapid movement leads to blurred high-contrast areas. Therefore, the feature detection and matching stages of the visual tracking system are negatively affected, and the overall tracking accuracy of the system is compromised.

[0038] A common strategy to mitigate motion blur is to perform feature detection and matching on downsampled versions of both the source and target images if matching at the original image resolution fails due to motion blur. Motion blur is reduced when visual information is lost in the downsampled image versions. Therefore, feature matching becomes more reliable. Typically, images are downsampled multiple times to obtain different resolutions for varying degrees of motion blur, and the collection of all these different versions is called an image pyramid. This downscaling process is also known as "image pyramid processing" or "image pyramid algorithm." However, image pyramid processing can be time-consuming and computationally intensive.

[0039] A typical image pyramid algorithm performs iterative downscaling at multiple levels on both the source and target images until features from the downscaled levels of the source image match features from the downscaled levels of the target image. For example, in a fine-to-coarse processing, the image pyramid algorithm starts at the finest level (highest image resolution) and continues until a match is successful. In a coarse-to-fine processing, the image pyramid algorithm starts at the coarsest level (lowest image resolution) and stops if a match fails. In either case, the image pyramid algorithm performs matching at many levels.

[0040] This application describes a method for identifying the optimal scale level for feature matching. Instead of attempting to match features at every scale level in an image pyramid algorithm until a successful match is detected, the method described herein predicts the optimal scale level for feature matching before matching processing based on multiple inputs, such as motion blur estimation and predicted scale change. Therefore, only one matching attempt is required per image for each feature, resulting in shorter processing time.

[0041] In one example implementation, this application describes a method for mitigating motion blur in a visual-inertial tracking system. The method includes: accessing a first image generated by an optical sensor of the visual tracking system; accessing a second image generated by the optical sensor of the visual tracking system after the first image; determining a first motion blur level of the first image; determining a second motion blur level of the second image; identifying a scale change between the first and second images; determining a first optimal scale level for the first image based on the first motion blur level and the scale change; and determining a second optimal scale level for the second image based on the second motion blur level and the scale change.

[0042] Therefore, one or more methods described herein help address the technical challenge of power saving by identifying the optimal scale level for the current image in image pyramid processing. The methods described herein provide improvements to computer functionality by offering reduced power consumption while still maintaining robustness of visual inertial tracking to motion blur. Thus, one or more methods described herein can avoid the need for certain efforts or computational resources. Examples of such computational resources include processor cycles, network traffic, memory usage, data storage capacity, power consumption, network bandwidth, and cooling capacity.

[0043] Figure 1 This is a network diagram illustrating an environment 100 suitable for operating an AR / VR display device 106 according to some example embodiments. Environment 100 includes a user 102, the AR / VR display device 106, and physical objects 104. The user 102 operates the AR / VR display device 106. The user 102 can be a human user (e.g., a human), a machine user (e.g., a computer configured by software programs to interact with the AR / VR display device 106), or any suitable combination thereof (e.g., a machine-assisted human or a machine supervised by a human). The user 102 is associated with the AR / VR display device 106.

[0044] AR / VR display device 106 can be a computing device with a display, such as a smartphone, tablet, or wearable computing device (e.g., a watch or glasses). The computing device can be handheld or removably mounted on the head of user 102. In one example, the display includes a screen that displays images captured by a camera device of AR / VR display device 106. In another example, the device's display can be transparent, for example, within the lenses of wearable computing glasses. In other examples, the display can be opaque, partially transparent, or partially opaque. In still other examples, the display can be worn by user 102 to cover user 102's field of vision.

[0045] AR / VR display device 106 includes an AR application that generates virtual content based on images detected by a camera device of AR / VR display device 106. For example, user 102 can instruct the camera device of AR / VR display device 106 to capture an image of a physical object 104. The AR application generates virtual content corresponding to the identified object (e.g., physical object 104) in the image, and the virtual content is displayed on the screen of AR / VR display device 106.

[0046] AR / VR display device 106 includes a visual tracking system 108. The visual tracking system 108 uses, for example, optical sensors (e.g., a 3D camera with depth capabilities, an image capture device), inertial sensors (e.g., a gyroscope, an accelerometer), wireless sensors (Bluetooth, Wi-Fi), GPS sensors, and audio sensors to track the posture (e.g., positioning and orientation) of the AR / VR display device 106 relative to the real-world environment 110. In one example, the AR / VR display device 106 displays virtual content based on the posture of the AR / VR display device 106 relative to the real-world environment 110 and / or physical object 104.

[0047] Figure 1 Any of the machines, databases, or devices shown can be implemented in a general-purpose computer that has been software-modified (e.g., configured or programmed) to perform one or more of the functions described herein for that machine, database, or device. For example, see below. Figures 6 to 7 This discussion focuses on computer systems capable of implementing any one or more of the methods described herein. As used herein, a "database" is a data storage resource and can store data structured as text files, tables, spreadsheets, relational databases (e.g., object-relational databases), triplet storage, hierarchical data storage, or any suitable combination thereof. Furthermore, Figure 1Any two or more of the machines, databases or devices shown may be combined into a single machine, and the functionality described herein for any single machine, database or device may be subdivided among multiple machines, databases or devices.

[0048] AR / VR display device 106 can operate via a computer network. The computer network can be any network that enables communication between or within machines, databases, and devices. Therefore, the computer network can be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. The computer network may include one or more components constituting a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0049] Figure 2 This is a block diagram illustrating the modules (e.g., components) of an AR / VR display device 106 according to some example embodiments. The AR / VR display device 106 includes a sensor 202, a display 204, a processor 206, and a storage device 208. Examples of the AR / VR display device 106 include wearable computing devices, mobile computing devices, navigation devices, portable media devices, or smartphones.

[0050] Sensor 202 includes, for example, optical sensors 212 (e.g., imaging devices such as color imaging devices, thermal imagers, depth sensors, and one or more grayscale, global / roll shutter tracking cameras) and inertial sensors 210 (e.g., gyroscopes, accelerometers, magnetometers). Other examples of sensor 202 include proximity sensors or position sensors (e.g., near-field communication, GPS, Bluetooth, Wi-Fi), audio sensors (e.g., microphones), thermal sensors, pressure sensors (e.g., barometers), or any suitable combination thereof. Note that sensor 202 described herein is for illustrative purposes and therefore sensor 202 is not limited to the sensors described above.

[0051] Display 204 includes a screen or monitor configured to display images generated by processor 206. In one example implementation, display 204 may be transparent or semi-opaque, allowing user 102 to view through it (in an AR use case). In another example implementation, display 204 covers user 102's eyes and obstructs user 102's entire field of vision (in a VR use case). In yet another example, display 204 includes a touchscreen display configured to receive user input via touch on a touchscreen display.

[0052] Processor 206 includes AR / VR application 214 and visual tracking system 108. AR / VR application 214 uses computer vision to detect and identify the physical environment or physical object 104. AR / VR application 214 retrieves virtual content (e.g., 3D object models) based on the identified physical object 104 or physical environment. AR / VR application 214 renders the virtual object in display 204. In one example implementation, AR / VR application 214 includes a local rendering engine that generates a visualization of the virtual content overlaid (e.g., superimposed on or otherwise co-displayed with) an image of the physical object 104 captured by optical sensor 212. The visualization of the virtual content can be manipulated by adjusting the positioning (e.g., its physical location, orientation, or both) of the physical object 104 relative to AR / VR display device 106. Similarly, the visualization of the virtual content can be manipulated by adjusting the pose of AR / VR display device 106 relative to physical object 104. For VR applications, AR / VR application 214 displays virtual content on display 204 at a position determined by the posture of AR / VR display device 106 (in display 204).

[0053] The visual tracking system 108 estimates the pose of the AR / VR display device 106. For example, the visual tracking system 108 uses image data and corresponding inertial data from the optical sensor 212 and the inertial sensor 210 to track the position and pose of the AR / VR display device 106 relative to a reference frame (e.g., the real-world environment 110). See below... Figure 3 The visual tracking system 108 is described in more detail.

[0054] Storage device 208 stores virtual content 216. Virtual content 216 includes a database of, for example, visual references (e.g., images of physical objects) and corresponding experiences (e.g., three-dimensional virtual object models).

[0055] Any one or more modules described herein may be implemented using hardware (e.g., a machine's processor) or a combination of hardware and software. For example, any module described herein may configure a processor to perform the operations described herein for that module. Furthermore, any two or more of these modules may be combined into a single module, and the functionality described herein for a single module may be subdivided among multiple modules. Moreover, modules described herein as being implemented within a single machine, database, or device, according to various example implementations, may be distributed across multiple machines, databases, or devices.

[0056] Figure 3A visual tracking system 108 according to an example embodiment is shown. The visual tracking system 108 includes an inertial sensor module 302, an optical sensor module 304, a blur mitigation module 306, and a pose estimation module 308. The inertial sensor module 302 accesses inertial sensor data from inertial sensor 210. The optical sensor module 304 accesses optical sensor data (e.g., images, camera device settings / operation parameters) from optical sensor 212. Examples of camera device operation parameters include, but are not limited to, the exposure time of optical sensor 212, the field of view of optical sensor 212, the ISO value of optical sensor 212, and the image resolution of optical sensor 212.

[0057] In one example implementation, the blur reduction module 306 determines the angular velocity of the optical sensor 212 based on IMU sensor data from the inertial sensor 210. The blur reduction module 306 estimates the motion blur level based on the angular velocity and camera device operating parameters without performing any analysis of the pixels in the image.

[0058] In another example implementation, the blur reduction module 306 considers both the angular velocity and linear velocity of the optical sensor 212 based on the current velocity estimate from the visual tracking system 108 and the 3D position of the currently tracked point in the current image. For example, the blur reduction module 306 determines the linear velocity of the optical sensor 212 and its effect on different regions of the current image based on the distance of the object (from the optical sensor 212) in the current image (e.g., determined by the 3D position of feature points). Therefore, objects closer to the optical sensor 212 appear more blurred than objects farther away (when the optical sensor 212 is moving).

[0059] The blur reduction module 306 downscales the image captured by the optical sensor 212 based on the image's motion blur level. For example, the blur reduction module 306 determines that the current image is blurry and applies an image pyramid algorithm to the current image to increase contrast. In one example implementation, the blur reduction module 306 identifies the optimal scale level for feature matching. Instead of attempting to match features at each scale level of the image pyramid algorithm until a successful match is detected, the blur reduction module 306 predicts the optimal scale level for feature matching before matching processing based on motion blur estimation and predicted scale changes. The higher the estimated motion blur, the lower the optimal resolution for feature matching. The greater the scale change between the source and target images, the more adjustments are made to the optimal scale level of the image pyramid algorithm. By predicting the optimal scale level for both the source and target images, the blur reduction module 306 performs only one feature matching attempt for each image, resulting in a shorter processing time. See below for reference. Figure 4A sample component of the blur reduction module 306 is described in more detail.

[0060] The pose estimation module 308 determines the pose (e.g., position, orientation, orientation) of the AR / VR display device 106 relative to a reference frame (e.g., real-world environment 110). In one example implementation, the pose estimation module 308 includes a VIO system that estimates the pose of the AR / VR display device 106 based on a 3D map of feature points from a current image captured by the optical sensor 212 and inertial sensor data captured by the inertial sensor 210.

[0061] In one example implementation, the pose estimation module 308 calculates the position and orientation of the AR / VR display device 106. The AR / VR display device 106 includes one or more optical sensors 212 mounted on a rigid platform (the frame of the AR / VR display device 106) along with one or more inertial sensors 210. The optical sensors 212 may be mounted with non-overlapping (distributed aperture) or overlapping (stereo or more) fields of view.

[0062] In some example implementations, the pose estimation module 308 includes an algorithm that combines inertial information from the inertial sensor 210 and image information from the pose estimation module 308, wherein the inertial sensor 210 and the pose estimation module 308 are coupled to a rigid platform (e.g., AR / VR display device 106) or a camera rig. In one implementation, the camera rig may consist of multiple camera devices mounted on the rigid platform together with an inertial navigation unit (e.g., inertial sensor 210). Therefore, the camera rig may have at least one inertial navigation unit and at least one camera device.

[0063] Figure 4 This is a block diagram illustrating a blur mitigation module 306 according to an example implementation. The blur mitigation module 306 includes a motion blur estimation module 402, a scale change estimation module 404, an optimal scale calculation module 406, a pyramid calculation engine 408, and a feature matching module 410.

[0064] Motion blur estimation module 402 estimates the level of motion blur for the image from optical sensor 212. In one example implementation, motion blur estimation module 402 estimates motion blur based on camera operating parameters (obtained from optical sensor module 304) and the angular velocity of inertial sensor 210 (obtained from inertial sensor module 302). Motion blur estimation module 402 retrieves camera operating parameters of optical sensor 212 from optical sensor module 304. For example, camera operating parameters include settings of optical sensor module 304 during the capture / exposure time of the current image. Motion blur estimation module 402 also retrieves inertial sensor data (wherein the inertial sensor data is generated during the capture / exposure time of the current image) from inertial sensor 210. Motion blur estimation module 402 retrieves the angular velocity from the IMU of inertial sensor module 302. In one example, motion blur estimation module 402 samples the angular velocity of inertial sensor 210 based on inertial sensor data sampled during the exposure time of the current image. In another example, motion blur estimation module 402 identifies the maximum angular velocity of inertial sensor 210 based on inertial sensor data captured during the exposure time of the current image.

[0065] In another example implementation, the motion blur estimation module 402 estimates motion blur based on the camera device operating parameters, angular velocity, and linear velocity of the vision tracking system 108. The motion blur estimation module 402 retrieves the angular velocity from the VIO data (from the pose estimation module 308). The motion blur estimation module 402 retrieves the linear velocity (from the VIO data) and estimates its impact on motion blur in various regions of the current image based on the 3D positions of feature points in the current image. As previously described above, objects depicted closer to the optical sensor 212 are shown as more blurred, while objects depicted farther from the optical sensor 212 are shown as less blurred. The pose estimation module 308 tracks the 3D positions of feature points and calculates the impact of the calculated linear velocities on various parts of the current image.

[0066] The scale change estimation module 404 estimates the scale change between the source image and the target image by tracking the 3D positions of feature points provided by the pose estimation module 308. For example, the positional changes of the matched feature points (in the source image and the target image) can indicate whether the optical sensor 212 is moving closer to or further away from the scene.

[0067] The optimal scale calculation module 406 determines the optimal scale level for the pyramid calculation engine 408 based on the estimated motion blur and scale change. Figures 8 to 12 Examples of different scenarios for the operation of the optimal scale calculation module 406 are shown.

[0068] exist Figure 8In this process, motion blur estimation module 402 estimates that there is no motion blur in either the source image or the target image. Scale change estimation module 404 estimates that the scale between the source target and the target image has not changed. Therefore, optimal scale calculation module 406 determines that the optimal scale level for both the source and target images is kept at the lowest level (e.g., full resolution), and therefore no downscaling is performed in this case.

[0069] exist Figure 9 In the process, motion blur estimation module 402 detects motion blur in both the source image and the target image. Scale change estimation module 404 estimates that the scale between the source target and the target image has not changed. Therefore, optimal scale calculation module 406 determines that the optimal scale level for both the source image and the target image is the first scale level.

[0070] exist Figure 10 In the process, motion blur estimation module 402 detects that there is no motion blur in either the source image or the target image. Scale change estimation module 404 estimates that the scale between the source target and the image target has changed because feature points in the target image are far away from feature points in the source image. Therefore, optimal scale calculation module 406 determines that the source optimal scale level for the source image is increased to the first scale level, while the target optimal scale level for the target image remains at full resolution.

[0071] exist Figure 11 In the process, motion blur estimation module 402 detects that there is no motion blur in either the source image or the target image. Scale change estimation module 404 estimates that the scale between the source target and the target image has changed because feature points in the source image are far from feature points in the target image. Therefore, optimal scale calculation module 406 determines that the source optimal scale level for the source image remains at full resolution, while the target optimal scale level for the target image is increased to the first scale level.

[0072] exist Figure 12 In the process, motion blur estimation module 402 detects motion blur in both the source image and the target image. Scale change estimation module 404 estimates that the scale between the source target and the target image has changed because feature points in the source image are farther away from feature points in the target image. Therefore, optimal scale calculation module 406 determines that the source optimal scale level for the source image is increased to a first scale level, while the target optimal scale level for the target image is increased to a second scale level.

[0073] The pyramid computing engine 408 applies the image pyramid algorithm to the source image at the source-optimal scale level to generate a downscaled version of the source image. The pyramid computing engine 408 applies the image pyramid algorithm to the target image at the target-optimal scale level to generate a downscaled version of the target image. In other examples where the optimal scale level corresponds to the full resolution of the image, the pyramid computing engine 408 does not apply the image pyramid algorithm to the image.

[0074] The feature matching module 410 matches features between a downscaled version of the source image and a downscaled version of the target image based on the corresponding optimal scale level determined by the optimal scale calculation module 406. In one example, the feature matching module 410 matches features between a full-resolution version of the source image and a downscaled version of the target image. In another example, the feature matching module 410 matches features between a downscaled version of the source image and a full-resolution version of the target image.

[0075] Figure 5 This is a block diagram illustrating an example process according to an example implementation. Visual tracking system 108 receives sensor data from sensor 202 to determine the pose of visual tracking system 108. Blur mitigation module 306 estimates motion blur of the source and target images based on sensor data (e.g., angular velocity from IMU or VIO, linear velocity from VIO data from pose estimation module 308) and camera device operating parameters associated with the source and target images (e.g., exposure time, field of view, resolution). Blur mitigation module 306 also estimates scale variation between the source and target images using VIO data (e.g., 3D points, pose) provided by pose estimation module 308. Blur mitigation module 306 identifies the source-optimal scale level for pyramid calculation engine 408 for the source image based on the motion blur of the source image and the scale variation between the source and target images. Blur mitigation module 306 identifies the target-optimal scale level for pyramid calculation engine 408 for the target image based on the motion blur of the target image and the scale variation between the source and target images.

[0076] The pyramid computing engine 408 applies the image pyramid algorithm to the source image to downscale it at the source-optimal scale level. The pyramid computing engine 408 also applies the image pyramid algorithm to the target image to downscale it at the target-optimal scale level. The pyramid computing engine 408 provides the pose estimation module 308 with downscaled / full versions of the source and target images.

[0077] The pose estimation module 308 identifies the pose of the visual tracking system 108 based on full-resolution or downscaled images provided by the pyramid computing engine 408. The pose estimation module 308 provides pose data to the AR / VR application 214.

[0078] AR / VR application 214 retrieves virtual content 216 from storage device 208 and displays the virtual content 216 at a location (in display 204) based on the pose of AR / VR display device 106. Note that the pose of AR / VR display device 106 is also referred to as the pose of visual tracking system 108 or optical sensor 212.

[0079] Figure 6 This is a flowchart illustrating a method 600 for mitigating motion blur according to an example implementation. The operations in method 600 can be performed by a vision tracking system 108 as described above. Figure 4 The described components (e.g., modules, engines) are used to perform this operation. Therefore, method 600 is described by way of example with reference to ambiguity mitigation module 306. However, it should be understood that at least some operations of method 600 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0080] In block 602, motion blur estimation module 402 estimates the source motion blur level in the source image. In block 604, motion blur estimation module 402 estimates the target motion blur level in the target image. In block 606, scale change estimation module 404 identifies the scale change between the source and target images. In block 608, optimal scale calculation module 406 determines the source optimal scale level for the source image based on the source motion blur level and scale change. In block 610, optimal scale calculation module 406 determines the target optimal scale level for the target image based on the target motion blur level and scale change. In block 612, optimal scale calculation module 406 determines the selected scale level based on the maximum value between the source optimal scale level and the target optimal scale level. In block 614, pyramid calculation engine 408 updates the source optimal scale level and the target optimal scale level based on the selected scale level. Method 600 continues to block A 616.

[0081] It should be noted that other implementations may use different sequencing, additional or fewer operations, and different nomenclature or terminology to accomplish similar functionality. In some implementations, various operations may be executed in parallel with other operations in a synchronous or asynchronous manner. The operations described herein have been chosen to illustrate some operational principles in a simplified form.

[0082] Figure 7 This is a flowchart illustrating a method 700 for mitigating motion blur according to an example implementation. The operations in method 600 can be performed by a visual tracking system 108 as described above. Figure 4The described components (e.g., modules, engines) are used to perform this operation. Therefore, method 600 is described by way of example with reference to ambiguity mitigation module 306. However, it should be understood that at least some operations of method 600 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0083] Method 700 continues from Method 600 at block A 616. In block 702, the pyramid computing engine 408 downscales the source image at the source-optimal scale level. In block 704, the pyramid computing engine 408 downscales the target image at the target-optimal scale level. In block 706, the feature matching module 410 identifies source features in the downscaled source image. In block 708, the feature matching module 410 identifies target features in the downscaled target image. In block 710, the feature matching module 410 matches the source features with the target features. In block 712, the pose estimation module 308 determines the pose based on the matched features.

[0084] Figure 8 An example of a first scenario of the subject matter according to one implementation is shown.

[0085] Figure 9 An example of a second scenario of the subject matter according to one implementation is shown.

[0086] Figure 10 An example of a third scenario of the subject matter according to one implementation is shown.

[0087] Figure 11 An example of a fourth scenario of the subject matter according to one implementation is shown.

[0088] Figure 12 An example of a fifth scenario of the subject matter according to one implementation is shown.

[0089] Figure 13 An example of pseudocode for motion blur mitigation according to one implementation is shown.

[0090] Figure 14 An example of an algorithm for motion blur reduction according to one embodiment is shown.

[0091] Systems with head-mounted devices

[0092] Figure 15 A network environment 1500 in which a head-mounted device 1502 can be implemented is shown according to an example embodiment. Figure 15 This is a high-level functional block diagram of an exemplary head-mounted device 1502 that is communicatively coupled to a mobile client device 1538 and a server system 1532 via various networks 1540.

[0093] The head-mounted device 1502 includes at least one of a visible light camera 1512, an infrared emitter 1514, and an infrared camera 1516. A client device 1538 may be able to connect to the head-mounted device 1502 using both communication 1534 and communication 1536. The client device 1538 connects to a server system 1532 and a network 1540. The network 1540 may include any combination of wired and wireless connections.

[0094] The head-mounted device 1502 also includes two image displays of the optical component image display 1504. These two image displays include one image display associated with the left side of the head-mounted device 1502 and one image display associated with the right side. The head-mounted device 1502 also includes an image display driver 1508, an image processor 1510, a low-power circuitry system 1526, and a high-speed circuitry system 1518. The optical component image display 1504 is used to present images and videos to a user of the head-mounted device 1102, including images that may include a graphical user interface.

[0095] The image display driver 1508 commands and controls the image display 1504 of the optical component to display images. The image display driver 1508 can directly transmit image data to the image display 1504 of the optical component for presentation, or it must convert the image data into a signal or data format suitable for transmission to the image display device. For example, the image data can be video data formatted according to compression formats such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., and still image data can be formatted according to compression formats such as Portable Network Group (PNG), Joint Photographic Experts Group (JPEG), Tag Image File Format (TIFF), or Exchangeable Image File Format (Exif).

[0096] As described above, the head-mounted device 1502 includes a frame and stems (or temples) extending from the sides of the frame. The head-mounted device 1502 also includes a user input device 1506 (e.g., a touch sensor or button), which includes an input surface on the head-mounted device 1502. The user input device 1506 (e.g., a touch sensor or button) receives input selections from the user to manipulate a graphical user interface displaying a presented image.

[0097] Figure 15The components shown for the head-mounted device 1502 are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the frame or temples. Alternatively or additionally, the depicted components may be located in blocks, frames, hinges, or beams of the head-mounted device 1502. The left and right sides may include digital camera elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, camera lenses, or any other corresponding visible light or light-capturing elements that can be used to capture data including images of a scene with unknown objects.

[0098] The head-mounted device 1502 includes a memory 1522 that stores instructions to perform a subset or all of the functions described herein. The memory 1522 may also include a storage device.

[0099] like Figure 15 As shown, the high-speed circuit system 1518 includes a high-speed processor 1520, a memory 1522, and a high-speed wireless circuit system 1524. In this example, an image display driver 1508 is coupled to the high-speed circuit system 1518 and operated by the high-speed processor 1520 to drive the left and right image displays of the image display 1504 of the optical components. The high-speed processor 1520 can be any processor capable of managing the operation of any general computing system required for high-speed communication and the head-mounted device 1502. The high-speed processor 1520 includes the processing resources required to manage the high-speed data transmission over communication 1536 to a wireless local area network (WLAN) using the high-speed wireless circuit system 1524. In some examples, the high-speed processor 1520 executes an operating system such as the LINUX operating system or another such operating system for the head-mounted device 1502, and this operating system is stored in the memory 1522 for execution. Among other duties, the high-speed processor 1520, which executes the software architecture of the head-mounted device 1502, is also used to manage data transmission with the high-speed wireless circuit system 1524. In some examples, the high-speed wireless circuit system 1524 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards can be implemented using the high-speed wireless circuit system 1524.

[0100] The low-power wireless circuitry system 1530 and high-speed wireless circuitry system 1524 of the headset 1502 may include a short-range transceiver (Bluetooth™) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or WiFi). A client device 1538, including transceivers communicating via communications 1534 and 1536, can be implemented using details of the architecture of the headset 1502, as can other elements of the network 1540.

[0101] Memory 1522 includes any storage device capable of storing various data and applications, including, in particular, camera data generated by the left and right infrared cameras 1516 and image processor 1510, and images generated by image display driver 1508 on image display 1504 of optical components for display. While memory 1522 is shown integrated with high-speed circuitry 1518, in other examples, memory 1522 may be a separate component of head-mounted device 1502. In some such examples, circuitry can be provided by lines connecting the image processor 1510 or low-power processor 1528 to memory 1522 via a chip including high-speed processor 1520. In other examples, high-speed processor 1520 may manage addressing of memory 1522 such that low-power processor 1528 will initiate high-speed processor 1522 whenever a read or write operation involving memory 1522 is required.

[0102] like Figure 15 As shown, the low-power processor 1528 or high-speed processor 1520 of the head-mounted device 1502 may be coupled to a camera device (visible light camera 1512; infrared emitter 1514 or infrared camera 1516), an image display driver 1508, a user input device 1506 (e.g., a touch sensor or button) and a memory 1522.

[0103] The head-mounted device 1502 is connected to a host computer. For example, the head-mounted device 1502 is paired with a client device 1538 via communication 1536, or connected to a server system 1532 via a network 1540. The server system 1532 may be one or more computing devices as part of a service or network computing system, and for example, it includes a processor, memory, and network communication interfaces to communicate with the client device 1538 and the head-mounted device 1502 via the network 1540.

[0104] Client device 1538 includes a processor and a network communication interface coupled to the processor. The network communication interface enables communication via network 1540, communication 1534, or communication 1536. Client device 1538 may also store at least a portion of the instructions for generating dual-channel audio content in the memory of client device 1538 to implement the functions described herein.

[0105] The output components of the head-mounted device 1502 include visual components, such as displays like liquid crystal displays (LCDs), plasma display panels (PDPs), light-emitting diode (LED) displays, projectors, or waveguides. The image display of the optical components is driven by an image display driver 1508. The output components of the head-mounted device 1502 also include acoustic components (e.g., speakers), haptic components (e.g., vibration motors), other signal generators, etc. The input components (e.g., user input devices 1506) of the head-mounted device 1502, client device 1538, and server system 1532 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide the position and force of a touch or touch gesture), audio input components (e.g., microphones), etc.

[0106] The head-mounted device 1502 may optionally include additional peripheral device elements. Such peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with the head-mounted device 1502. For example, peripheral device elements may include any I / O components, including output components, motion components, positioning components, or any other such components described herein.

[0107] For example, biometric components include those that detect expressions (e.g., hand gestures, facial expressions, voice expressions, body posture, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identify people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion components include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Positioning components include position sensor components (e.g., GPS receiver components) for generating position coordinates, WiFi or Bluetooth™ transceivers for generating positioning system coordinates, altitude sensor components (e.g., altimeters or barometers that detect air pressure to obtain altitude), orientation sensor components (e.g., magnetometers), etc. Such positioning system coordinates can also be received from client device 1538 via communication 1536 through low-power wireless circuit system 1530 or high-speed wireless circuit system 1524.

[0108] When phrases such as “at least one of A, B or C”, “at least one of A, B and C”, “one or more of A, B, or C” or “one or more of A, B and C” are used, it is intended that the phrase be interpreted as meaning that A may exist alone in an embodiment, B may exist alone in an embodiment, C may exist alone in an embodiment, or any combination of elements A, B and C may exist in a single embodiment; for example, A and B, A and C, B and C, or A and B and C.

[0109] Changes and modifications may be made to the disclosed embodiments without departing from the scope of this disclosure. These and other changes or modifications are intended to be included within the scope of this disclosure as set forth in the appended claims.

[0110] Figure 16 This is a block diagram 1600 illustrating a software architecture 1604 that can be installed on any one or more of the devices described herein. The software architecture 1604 is supported by hardware such as machine 1602, which includes a processor 1620, memory 1626, and I / O components 1638. In this example, the software architecture 1604 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1604 includes layers such as an operating system 1612, libraries 1610, frameworks 1608, and applications 1606. Operationally, application 1606 invokes API call 1650 through the software stack and receives message 1652 in response to API call 1650.

[0111] Operating system 1612 manages hardware resources and provides public services. Operating system 1612 includes, for example, a kernel 1614, services 1616, and drivers 1622. Kernel 1614 serves as an abstraction layer between hardware and other software layers. For example, kernel 1614 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1616 can provide other public services to other software layers. Drivers 1622 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1622 may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® low-power drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, etc.

[0112] Library 1610 provides low-level public infrastructure used by application 1606. Library 1610 may include system library 1618 (e.g., the C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Furthermore, library 1610 may include API library 1624, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., OpenGL frameworks for rendering graphic content on a display in two-dimensional (2D) and three-dimensional (3D) formats), database libraries (e.g., SQLite for providing various relational database functions), web libraries (e.g., WebKit for providing web browsing capabilities), etc. Library 1610 may also include a wide variety of other libraries 1628 to provide many other APIs to application 1606.

[0113] Framework 1608 provides high-level common infrastructure for use by Application 1606. For example, Framework 1608 provides various graphical user interface (GUI) functions, high-level resource management, and high-level location services. Framework 1608 can provide a wide range of other APIs that can be used by Application 1606, some of which may be specific to a particular operating system or platform.

[0114] In an example implementation, application 1606 may include home application 1636, contacts application 1630, browser application 1632, book reader application 1634, location application 1642, media application 1644, messaging application 1646, game application 1648, and a wide variety of other applications such as third-party application 1640. Application 1606 is a program that performs the functions defined in the program. One or more applications 1606 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 1640 (e.g., an application developed by an entity other than a platform vendor using the Android™ or iOS™ Software Development Kit (SDK)) may be mobile software running on mobile operating systems such as iOS™, Android™, Windows® Phone, or other mobile operating systems. In this example, a third-party application 1640 can invoke API call 1650 provided by the operating system 1612 to facilitate the functionality described herein.

[0115] Figure 17 This is a schematic representation of machine 1700, within which instructions 1708 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 1700 to perform any or more of the methods discussed herein. For example, instructions 1708 can cause machine 1700 to perform any or more of the methods described herein. Instructions 1708 transform the general, unprogrammed machine 1700 into a specific machine 1700 programmed to perform the described and illustrated functions in the described manner. Machine 1700 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 1700 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1700 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 1708 specifying actions to be taken by machine 1700. Furthermore, although only a single machine 1700 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 1708 to perform any or more of the methods discussed herein.

[0116] Machine 1700 may include processor 1702, memory 1704, and I / O components 1742 that can be configured to communicate with each other via bus 1744. In an example implementation, processor 1702 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, processor 1706 and processor 1710 that execute instruction 1708. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 17Multiple processors 1702 are shown, but machine 1700 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0117] Memory 1704 includes main memory 1712, static memory 1714, and storage cells 1716, all of which are accessible by processor 1702 via bus 1744. Main memory 1712, static memory 1714, and storage cells 1716 store instructions 1708 embodying any one or more of the methods or functions described herein. Instructions 1708 may also reside wholly or partially in main memory 1712, in static memory 1714, in machine-readable medium 1718 within storage cell 1716, within at least one processor in processor 1702 (e.g., within the processor's cache memory), or in any suitable combination thereof during execution by machine 1700.

[0118] I / O component 1742 may include a wide variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O component 1742 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine is unlikely to include such a touch input device. It should be understood that I / O component 1742 may be included in... Figure 17 Many other components are not shown. In various example embodiments, I / O component 1742 may include output component 1728 and input component 1730. Output component 1728 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input component 1730 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide the position and / or force of a touch or touch gesture), audio input components (e.g., microphones), etc.

[0119] In other example implementations, I / O component 1742 may include biometric component 1732, motion component 1734, environmental component 1736 or positioning component 1738, and various other components. For example, biometric component 1732 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 1734 includes accelerometer components (e.g., accelerometer), gravity sensor components, rotation sensor components (e.g., gyroscope), etc. Environmental component 1736 includes, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an hearing sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases to ensure safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 1738 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0120] A wide variety of technologies can be used to implement communication. I / O component 1742 also includes communication component 1740, which is operable to couple machine 1700 to network 1720 or device 1722 via couplings 1724 and 1726, respectively. For example, communication component 1740 may include a network interface component or another suitable device to interface with network 1720. In other examples, communication component 1740 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components for providing communication via other modalities. Device 1722 may be another machine or any peripheral device from a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0121] Furthermore, the communication component 1740 may detect identifiers or include components operable to detect identifiers. For example, the communication component 1740 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying audio signals from tags). Additionally, various information can be derived via the communication component 1740, such as location obtained via Internet Protocol (IP) geolocation, location obtained via Wi-Fi® signal triangulation, location obtained by detecting NFC beacon signals that can indicate a specific location, etc.

[0122] Various memories (e.g., memory 1704, main memory 1712, static memory 1714, and / or the memory of processor 1702) and / or storage units 1716 may store one or more sets of instructions and data structures (e.g., software) that implement or are used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 1708) cause various operations to implement the disclosed embodiments when executed by processor 1702.

[0123] Instruction 1708 can be sent or received over network 1720 via a transmission medium using a network interface device (e.g., a network interface component included in communication component 1740) and using any of a plurality of known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instruction 1708 can be sent or received to device 1722 via a transmission medium using coupling 1726 (e.g., peer-to-peer coupling).

[0124] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” refer to the same thing and may be used interchangeably in this disclosure. These terms refer to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Therefore, the terms should be considered to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field-programmable gate arrays (FPGAs), and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term “signal medium” discussed below.

[0125] The terms "transmission medium" and "signal medium" refer to the same thing and may be used interchangeably in this disclosure. The terms "transmission medium" and "signal medium" should be considered to include any intangible medium capable of storing, encoding, or carrying instructions 1416 for execution by machine 1400, and include digital or analog communication signals or other intangible media to facilitate communication of such software. Therefore, the terms "transmission medium" and "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal.

[0126] The terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” mean the same thing and can be used interchangeably in this disclosure. These terms are defined to include both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals.

[0127] Although embodiments have been described with reference to specific example embodiments, it will be apparent that various modifications and changes can be made to these embodiments without departing from the broader scope of this disclosure. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The accompanying drawings, which form a part of this invention, illustrate specific embodiments in which the subject matter can be practiced by way of illustration rather than limitation. The illustrated embodiments have been described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments can be utilized and derived therefrom, allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. Therefore, the specific embodiments should not be construed as restrictive, and the scope of the various embodiments is defined only by the appended claims and the full scope of their equivalents.

[0128] These embodiments of the subject matter of this invention may be referred to herein individually and / or collectively by the term "invention," merely for convenience, and if more than one invention or inventive concept is disclosed, it is not intended to voluntarily limit the scope of this application to any single invention or inventive concept. Therefore, although specific embodiments have been shown and described herein, it should be understood that any arrangement calculated to achieve the same purpose may substitute for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of the various embodiments. Combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art upon review of the foregoing description.

[0129] An abstract of this disclosure is provided to allow the reader to quickly determine the nature of this technical disclosure. The abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, as can be seen in the foregoing detailed description, various features are combined in a single embodiment for the purpose of simplifying this disclosure. This approach of the disclosure should not be construed as reflecting an intention to require more features than expressly stated in each claim. Rather, as reflected in the appended claims, the subject matter of the invention lies in fewer than all features of a single disclosed embodiment. Therefore, the claims are hereby incorporated into the detailed description, wherein each claim is considered an independent, separate embodiment.

[0130] Example

[0131] Example 1 is a method for selective motion blur mitigation in a visual tracking system, comprising: accessing a first image generated by an optical sensor of the visual tracking system; accessing a second image generated by the optical sensor of the visual tracking system, the second image being after the first image; determining a first motion blur level of the first image; determining a second motion blur level of the second image; identifying a scale variation between the first image and the second image; determining a first optimal scale level for the first image based on the first motion blur level and the scale variation; and determining a second optimal scale level for the second image based on the second motion blur level and the scale variation.

[0132] Example 2 includes Example 1, and further includes: using a multi-level downscaling algorithm to downscale the first image at the first optimal scale level to generate a first downscaled image; and using the multi-level downscaling algorithm to downscale the second image at the first optimal scale level to generate a second downscaled image.

[0133] Example 3 includes Example 2, and further includes: identifying a first feature in the first downscaled image; identifying a second feature in the second downscaled image; and matching the first feature with the second feature.

[0134] Example 4 includes Example 1, wherein determining a first optimal scale level for the first image includes: calculating a first matching level based on the first motion blur level; applying the scale change to the first matching level to generate a scaled matching level for the first image; identifying a selected scale level based on the maximum level between the scaled matching level of the first image and a second matching level based on the second motion blur level; and applying the selected scale level to the first optimal scale level for the first image.

[0135] Example 5 includes Example 1, wherein determining a second optimal scale level for the second image includes: calculating a second matching level based on the second motion blur level; identifying a selected scale level based on the maximum level between the scaling matching level of the first image and the second matching level based on the second motion blur level; and applying the selected scale level to the second optimal scale level for the second image.

[0136] Example 6 includes Example 1, and further includes: calculating a first matching level based on the first motion blur level; calculating a second matching level based on the second motion blur level; determining a base matching level based on the maximum value of the first matching level and the second matching level; and adjusting the base matching level based on the scale change.

[0137] Example 7 includes Example 1, wherein determining the first motion blur level includes: identifying first camera device operating parameters of the optical sensor for the first image; and determining a first motion of the optical sensor for the first image, wherein determining the second motion blur level includes: identifying second camera device operating parameters of the optical sensor for the second image; and determining a second motion of the optical sensor for the second image.

[0138] Example 8 includes Example 7, wherein determining a first motion of the optical sensor for the first image includes: retrieving first inertial sensor data from the inertial sensor of the visual tracking system for the first image; and determining a first angular velocity of the visual tracking system based on the first inertial sensor data, wherein the first motion blur level is based on the first camera device operating parameters of the visual tracking system and the first angular velocity, without analyzing the content of the first image; wherein determining a second motion of the optical sensor for the second image includes: retrieving second inertial sensor data from the inertial sensor of the visual tracking system for the second image; and determining a second angular velocity of the visual tracking system based on the second inertial sensor data, wherein the second motion blur level is based on the second camera device operating parameters of the visual tracking system and the second angular velocity, without analyzing the content of the second image.

[0139] Example 9 includes Example 7, wherein determining the first motion of the optical sensor for the first image includes: accessing first VIO data from the VIO system of the vision tracking system, the first VIO data including a first estimated angular velocity of the optical sensor, a first estimated linear velocity of the optical sensor, and the position of feature points in the first image, wherein the first motion blur level is based on the operating parameters of the first camera device and the first VIO data, without analyzing the content of the first image, wherein the first motion blur in different regions of the first image is based on the first estimated angular velocity of the optical sensor, the first estimated linear velocity of the optical sensor, and the position of the feature points in the corresponding different regions of the first image relative to the optical sensor. Position D, wherein determining the first motion of the optical sensor for the first image includes: accessing second VIO data from the VIO system of the vision tracking system, the second VIO data including a second estimated angular velocity of the optical sensor, a second estimated linear velocity of the optical sensor, and the position of feature points in the second image, wherein the second motion blur level is based on the operating parameters of the second camera device and the second VIO data, without analyzing the content of the second image, wherein the second motion blur in different regions of the second image is based on the second estimated angular velocity of the optical sensor, the second estimated linear velocity of the optical sensor, and the 3D position of the feature points in the corresponding different regions of the second image relative to the optical sensor.

[0140] Example 10 includes Example 7, wherein the operating parameters of the first source camera device or the operating parameters of the second source camera device include a combination of the exposure time of the optical sensor, the field of view of the optical sensor, the ISO value of the optical sensor, and the image resolution, wherein the first image includes a source image, and wherein the second image includes a target image.

[0141] Example 11 is a computing device including: a processor; and a memory storing instructions that, when executed by the processor, configure the device to: access a first image generated by an optical sensor of a vision tracking system; access a second image generated by the optical sensor of the vision tracking system, the second image being after the first image; determine a first motion blur level of the first image; determine a second motion blur level of the second image; identify a scale variation between the first image and the second image; determine a first optimal scale level for the first image based on the first motion blur level and the scale variation; and determine a second optimal scale level for the second image based on the second motion blur level and the scale variation.

[0142] Example 12 includes Example 11, wherein the instructions further configure the apparatus to: generate a first downscaled image by downscaling the first image at the first optimal scale level using a multi-level downscaling algorithm; and generate a second downscaled image by downscaling the second image at the first optimal scale level using the multi-level downscaling algorithm.

[0143] Example 13 includes Example 12, wherein the instructions further configure the apparatus to: identify a first feature in the first downscaled image; identify a second feature in the second downscaled image; and match the first feature with the second feature.

[0144] Example 14 includes Example 11, wherein determining a first optimal scale level for the first image includes: calculating a first matching level based on the first motion blur level; applying the scale change to the first matching level to generate a scaled matching level for the first image; identifying a selected scale level based on the maximum level between the scaled matching level of the first image and a second matching level based on the second motion blur level; and applying the selected scale level to the first optimal scale level for the first image.

[0145] Example 15 includes Example 11, wherein determining a second optimal scale level for the second image includes: calculating a second matching level based on the second motion blur level; identifying a selected scale level based on the maximum level between the scaling matching level of the first image and the second matching level based on the second motion blur level; and applying the selected scale level to the second optimal scale level for the second image.

[0146] Example 16 includes Example 11, wherein the instructions further configure the device to: calculate a first matching level based on the first motion blur level; calculate a second matching level based on the second motion blur level; determine a base matching level based on the maximum value of the first matching level and the second matching level; and adjust the base matching level based on the scale change.

[0147] Example 17 includes Example 11, wherein determining the first motion blur level includes: identifying first camera device operating parameters of the optical sensor for the first image; and determining a first motion of the optical sensor for the first image, wherein determining the second motion blur level includes: identifying second camera device operating parameters of the optical sensor for the second image; and determining a second motion of the optical sensor for the second image.

[0148] Example 18 includes Example 17, wherein determining a first motion of the optical sensor for the first image includes: retrieving first inertial sensor data from the inertial sensor of the visual tracking system for the first image; and determining a first angular velocity of the visual tracking system based on the first inertial sensor data, wherein the first motion blur level is based on the first camera device operating parameters of the visual tracking system and the first angular velocity, without analyzing the content of the first image; wherein determining a second motion of the optical sensor for the second image includes: retrieving second inertial sensor data from the inertial sensor of the visual tracking system for the second image; and determining a second angular velocity of the visual tracking system based on the second inertial sensor data, wherein the second motion blur level is based on the second camera device operating parameters of the visual tracking system and the second angular velocity, without analyzing the content of the second image.

[0149] Example 19 includes Example 17, wherein determining a first motion of the optical sensor for the first image includes: accessing first VIO data from the VIO system of the vision tracking system, the first VIO data including a first estimated angular velocity of the optical sensor, a first estimated linear velocity of the optical sensor, and the positions of feature points in the first image, wherein the first motion blur level is based on the operating parameters of the first camera device and the first VIO data, without analyzing the content of the first image, wherein the first motion blur in different regions of the first image is based on the first estimated angular velocity of the optical sensor, the first estimated linear velocity of the optical sensor, and the position of the feature points in the corresponding different regions of the first image relative to the optical sensor. 3D position, wherein determining the first motion of the optical sensor for the first image includes: accessing second VIO data from the VIO system of the vision tracking system, the second VIO data including a second estimated angular velocity of the optical sensor, a second estimated linear velocity of the optical sensor, and the position of feature points in the second image, wherein the second motion blur level is based on the operating parameters of the second camera device and the second VIO data, without analyzing the content of the second image, wherein the second motion blur in different regions of the second image is based on the second estimated angular velocity of the optical sensor, the second estimated linear velocity of the optical sensor, and the 3D position of the feature points in the corresponding different regions of the second image relative to the optical sensor.

[0150] Example 20 is a non-transitory computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform the following operations: access a first image generated by an optical sensor of a vision tracking system; access a second image generated by the optical sensor of the vision tracking system, the second image being after the first image; determine a first motion blur level of the first image; determine a second motion blur level of the second image; identify a scale variation between the first image and the second image; determine a first optimal scale level for the first image based on the first motion blur level and the scale variation; and determine a second optimal image scale level for the second image based on the second motion blur level and the scale variation.

Claims

1. A method comprising: Access a first image and a second image following the first image, the first image and the second image being generated by the optical sensors of the vision tracking system; Determine the first motion blur level of the first image and the second motion blur level of the second image; Identify the scale variation between the first image and the second image; A first matching level is calculated based on the first motion blur level, and a second matching level is calculated based on the second motion blur level; The base matching level is determined based on the maximum value between the first matching level and the second matching level; as well as The basic matching level is adjusted based on the scale change.

2. The method according to claim 1, further comprising: A first optimal scale level for the first image is determined based on the first motion blur level and the scale change. as well as A second optimal scale level for the second image is determined based on the second motion blur level and the scale change.

3. The method according to claim 2, further comprising: The first image is downscaled at the first optimal scale level to generate a first downscaled image; as well as The second image is downscaled using the first optimal scale level to generate a second downscaled image.

4. The method according to claim 3, further comprising: Identify a first feature in the first downscaled image; Identify the second feature in the second downscaled image; as well as The first feature is matched with the second feature.

5. The method according to claim 2, wherein, Determining the first optimal scale level for the first image includes: The scale change is applied to the first matching level to generate a scaled matching level for the first image; The selected scale level is identified based on the maximum level between the scaling matching level of the first image and the second matching level based on the second motion blur level; and The selected scale level is applied to the first optimal scale level for the first image.

6. The method according to claim 2, wherein, Determining the second optimal scale level for the second image includes: The selected scale level is identified based on the maximum level between the scaling matching level of the first image and the second matching level based on the second motion blur level; and The selected scale level is applied to the second optimal scale level for the second image.

7. The method according to claim 1, wherein, Determining the first motion blur level includes: Operating parameters of the first camera device for recognizing the optical sensor in relation to the first image; and The first motion of the optical sensor is determined based on the first image. Determining the second motion blur level includes: Operating parameters of the second camera device for recognizing the optical sensor in relation to the second image; and The second motion of the optical sensor is determined based on the second image.

8. The method according to claim 7, wherein, Determining the first motion of the optical sensor for the first image includes: Retrieve first inertial sensor data from the inertial sensor of the visual tracking system for the first image; and The first angular velocity of the visual tracking system is determined based on the data from the first inertial sensor. The first motion blur level is based on the operating parameters of the first camera device and the first angular velocity of the visual tracking system, without analyzing the content of the first image. Determining the second motion of the optical sensor for the second image includes: Retrieve second inertial sensor data from the inertial sensor of the visual tracking system for the second image; and The second angular velocity of the visual tracking system is determined based on the data from the second inertial sensor. The second motion blur level is based on the operating parameters of the second camera device of the visual tracking system and the second angular velocity, without analyzing the content of the second image.

9. The method according to claim 7, wherein, Determining the first motion of the optical sensor for the first image includes: Access first VIO data from the VIO system of the vision tracking system, the first VIO data including the first estimated angular velocity of the optical sensor, the first estimated linear velocity of the optical sensor, and the position of feature points in the first image. The first motion blur level is based on the operating parameters of the first camera device and the first VIO data, without analyzing the content of the first image. The first motion blur level in different regions of the first image is based on the first estimated angular velocity of the optical sensor, the first estimated linear velocity of the optical sensor, and the 3D position of the feature points in the corresponding different regions of the first image relative to the optical sensor. Determining the first motion of the optical sensor for the first image includes: Access the second VIO data from the VIO system of the vision tracking system, the second VIO data including the second estimated angular velocity of the optical sensor, the second estimated linear velocity of the optical sensor, and the position of feature points in the second image. The second motion blur level is based on the operating parameters of the second camera device and the second VIO data, without analyzing the content of the second image. The second motion blur level in different regions of the second image is based on the second estimated angular velocity of the optical sensor, the second estimated linear velocity of the optical sensor, and the 3D position of the feature points in the corresponding different regions of the second image relative to the optical sensor.

10. The method according to claim 7, wherein, The operating parameters of the first camera device and the operating parameters of the second camera device each include a combination of the exposure time of the optical sensor, the field of view of the optical sensor, the ISO value of the optical sensor, and the image resolution. The first image includes a source image, and the second image includes a target image.

11. A computing device, comprising: One or more processors; as well as A memory storing instructions that, when executed by the one or more processors, configure the device to perform operations including: Access a first image and a second image following the first image, the first image and the second image being generated by the optical sensors of the vision tracking system; Determine the first motion blur level of the first image and the second motion blur level of the second image; Identify the scale variation between the first image and the second image; A first matching level is calculated based on the first motion blur level, and a second matching level is calculated based on the second motion blur level; The basic matching level is determined based on the maximum value between the first matching level and the second matching level; and The basic matching level is adjusted based on the scale change.

12. The computing device according to claim 11, wherein, The operation also includes: A first optimal scale level for the first image is determined based on the first motion blur level and the scale change; and A second optimal scale level for the second image is determined based on the second motion blur level and the scale change.

13. The computing device according to claim 12, wherein, The operation also includes: The first image is downscaled at the first optimal scale level to generate a first downscaled image; and The second image is downscaled using the first optimal scale level to generate a second downscaled image.

14. The computing device according to claim 13, wherein, The operation also includes: Identify the first feature in the first downscaled image; Identify the second feature in the second downscaled image; and The first feature is matched with the second feature.

15. The computing device according to claim 12, wherein, Determining the first optimal scale level for the first image includes: The scale change is applied to the first matching level to generate a scaled matching level for the first image; The selected scale level is identified based on the maximum level between the scaling matching level of the first image and the second matching level based on the second motion blur level; and The selected scale level is applied to the first optimal scale level for the first image.

16. The computing device according to claim 12, wherein, Determining the second optimal scale level for the second image includes: The selected scale level is identified based on the maximum level between the scaling matching level of the first image and the second matching level based on the second motion blur level; and The selected scale level is applied to the second optimal scale level for the second image.

17. The computing device according to claim 11, wherein, Determining the first motion blur level includes: Operating parameters of the first camera device for recognizing the optical sensor in relation to the first image; and The first motion of the optical sensor is determined based on the first image. Determining the second motion blur level includes: Operating parameters of the second camera device for recognizing the optical sensor in relation to the second image; and The second motion of the optical sensor is determined based on the second image.

18. The computing device according to claim 17, wherein, Determining the first motion of the optical sensor for the first image includes: Retrieve first inertial sensor data from the inertial sensor of the visual tracking system for the first image; and The first angular velocity of the visual tracking system is determined based on the data from the first inertial sensor. The first motion blur level is based on the operating parameters of the first camera device and the first angular velocity of the visual tracking system, without analyzing the content of the first image. Determining the second motion of the optical sensor for the second image includes: Retrieve second inertial sensor data from the inertial sensor of the visual tracking system for the second image; and The second angular velocity of the visual tracking system is determined based on the data from the second inertial sensor. The second motion blur level is based on the operating parameters of the second camera device of the visual tracking system and the second angular velocity, without analyzing the content of the second image.

19. The computing device according to claim 17, wherein, Determining the first motion of the optical sensor for the first image includes: Access first VIO data from the VIO system of the vision tracking system, the first VIO data including the first estimated angular velocity of the optical sensor, the first estimated linear velocity of the optical sensor, and the position of feature points in the first image. The first motion blur level is based on the operating parameters of the first camera device and the first VIO data, without analyzing the content of the first image. The first motion blur level in different regions of the first image is based on the first estimated angular velocity of the optical sensor, the first estimated linear velocity of the optical sensor, and the 3D position of the feature points in the corresponding different regions of the first image relative to the optical sensor. Determining the first motion of the optical sensor for the first image includes: Access the second VIO data from the VIO system of the vision tracking system, the second VIO data including the second estimated angular velocity of the optical sensor, the second estimated linear velocity of the optical sensor, and the position of feature points in the second image. The second motion blur level is based on the operating parameters of the second camera device and the second VIO data, without analyzing the content of the second image. The second motion blur level in different regions of the second image is based on the second estimated angular velocity of the optical sensor, the second estimated linear velocity of the optical sensor, and the 3D position of the feature points in the corresponding different regions of the second image relative to the optical sensor.

20. A non-transitory computer-readable storage medium, the computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform an operation, the operation comprising: Access a first image and a second image following the first image, the first image and the second image being generated by the optical sensors of the vision tracking system; Determine the first motion blur level of the first image and the second motion blur level of the second image; Identify the scale variation between the first image and the second image; A first matching level is calculated based on the first motion blur level, and a second matching level is calculated based on the second motion blur level; The base matching level is determined based on the maximum value between the first matching level and the second matching level; as well as The basic matching level is adjusted based on the scale change.

Images