Field calibration of augmented reality devices

Abstract

A method for recalibrating an augmented reality (AR) device includes generating and storing a truth-valued map of a real-world environment when the AR device operates with a high likelihood with accurate factory calibration. During operation of the AR device, new map data is generated for the real-world environment. The new map data is compared to the truth map to detect potential calibration errors. If a calibration error is detected, the recalibration process is performed by determining an optimal path through the real-world environment that allows observation of parameters requiring recalibration. Visual cues are generated to guide a user of the AR device to pass through the optimal path. When the user follows a visual cue, the calibration parameter is iteratively adjusted to eliminate the detected calibration error. The recalibration process may be presented as an interactive game to improve user engagement and provide a reward to accurately follow the guidance.

Description

[0001] Priority requirements

[0002] This application claims the priority benefit of U.S. Patent Application Serial No. 18 / 477,297, filed on September 28, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0003] This application relates to the field of wearable augmented reality (AR) devices, such as AR glasses. More specifically, the subject matter of this application relates to techniques for recalibrating AR devices in the field after detecting potential degradation of the factory calibration of the AR device. Background Technology

[0004] Augmented reality (AR) glasses are cutting-edge technology that overlays digital information onto a user's real-world field of vision. AR glasses project digital content—such as images, videos, or data—into the user's field of vision using a combination of sensors, cameras, and microdisplays. This digital content is seamlessly integrated with the user's physical environment, creating an immersive user experience. AR glasses can recognize and respond to user input (such as voice commands and gestures), and in some cases, recognize and respond to eye movements, allowing for interactive and dynamic digital experiences. Proper calibration of AR glasses is crucial to ensuring accurate alignment of digital content with the real world, thereby enhancing the user's perception and interaction with their surroundings. The fusion of the digital and physical worlds provided by AR glasses offers a unique augmented reality experience, transforming how users interact with information and their environment. Attached Figure Description

[0005] In accompanying drawings that are not necessarily drawn to scale, the same reference numerals may describe similar parts in different views. For ease of identification of any particular element or operation being discussed, one or more of the highest-order digits in the reference numerals indicate the drawing number in which that element was first introduced. Some non-limiting examples are shown in the accompanying drawings:

[0006] Figure 1A This diagram illustrates how various parameters of stereoscopic AR glasses can be calibrated using a known pattern (e.g., a checkerboard pattern) during factory calibration operations.

[0007] Figure 1B This is a diagram illustrating how the size of a user's head can cause a pair of AR glasses to bend, thus affecting the orientation of one or two image sensors.

[0008] Figure 2 This is a flowchart illustrating a method operation that, according to some embodiments, can be performed by a pair of AR glasses, resulting in the activation of a recalibration operation.

[0009] Figure 3 This is a diagram illustrating how a pair of stereoscopic vision AR glasses, consistent with some implementations, performs a combination of map building and localization operations to generate an accurate base map of a specific environment, which will be used as a ground truth map for subsequent recalibration operations.

[0010] Figure 4 This illustrates how a pair of stereoscopic vision AR glasses, according to some embodiments, detects misaligned and inconsistent maps when performing map building and localization operations in an environment for which an accurate map (e.g., a truth map) has previously been generated.

[0011] Figure 5 This is a diagram illustrating how a user wearing AR glasses can be guided through an optimal trajectory and look in the best direction during a recalibration operation consistent with some examples.

[0012] Figure 6 This is a diagram illustrating an AR view that can be presented to a user of AR glasses according to some embodiments, wherein the AR view includes several virtual content items in the form of visual cues or augmentations to guide the user to move along a trajectory and look in directions that are optimal for observing objects in the environment associated with an accurate map (e.g., a truth map).

[0013] Figure 7 This is a block diagram illustrating examples of functional components (e.g., hardware components) of an AR device (e.g., AR glasses) consistent with embodiments of the present invention, which can be used to implement the methods and techniques described herein.

[0014] Figure 8 It is a block diagram illustrating the software architecture that can be installed on any one or more devices described herein. Detailed Implementation

[0015] This document describes methods, systems, and computer program products for performing a recalibration process to recalibrate various parameters of an AR glasses pair after detecting potential degradation of the factory calibration. In the following description, numerous specific details are set forth for illustrative purposes to provide a thorough understanding of various aspects of different embodiments of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without all of these specific details.

[0016] In the factory setup, calibrating the AR glasses 100 is a critical process to ensure that digital content is accurately overlaid onto the real-world field of view. Only with correct camera intrinsics, sensor behavior, and the precise location of components measured and displayed can an accurate and consistent AR experience be achieved. A common method for calibrating AR glasses involves using a known pattern, often referred to as a calibration target, such as a checkerboard image 102. Other examples of known patterns that can be used as calibration targets are dot grids, circular grids, symmetrical pattern shapes, colored patterns, barcodes, etc. Figure 1A An example of factory calibration using a known pattern is shown. The factory calibration process begins with the AR glasses 100 capturing an image of a checkerboard pattern 102 using its image sensor (e.g., a camera device). These images are then analyzed by a factory calibration algorithm. The checkerboard pattern is particularly useful because its regular, high-contrast squares make it easy for the algorithm to identify specific points in the image (e.g., the corners of the squares).

[0017] In the context of calibrating AR devices such as AR glasses, automated or robotic trajectories can be used to manipulate AR glasses using predefined and controlled movement paths followed by a robotic system while carrying the AR device. These trajectories are carefully designed to cover various locations and orientations in the environment, allowing the AR device to capture data from different angles—for example, via images of known patterns. The robotic trajectory is used during the calibration process to collect data from known and controlled poses. By moving the AR device along this trajectory, the calibration system can capture sensor data, images, and other measurements, which are later used to calibrate and fine-tune various parameters of the AR device. Knowing the actual size and layout of the chessboard and the actual distance between the AR glasses and the image sensor as it traverses the known trajectory, the factory calibration algorithm can compare the captured images with known images. Based on these comparisons, various parameters can be set with different initial calibration values.

[0018] Internal calibration parameters, such as focal length, principal point, and lens distortion, are adjusted based on the differences between the captured image and known images. For example, if lines in the chessboard appear curved in the captured image, the lens distortion parameter is adjusted until the lines appear straight. For external calibration parameters (such as the position and orientation of the image sensors), the process involves determining how the image sensors need to be positioned and oriented to capture the chessboard pattern when it appears in the captured image. Because AR glasses use two image sensors for stereo vision, the stereo baseline (e.g., the distance between the cameras) is also calibrated using the chessboard pattern. The image sensors capture images of the chessboard from different angles, and the stereo baseline is adjusted until the perceived depth in the captured image matches the actual depth of the chessboard.

[0019] Once AR glasses have been calibrated in the factory, several factors can affect the integrity of the initial factory calibration. For example, if the AR glasses are dropped, bumped, or otherwise subjected to physical impact, this could alter the alignment or positioning of the image sensor or other components, potentially affecting the integrity of the initial calibration. Over time, normal wear and tear from using the glasses can cause hardware changes. For example, lenses may be scratched, camera mounts may loosen, or electronic components may degrade. These changes can affect both internal and external calibration parameters. Temperature variations can cause the AR glasses' materials to expand or contract, potentially affecting the integrity of calibration parameters. Temperature variations can also affect the performance of electronic components.

[0020] Through examples and such Figure 1B As shown, over time, the size of a user's head can cause AR glasses to bend or warp, potentially affecting both internal and external parameters. The bending of the AR glasses can shift the position and orientation of the image sensor. This can affect how the image sensor perceives the world, potentially leading to inaccurate overlay of digital content onto the real-world field of view, such as... Figure 1B As shown in Figures 104 and 106, for AR glasses using two image sensors for stereo vision, bending of the AR glasses alters the distance between the image sensors (e.g., the stereo baseline). This can negatively impact the AR glasses' ability to accurately perceive depth, which is crucial for placing digital content at the correct distance within the real-world field of view. If bending of the AR glasses causes a lens or image sensor to shift, this can shift the principal point, the point in the image where the optical axis intersects the image plane. This can affect the alignment of digital content with the real-world field of view. Finally, bending of the AR glasses can affect the integrity of the factory calibration of the inertial measurement unit (IMU), leading to inaccurate tracking of movement and orientation by the AR glasses.

[0021] Once the integrity of an AR glasses' factory calibration is compromised, recalibration is typically expected to restore accurate and high-quality performance in areas such as tracking and content rendering. Recalibrating AR glasses in the same manner as in a factory setting presents several challenges. One major issue is the inconvenience and undesirability of sending AR glasses back to the factory for recalibration. This not only interrupts user access to the AR glasses but also incurs additional costs and time delays associated with shipping and processing. Furthermore, factory recalibration may not take into account the unique usage conditions and environmental factors experienced by users in the field, potentially leading to suboptimal calibration results.

[0022] Another method involves having the user print out a known pattern or calibration target (such as the chessboard image 102 shown in Figure 1) and then guiding the user through the calibration process. However, this technique has several drawbacks. First, this method increases friction in the user experience because it requires the user to actively participate in a technical process they may not fully understand. Furthermore, the quality of the printed calibration target can vary significantly depending on the printer used and the requirement for a perfectly flat surface. These factors can lead to inaccurate recalibration, negatively impacting the performance of the AR glasses.

[0023] In another approach, online optimization systems can be used to attempt to calibrate one or more parameters on an instant, for example, during the operation of AR glasses. However, techniques consistent with this approach face several technical challenges. First, these online optimization systems tend to be computationally expensive, which can drain the device's battery and degrade its performance. The quality and observability of the calibration depend heavily on the user's actions, making it difficult to obtain consistent results. Furthermore, these methods typically do not utilize known patterns or calibration targets, but rather rely on objects in a random environment for calibration. This usually requires extensive data collection under ideal conditions, which is not always feasible in real-world use cases.

[0024] This paper describes an improved field-based recalibration technique. Here, recalibration is referred to as a "field-based" calibration technique to distinguish it from the initial factory calibration process. The improved field-based recalibration technique described herein addresses several technical challenges associated with the aforementioned conventional techniques by capturing and storing a truth map associated with the real-world environment when the AR glasses are known to have accurate and reliable factory calibration—for example, shortly after the AR glasses have been factory calibrated. The integrity of the factory calibration can then be periodically and systematically verified as the user continues to use the AR glasses by comparing newly generated map data for the real-world environment with previously captured truth maps for the same real-world environment. A recalibration operation can be activated when a potential degradation of the AR glasses' factory calibration is detected. During the recalibration operation, an optimal path or trajectory is determined, where the optimal trajectory allows for the observation and / or calculation of various calibration parameters, particularly by prompting the user to move or follow a specific path and look in various directions. For example, the user can be prompted to traverse a path through the real-world environment by following visual cues (e.g., virtual content) displayed via the AR glasses' display.

[0025] In some examples, the specific trajectory that the user is prompted to follow can be determined based on the specific calibration parameters that require recalibration, such as based on the nature or type of calibration errors that have been detected. Therefore, the trajectory can be selected as the optimal one that allows observation of the specific parameters that may require recalibration. The trajectory prompted to follow will typically allow the AR glasses to capture images of real-world scenes where a ground truth map exists. Therefore, the ground truth map can be used as a calibration target or a recalibration target. During the recalibration process, the recalibration method utilizes a previously recorded calibration target—such as a ground truth map—generated using the initial factory calibration parameters of the AR glasses, thereby ensuring high-quality recalibration without requiring a physical calibration target or extensive data collection. During the recalibration process and as the user traverses the trajectory guided by visual cues, newly observed map data can be compared with ground truth map data. When differences between the two datasets are determined, one or more calibration parameters can be iteratively adjusted to reduce or eliminate these differences. Further advantages and aspects of the subject matter of the invention will become apparent from the following description of several figures.

[0026] Figure 2 This is a flowchart illustrating the operation of a method 200 that, according to some embodiments, can be performed by a pair of AR glasses, resulting in the activation of a recalibration process. Figure 2 As shown, at method operation 202, when the AR glasses are in a state with a high probability of having accurate factory calibration (e.g., "new" factory calibration), the AR glasses generate a map of the real-world environment. This map is stored in the AR glasses' non-volatile memory 206 and will be used as a truth map 204 to detect potential degradation of one or more calibration parameters in one or more subsequent runs, and to recalibrate one or more parameters.

[0027] Consistent with some examples, the AR glasses are configured to acquire a truth map of the real-world environment over a period of time assuming the AR glasses have accurate factory calibration. Therefore, during this period, truth maps can be captured and stored for several different real-world environments as the user wears and operates the AR glasses.

[0028] Various techniques can be deployed to determine if AR glasses are operating in a state with a high probability of accurate factory calibration. For example, consistent with one example, AR glasses can be configured at the factory with a digital flag or configuration setting in their firmware or software that is initially set to indicate that the AR glasses are new and factory calibrated, and in some cases, to indicate the specific date and time when the initial factory calibration occurred. When the AR glasses are first powered on and used by a user, this flag can be checked, and if verified, a process can be triggered to generate and store a truth map. The flag can then be toggled or reset to indicate that the glasses have been used and the factory calibration has been initially verified. Alternatively, when the AR glasses are first turned on and operated by a user, an activation process can be performed, and as part of the AR glasses' activation, a timestamp indicating the date and time of the AR glasses' first activation can be written to memory. Therefore, when the AR glasses are used during a period of time starting from when the AR glasses were first activated or initially calibrated, the AR glasses can execute a background process to generate a truth map as the AR glasses are used by the user.

[0029] In another example, AR glasses can be configured with a usage log or counter in their software or firmware to track overall usage. When the glasses are factory-calibrated and new, this counter starts from zero. Each time the AR glasses are powered on and used, the counter increments by one, providing a record of how many times the AR glasses have been used. Alternatively, this setting could be a more detailed usage log. This usage log can record not only the number of times the glasses are used, but also the duration of each use, the date and time of use, and possibly other information such as the type of application used or the amount of data processed. In both cases, the counter or usage log can be used to determine whether the glasses are assumed to have accurate factory calibration. For example, if the counter is zero or the usage log is empty, it can be assumed that the AR glasses are new and have accurate factory calibration. Once the glasses are used, the counter or log is updated. At some point, the counter or log data can be compared to a predetermined threshold to determine whether processing should be activated to capture a truth map.

[0030] When AR glasses are new and rarely used, their factory calibration is assumed to be the most accurate. This is because wear and tear and environmental factors can potentially alter AR glasses calibration over time. Therefore, usage logs can be used to determine the optimal time period for capturing one or more ground truth maps. If the usage logs indicate that the AR glasses have been used infrequently or for a short period, it can be assumed that the factory calibration remains substantially intact and accurate. During this period, the device can capture potentially highly accurate ground truth maps, serving as a reliable reference for future observations and recalibration. If the usage logs indicate usage exceeding a predetermined threshold, the assumption of intact factory calibration may no longer hold, and the AR glasses will no longer perform background operations to acquire and store ground truth maps.

[0031] In some cases, the integrity of an AR glasses' factory calibration can be assessed by detecting extreme acceleration events rather than simply relying on duration or activations. AR glasses may include an inertial measurement unit (IMU) or other dedicated accelerometers capable of detecting sudden spikes in acceleration or gravity. These spikes can indicate physical shocks or drops that could interfere with careful factory calibration. As an alternative to or in addition to an IMU, AR glasses may include fine conductive wiring incorporated into the frame, designed to break or separate in response to any meaningful structural deformation. If the glasses bend or are impacted beyond a certain threshold, this conductive wiring will disconnect, serving as a signal that the calibration status may be compromised. By detecting these types of extreme acceleration events, drops, or physical deformations, AR glasses can determine whether they are operating in a state with a high probability of accurate factory calibration.

[0032] Beyond determining when ground truth map generation should cease, detecting extreme acceleration events can also directly initiate recalibration of AR glasses. If acceleration spikes, deformation, or severed wiring indicate potential damage to the glasses, the event itself may automatically trigger the recalibration process. The glasses can proactively recalibrate after experiencing a physical impact that could affect calibration accuracy, rather than waiting for map-building inconsistencies to be detected. This acceleration event-driven recalibration provides a more sensitive and robust mechanism. It ensures that recalibration occurs as quickly as possible after a destructive event, rather than waiting for a large accumulation of map-building errors. AR glasses can automatically recalibrate using visual cues, or prompt the user to initiate recalibration after experiencing a potentially destructive acceleration event.

[0033] Figure 3This diagram illustrates how a pair of stereoscopic AR glasses 300, consistent with some examples, performs a combination of map-building and localization operations to generate an accurate map of a specific environment, which will be used as a ground truth map for subsequent calibration verification and recalibration operations. When a user wears the AR glasses 300 in a new environment 302, the AR glasses 300 generates a map based on the observed images using a process called Simultaneous Localization and Mapping (SLAM). SLAM is a computational problem of building or updating a map of an unknown environment while tracking the position of the AR glasses 300 within that environment. This process begins with the AR glasses 300 capturing images of the surrounding environment 302 through its image sensors. These images are then processed to identify different features, such as corners, edges, or other unique visual elements of various objects depicted within the images. Figure 3 In this context, these features or unique visual elements are depicted as map points 304 in the leftmost tree 306. These map points 304—sometimes referred to as landmarks or visual landmarks—are extracted from the environment and represented in an internal map generated by the AR glasses 300. Map points 304 reflect real-world objects, geometry, markings, etc. The relative positions of these map points 304 are used to create a three-dimensional (“3-D”) spatial map of the environment 302 as the user moves through and observes it. Using the spatial map of the environment, the AR glasses 300 can correctly align and render virtual content or augmentations 308 (e.g., stars) relative to corresponding objects (e.g., a group of trees 310) observed in the image of the real-world environment 302. Therefore, the correct alignment of augmentations 308 often depends on the accurate identification and generation of map points 304.

[0034] As the AR glasses 300 move through their environment, they continue to capture images and identify features (e.g., map point 304). The SLAM algorithm uses this new data to update the internal spatial map and determine the current position and orientation of the AR glasses 300 relative to the map. This process is repeated continuously, allowing the AR glasses 300 to maintain an up-to-date map of the environment and its position within it. When this map of the real-world environment is generated using the “new” factory calibration, it serves as a truth map. The AR glasses 300 can then verify the integrity of its factory calibration and perform a recalibration operation if a calibration error is detected. It should also be noted that, for some examples, this truth map can be generated using background processing, e.g., without requiring explicit user activation of the process. Therefore, the process of generating a truth map can occur when the AR glasses are known to be in a factory calibration state.

[0035] Refer again Figure 2After a period of time 203 has elapsed since the AR glasses were put into use, the AR glasses are assumed to have reached the end of accurate factory calibration. Therefore, after the user has operated the AR glasses and the AR glasses have generated one or more truth maps of the real-world environment, during subsequent operation at the location for which the truth map has been generated, the AR glasses can load an accurate truth map of the specific real-world environment (e.g., the user's current location) into its working memory. This is in... Figure 2 It is shown in the figure with reference numeral 206.

[0036] AR glasses can determine their location in the real-world environment using various methods. Consistent with some examples, AR glasses may include an integrated GPS device for generating location data. In another example, AR glasses may utilize a wirelessly connected mobile computing device, such as a smartphone, with built-in GPS capabilities. The AR glasses can connect to the smartphone via Bluetooth or Wi-Fi, and the smartphone can provide GPS data to the AR glasses. In either case, once the AR glasses have determined their location, they can check their memory or a connected database to see if a ground truth map of that location already exists. If such a map exists, it can be loaded into the AR glasses' working memory.

[0037] At method operation 208, after an accurate ground truth map has been loaded for the real-world environment, the AR glasses perform map optimization or update processing. During this process, new map data can be combined and / or compared with the current ground truth map 204 for the real-world environment. For example, at method operation 210, during map update processing, the AR glasses use the ground truth map 204 as a reference to detect or determine possible parameter calibration errors or inconsistencies. The ground truth map 204, assumed to be an accurate representation of the current real-world environment, can be used as a benchmark against which new map data can be compared. When the AR glasses capture new data and generate a new map, they can compare this new map with the ground truth map. If a significant difference exists between the new map and the ground truth map, this can indicate a calibration error or inconsistency. Consistent with some examples, the nature of the detected difference can also indicate which of several calibration parameters caused the difference and therefore requires recalibration.

[0038] For example, if the locations of certain features or objects in the new map do not match their locations in the ground truth map, this could indicate a calibration problem with the AR device's sensors or cameras. Similarly, if the scale or orientation of the new map does not match the ground truth map, this could indicate a calibration issue. AR devices can use various algorithms and techniques to quantify the differences between the new and ground truth maps and determine the likely sources of these differences. This could involve comparing features and points of interest in the maps, comparing the geometric properties of the maps, or comparing the sensor data used to generate the maps. Figure 4 The image shows an example of an misaligned map point.

[0039] Figure 4 This illustrates how, for a specific real-world environment, misaligned map points associated with objects in a captured image can represent potential calibration errors or discrepancies between the newly generated map and the ground truth map. Figure 4 As shown, the AR glasses 300 have captured the surrounding environment 302 (and... Figure 3 A new image (of the same environment as shown) was generated, and features or points of interest associated with various objects (e.g., tree 406) were identified. However, the locations of these features, as represented by map points in the new map, do not match the locations of the same features in the ground truth map. For example, as... Figure 4 As shown, the map point labeled 402 is associated with the ground truth map, while the corresponding map point labeled 404 is associated with the newly generated map. The positional difference between these two map points suggests a possible calibration error.

[0040] Misalignment between map points indicates that the new map does not accurately represent the geometry and spatial relationships of the real-world environment. This suggests a potential calibration problem with the AR glasses 300, affecting how the device perceives and maps its surroundings. For example, the scale, orientation, or alignment of the new map may be incorrect, indicating potential errors in the calibration of the sensors or camera devices. Differences between maps can be quantified to estimate the magnitude of these potential calibration errors, which can then be addressed by adjusting relevant parameters. By comparing the new map with a ground truth map, calibration problems can be detected and resolved to ensure accurate map building and positioning.

[0041] Besides directly comparing the new map to the ground truth map, calibration errors can be detected by analyzing the reprojection errors that occur when aligning the ground truth map with the AR glasses' current estimated pose. As AR glasses move through their environment, they use visual odometry or other positioning techniques to estimate their new pose. This estimated pose can then be used to reproject the ground truth map onto the current camera image. If the reprojected map misaligns with actual visual features in the current image, exhibiting a large reprojection error, this indicates inaccurate pose estimation and suggests a potential calibration error in the AR glasses' sensors or perception system. By quantifying these reprojection errors and how they change as the user moves, the magnitude of the calibration inaccuracy can be estimated, and the inaccuracy can be addressed by adjusting relevant parameters. Analyzing reprojection errors provides another diagnostic tool for detecting calibration drift, in addition to direct map comparison.

[0042] When calibration errors exist in the sensors or perception system of AR glasses, these errors introduce inaccuracies into map points generated from the raw sensor data. For example, a depth calibration error might cause map points to be positioned at an incorrect depth compared to their true location. Alternatively, a position calibration error could cause the entire set of map points to deviate from their accurate positions. When map points are used as a reference to render augmentations, these inaccuracies in map point positions will ultimately lead to misalignment. For instance, if a map point for a particular object is incorrectly positioned, any augmentations attached to or associated with that map point will also be misaligned with the object's true location. Calibration errors introduce inaccuracies into the spatial map (map points). When these inaccurate map points are then used to render augmentations, this causes virtual content to be misaligned with real-world objects and geometry. Calibration errors propagate through the map-building process and into augmentation alignment. This example is in... Figure 4 The magnified circular view 408 shows the misaligned enhancement 410 compared to the correctly aligned enhancement 412.

[0043] Refer again Figure 2At method operation 212, if no inconsistency is detected between the new map and the ground truth map, the user continues using the AR glasses, where the AR glasses do not take further correction actions. However, once a potential or possible calibration problem is detected, the AR glasses will typically mark the inconsistent calibration parameters as potentially invalid, as shown in method operation 214. The AR glasses can record detected calibration errors in various ways (e.g., using a counter or error log) to determine when recalibration is needed. For example, the AR glasses can maintain a counter that increments each time a possible calibration error is detected. This counter is then compared to a predetermined threshold such that recalibration is performed only after the counter exceeds the threshold. The threshold can be a fixed value, such as 2, 3, 4, or 10 possible errors, or it can be a variable value depending on usage and environmental conditions.

[0044] AR glasses can record details about detected calibration errors in an error log instead of a basic counter. The error log can record information such as the date and time of detection, the type of error (e.g., scale, orientation, location), the magnitude of the error, and the location, real-world environment, or ground truth map. By analyzing trends in this error log, the system can determine when recalibration is needed for specific types of errors or in specific environments. For example, recalibration can be triggered when a certain number of positional errors exceeding a predetermined size are detected within a specific time period (e.g., an hourly period).

[0045] The error log approach allows for more intelligent and customized determination of when recalibration is needed, based on individual user usage patterns and conditions. A combination of counters and error logs can also be used, where minor errors increment the counter but are also logged in detail, and recalibration is performed when the counter exceeds a threshold or when the error log indicates a critical amount for a specific type of error. This hybrid approach balances simplicity and customizability.

[0046] Refer again Figure 2 As indicated by the operation labeled 216 in the attached figure, after each possible calibration error is recorded, the AR glasses perform an evaluation to determine whether a threshold condition has been met, prompting a recalibration operation. Once the AR glasses detect a calibration error that meets or exceeds a predetermined threshold, a flag can be set in the device's software or firmware to indicate that recalibration is required. However, recalibration cannot be performed until the AR glasses are in an environment where an accurate truth map exists. Therefore, consistent with some examples, the flag acts as a prompt that recalibration should be initiated the next time the AR glasses are used in such a location. In at least some examples, because the truth map is used to detect calibration errors, the AR glasses may frequently be in a location associated with the truth map when the threshold condition for performing a recalibration operation is met.

[0047] In any case, the recalibration flag can be a binary value (0 or 1) stored in the AR glasses' memory. This value switches from 0 to 1 when a threshold is met and recalibration is required. The AR glasses continuously check this flag value, and when it is 1, the system begins monitoring the user's location. Once this location matches an environment with a truth map, the recalibration process is triggered. Upon completion of the recalibration operation, the flag is then reset to 0 to indicate that recalibration has been performed.

[0048] Alternatively, a more complex data structure can be used instead of a binary flag to store details about the required recalibration, such as the type of error detected or the parameters that need adjustment. This data structure is accessed when the AR glasses are used in an appropriate location (e.g., where a truth map exists) to determine how to handle the recalibration. For example, if a location error is detected, the recalibration process could focus on adjusting the image sensor calibration. Similar to the binary flag, this data structure is reset once recalibration is complete. Regardless of whether a binary flag or a more complex data structure is used, the AR glasses will perform recalibration at the next opportunity when they are in a location associated with an existing truth map. The flag acts as a reminder, storing the need for recalibration in memory until the AR glasses are able to act on it. Once recalibration is complete, the flag is reset the next time it is needed.

[0049] like Figure 2 As shown by reference numeral 218 in the attached figure, a recalibration process is initiated when the AR glasses enter a location or environment associated with a ground truth map. Consistent with some examples, the first step in the recalibration operation involves determining the optimal path or trajectory through the real-world environment, which will allow observation of the parameters requiring recalibration. In some examples, this path is calculated based on details of detected calibration errors stored in a data structure in the error log. For example, if a positional error is detected, the optimal path could traverse highly textured geometry at different depths and orientations. Similarly, if the error indicates a potential calibration problem with the IMU, the calculated trajectory could involve various orientation variations, including stops and starts.

[0050] Once the optimal path or trajectory is calculated, visual cues are generated to guide the user along that path. These visual cues can include circles, arrows, highlights, or other enhancements overlaid on the real-world view and displayed on the AR glasses. Visual cues guide the user to move in a specific direction, turn their head, or look at a specific object. By following these cues, the user traverses the optimal path, allowing the AR glasses to capture the data needed to recalibrate the necessary parameters.

[0051] As the user moves through the environment, the AR glasses use a ground truth map as a reference to track the glasses' position and orientation. New data, such as images and depth measurements, is captured and compared to the ground truth map. The differences between the data and the map are used to adjust calibration parameters, aiming to eliminate or reduce detected errors. After following visual cues for an optimal path, additional cues can be used to prompt the user for a second pass to verify the recalibration. If the detected errors have been resolved, the flags are reset and normal operation resumes.

[0052] Figure 5 This is a diagram illustrating how a user wearing AR glasses 300 can be guided, via augmentation 502 or visual cues, through a recalibration operation consistent with some examples to traverse the optimal trajectory and look in the optimal direction in the real-world environment 500. (See diagram for reference.) Figure 5 As shown, the AR glasses 300 guides the user through an optimal path for recalibration by displaying visual cues 502 in the form of virtual objects superimposed on the real-world environment 500. These objects (such as a heart 502-A, a moon 502-B, a cloud 502-C, and a smiley face 502-D) prompt the user to approach each object and turn to look in the direction of the next object along the optimal path. By following these visual cues, the user traverses a customized trajectory 506, which enables the AR glasses 300 to capture target data for recalibrating specific parameters.

[0053] As the user moves through the environment along the optimal path 506, the AR glasses 300 use a ground truth map as a reference to track the movement and orientation of the AR glasses 300. With each movement or turn of the head, new map data is captured by the image sensor and other sensors. This new data is compared with the ground truth map to detect any discrepancies or misalignments. For example, if the user turns to look at a tree 504 after being prompted to look at the direction of the virtual object 402, and if map points associated with various features of the tree appear at a different location or orientation than those recorded in the ground truth map, this indicates a potential calibration problem. These discrepancies can then be used to adjust the necessary calibration parameters to correct the alignment.

[0054] The process is iterative, where the user continues along path 506, guided by visual cues 502, as the system makes incremental adjustments to the calibration. Once the user reaches the end of the path, the AR glasses can prompt the user to repeat the path to verify the recalibration. If additional discrepancies are detected, further adjustments are made. This loop can be repeated until the new data closely matches the ground truth map, indicating that the calibration error has been resolved.

[0055] Figure 6An example of an AR view 600 that can be displayed to a user during AR glasses recalibration is shown. In AR view 600, the user sees the real-world environment normally through the AR glasses. However, superimposed on the environment are virtual visual cues in the form of circles 602-A, 602-B, and 602-C, highlighting specific virtual objects such as a cloud 604 and a smiley face 606. These visual cues guide the user toward the surrounded objects and turn to look at them, thus guiding the user through the optimal path for recalibration. By following the visual cues, the user is able to capture target data represented in the truth map for adjusting calibration parameters that need to be recalibrated.

[0056] Visual cues overlaid on the AR View 600 provide an intuitive interface for the recalibration process. Instead of providing technical instructions for data capture, the circles visually guide the user through necessary actions and directions of observation. The user simply walks towards the highlighted object and turns to look in the indicated direction. In some examples, auditory navigation cues can be played, for instance, through the AR glasses' speakers to enhance the guidance presented to the user. In the background, the system tracks the user's trajectory through the environment and captures new map data from image sensors and other sensors. This data is compared to a ground truth map to identify discrepancies and adjust calibration parameters. By presenting visual cues and specific objects in the AR View, the system guides the user to efficiently capture data for recalibrating necessary parameters with minimal input. The AR View 600 with overlaid visual cues provides an attractive interface for recalibrating AR glasses.

[0057] To make the recalibration process more engaging for users, AR glasses can present the action as an interactive game. As the user follows visual cues and reaches each highlighted object, animations or visual rewards can play to indicate that the cue has been followed correctly. For example, when the user turns to look at a smiley face object, the smiley face can spin, change color, or animate. When the user walks to a cloud object, it can release virtual rain or change shape. These animations provide positive feedback, rewarding the user for accurately following the guidance.

[0058] As users navigate the recalibration path, the game also tracks points, or scores. The score increases with each triggered animation. Music or sounds may play as the score rises, creating an exciting and meaningful experience. No animations or points are awarded if the user fails to follow visual cues or hints. Scores and any unlocked achievements can be displayed on the AR view to motivate the user. By presenting the recalibration process as an interactive game, the user experience becomes more engaging and fun. The game encourages users to accurately follow visual cues and guidance so they can achieve higher scores and unlock more achievements and rewards. This helps ensure that the necessary data for recalibrating AR glasses is captured correctly.

[0059] In summary, gamifying the recalibration process helps create an engaging user experience. Visual rewards, points, music, and sound can be used to motivate users to follow guidance provided by overlaid visual cues. Failure to follow cues results in no reward or points, thus encouraging users to accurately capture the data needed for recalibration. By making the experience interactive and gamified, the recalibration process becomes an entertaining activity rather than a tedious technical process, leading to a better overall AR glasses experience. Gamifying the recalibration process is one way to increase user engagement and satisfaction.

[0060] In several examples presented herein, the ground truth map is described as having been captured or created by the AR device on which recalibration will take place. In various alternative implementations, the ground truth map used for detecting calibration errors and recalibration does not need to be generated by the AR glasses themselves. Instead of capturing and storing its own ground truth map when calibration is assumed to be accurate, the AR glasses can obtain reliable ground truth maps or data from other sources. For example, if operating in a location where a highly accurate, publicly available map exists (e.g., a park or a well-known landmark), the AR glasses can download the public map to use as ground truth. Alternatively, the AR glasses can connect to other trusted user devices (including another AR device or a shared server) and obtain ground truth maps captured by others when their AR devices are recalibrated. By crowdsourcing ground truth maps from other calibration sources, the AR glasses can save the effort of mapping the environment itself before recalibration and / or perform recalibration, for example, to capture a ground truth map, in locations where the AR glasses have not previously operated. As long as the acquired ground truth map meets an accuracy threshold, contains sufficient spatial features, and covers the desired environment, it can be used as valid recalibration data even if generated on a different device.

[0061] Example Augmented Reality (AR) Devices

[0062] Figure 7This is a block diagram illustrating examples of functional components (e.g., hardware components) of an AR device (e.g., AR glasses 200) consistent with embodiments of the present invention, utilizing which the methods and techniques described herein can be implemented. Those skilled in the art will readily understand that... Figure 7 The AR glasses 200 depicted are merely one example of many different devices to which the subject matter of this invention can be applied. For example, embodiments of the invention are not limited to AR glasses, but are also applicable to AR headsets and other wearable virtual reality and mixed reality devices.

[0063] AR glasses 200 include a data processor 702, a display 710, two or more image sensors 708, and additional input / output elements 716. The input / output elements 716 may include a microphone, audio speaker, biometric sensor, additional sensor, or additional display element integrated with the data processor 702. For example, the input / output elements 716 may include any of the I / O components containing moving parts, etc.

[0064] Consistent with one example and as described herein, display 710 includes a first sub-display for the user's left eye and a second sub-display for the user's right eye. Therefore, although referred to in the singular (display), in some examples, the display may include two separate displays operating together. Each display of AR glasses 200 may include: a forward-facing optical assembly (not shown) comprising a right projector and a right near-eye display; and a forward-facing optical assembly comprising a left projector and a left near-eye display. In some examples, the near-eye display is a waveguide. The waveguide includes a reflective or diffractive structure (e.g., a grating and / or optical elements such as mirrors, lenses, or prisms). Light emitted by the right projector encounters the diffractive structure of the waveguide of the right near-eye display, which directs the light toward the user's right eye to provide an image on or within the right optical element, superimposed with a view of the real world seen by the user. Similarly, light emitted by the left projector encounters the diffraction structure of the waveguide of the left near-eye display, which directs the light toward the user's left eye to provide an image on or in the left optics, which is superimposed on the view of the real world seen by the user.

[0065] The data processor 702 includes an image processor 706 (e.g., a video processor), a graphics processing unit (GPU), a display driver 748, a tracking processor 740, an interface 712, a low-power circuit system 704, and a high-speed circuit system 720. The components of the data processor 702 are interconnected via a bus 742.

[0066] Interface 712 refers to any source of user commands provided as input to data processor 702. In one or more examples, interface 712 is a physical button that, when pressed, sends a user input signal from interface 712 to low-power processor 714. Low-power processor 714 may process pressing such a button followed immediately by release as a request to capture a single image, or vice versa. Low-power processor 714 may process pressing such a button for a first time period as a request to capture video data while the button is pressed and to stop video capture when the button is released, wherein the video captured while the button is pressed is stored as a single video file. Alternatively, pressing the button for an extended time period may capture a still image. In some examples, interface 712 may be any mechanical switch or physical interface capable of accepting and detecting user input associated with data requests from image sensor 708. In other examples, interface 712 may have software components or may be associated with commands received wirelessly from another source (e.g., from client device 728).

[0067] Image processor 706 includes circuitry for receiving signals from image sensor 708 and processing those signals from image sensor 708 into a format suitable for storage in memory 724 or for transmission to client device 728. In one or more examples, image processor 706 (e.g., video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from image sensor 708, and volatile memory used by the microprocessor in operation.

[0068] The low-power circuit system 704 includes a low-power processor 714 and a low-power wireless circuit system 718. These components of the low-power circuit system 704 can be implemented as separate components or as part of a single-chip system on a single IC. The low-power processor 714 includes logic for managing other components of the AR glasses 200. As described above, for example, the low-power processor 714 can accept user input signals from interface 712. The low-power processor 714 can also be configured to receive input signals or command communications from client device 728 via a low-power wireless connection. The low-power wireless circuit system 718 includes circuit elements for implementing a low-power wireless communication system. Bluetooth™ Smart, also known as Bluetooth™ Low Energy, is a standard implementation of the low-power wireless communication system that can be used to implement the low-power wireless circuit system 718. In other examples, other low-power communication systems may be used.

[0069] The high-speed circuit system 720 includes a high-speed processor 722, a memory 724, and a high-speed wireless circuit system 726. The high-speed processor 722 can be any processor capable of managing high-speed communication and operation for any general-purpose computing system used by the data processor 702. The high-speed processor 722 includes processing resources for managing high-speed data transmission over the high-speed wireless connection 734 using the high-speed wireless circuit system 726. In some examples, the high-speed processor 722 executes an operating system, such as the LINUX operating system or another such operating system. Among other responsibilities, the high-speed processor 722, which executes the software architecture of the data processor 702, manages data transmission with the high-speed wireless circuit system 726. In some examples, the high-speed wireless circuit system 726 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards can be implemented via the high-speed wireless circuit system 726.

[0070] Memory 724 includes any storage device capable of storing camera device data generated by image sensor 708 and image processor 706. While memory 724 is shown as integrated with high-speed circuitry 720, in other examples, memory 724 may be a separate, independent component of data processor 402. In some such examples, electrical wiring may provide a connection from image processor 706 or low-power processor 714 to memory 724 via a chip including high-speed processor 722. In other examples, high-speed processor 722 may manage addressing of memory 724 such that low-power processor 714 will activate high-speed processor 722 whenever a read or write operation involving memory 724 is required.

[0071] The tracking processor 740 estimates the pose of the AR glasses 200. For example, the tracking processor 740 uses image data and corresponding inertial data from the image sensor 708 and the position component, as well as GPS data, to track the position and determine the pose of the AR glasses 200 relative to a reference frame (e.g., a real-world scene). The tracking module 740 continuously collects and uses updated sensor data describing the movement of the AR glasses 200 to determine an updated 3D pose of the AR glasses 200, which indicates changes in the relative position and orientation of the AR glasses 200 with respect to physical objects in the real-world environment. The tracking processor 740 allows the AR glasses 200 to visually position virtual objects relative to physical objects within the user's field of vision via the display 710.

[0072] When the AR glasses 200 are running in traditional AR mode, the GPU and display driver 738 can use the pose of the AR glasses 200 to generate frames of virtual content or other content to be displayed on the display 410. In this mode, the GPU and display driver 738 generate updated frames of virtual content based on the updated 3D pose of the AR glasses 200, which reflects changes in the position and orientation of the user relative to physical objects in the user's real-world environment.

[0073] One or more functions or operations described herein can also be performed on an application residing on AR glasses 200, client device 728, or remote server 730. Consistent with some examples, AR glasses 200 can operate in a networked system comprising AR glasses 200, client computing device 728, and server 730, which can be communicatively coupled via a network. Client device 728 can be a smartphone, tablet, phablet, laptop, access point, or any other such device capable of connecting to AR glasses 200 using low-power wireless and / or high-speed wireless connections. Client device 728 is connected to server system 730 via a network. The network can include any combination of wired and wireless connections. Server 730 can be one or more computing devices as part of a service or network computing system.

[0074] Software Architecture

[0075] Figure 8 This is a block diagram 800 illustrating a software architecture 804 that can be installed on any one or more of the devices described herein. The software architecture 804 is supported by hardware such as a machine 802 including a processor 820, memory 826, and I / O components 838. In this example, the software architecture 804 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 804 includes layers such as an operating system 812, libraries 808, frameworks 810, and applications 806. Operationally, application 806 activates API calls 850 through the software stack and receives messages 852 in response to API calls 850.

[0076] Operating system 812 manages hardware resources and provides public services. Operating system 812 includes, for example, a kernel 814, services 816, and drivers 822. Kernel 814 acts as an abstraction layer between the hardware layer and other software layers. For example, kernel 814 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 816 can provide other public services to other software layers. Drivers 822 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 822 may include display drivers, camera device drivers, BLUETOOTH® or BLUETOOTH® Low Power drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, etc.

[0077] Library 808 provides low-level common infrastructure used by application 806. Library 808 may include system library 818 (e.g., the C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 808 may include API library 824, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering two-dimensional (2D) and three-dimensional (3D) graphical content on a display, GLMotif for implementing 3D user interfaces), image feature extraction libraries (e.g., OpenIMAJ), database libraries (e.g., SQLite for providing various relational database functions), web libraries (e.g., WebKit for providing web browsing functionality), etc. Library 808 may also include various other libraries 828 to provide many other APIs to application 806.

[0078] Framework 810 provides high-level common infrastructure for use by application 806. For example, framework 810 provides various graphical user interface (GUI) functions, high-level resource management, and high-level location services. Framework 810 can provide a wide range of other APIs that can be used by application 806, some of which may be specific to a particular operating system or platform.

[0079] In the example, application 806 may include home application 836, contact application 830, browser application 832, book reader application 834, location application 842, media application 844, messaging application 846, game application 848, and a variety of other applications such as third-party application 840. Application 806 is a program that performs the functions defined in the program. One or more applications 806 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 840 (e.g., an application developed by an entity other than a platform-specific vendor using the Android™ or iOS™ Software Development Kit (SDK)) may be mobile software running on a mobile operating system such as iOS™, Android™, Windows® Phone, or another mobile operating system. In this example, third-party application 840 may activate API calls 850 provided by operating system 812 to facilitate the functions described herein.

[0080] Example

[0081] Example 1 is an augmented reality (AR) device configured to perform a field-based recalibration process. The AR device includes: a display; a processor; two or more image sensors; an inertial measurement unit (IMU); and a memory storing instructions that, when executed by the processor, cause the AR device to perform operations including: generating map data of a real-world environment by processing images obtained from the image sensors and motion data from the IMU; detecting potential calibration errors by comparing the map data of the real-world environment with ground truth map data of the real-world environment (previously generated by the AR device); evaluating rules to determine to perform a recalibration process in response to detecting a potential calibration error; and performing the recalibration process by generating visual cues and displaying them via the display, the visual cues guiding the user of the AR device through a path in the real-world environment during the recalibration process and looking in a specific direction while traversing the path.

[0082] In Example 2, the subject of Example 1 includes the fact that the truth map data of the real-world environment was previously generated by an AR device operating in the real-world environment.

[0083] In Example 3, the subject of Examples 1 to 2 includes receiving, at the AR device, truth map data of the real-world environment captured by another computing device before comparing map data of the real-world environment with truth map data of the real-world environment.

[0084] In Example 4, the subject matter of Examples 1 to 3 includes a memory storing additional instructions thereon that, when executed by a processor, cause the AR device to perform additional operations, the additional operations including: determining, before generating map data of the real-world environment, that the AR device is operating in a state that is highly likely to have accurate factory calibration; and in response to determining that the AR device is operating in this state, i) generating truth map data of the real-world environment, and ii) storing the truth map data of the real-world environment.

[0085] In Example 5, the subject matter of Examples 1 through 4 includes determining that an AR device is likely operating in a state with accurate factory calibration by: evaluating usage logs to determine that the duration for which the AR device has been in active use since it was factory calibrated is less than a threshold; evaluating usage logs to determine that the count of times the AR device has been activated since it was factory calibrated is less than a threshold; evaluating configuration settings to determine that the duration for which the AR device has been activated since it was first activated is less than a threshold; evaluating configuration settings to determine that the elapsed duration for which the AR device has been factory calibrated is less than a threshold; or any combination thereof.

[0086] In Example 6, the subject matter of Examples 1 through 5 includes, wherein the evaluation rule to determine the recalibration process to be performed includes: the evaluation rule to determine the number of times a possible calibration error has been detected that exceeds a threshold.

[0087] In Example 7, the subject matter of Examples 1 through 6 includes the following, wherein detecting possible calibration errors by comparing map data of a real-world environment with ground truth map data of a real-world environment includes: determining that the depth or location of a feature or object in the map data does not match the depth or location of a feature or object in the ground truth map data; determining that the scale or orientation of the map data does not match the scale or orientation of the ground truth map data; or a combination thereof.

[0088] In Example 8, the subject matter of Examples 1 through 7 includes the following: visual cues that prompt a user of an AR device to traverse a path in a real-world environment and look in a particular direction while traversing that path are derived by first determining an optimal path that allows for high observability of one or more parameters to be recalibrated, wherein the one or more parameters to be recalibrated are selected based on the types of possible calibration errors that have been detected.

[0089] In Example 9, the subject of Example 8 includes, wherein performing the recalibration process further includes, during the recalibration process and while the user is traversing a path guided by visual cues displayed via the AR device’s display, iteratively adjusting one or more calibration parameters of the AR device to reduce the difference between the newly generated map data and the ground truth map data.

[0090] In Example 10, the subject matter of Examples 1 through 9 includes, wherein performing the recalibration process further includes: during the recalibration process and while the user is traversing a path guided by visual cues displayed via a display of an AR device, in response to determining that the user has traversed the path in accordance with the displayed visual cues and is looking in various directions while traversing the path, providing visual feedback to the user via the display.

[0091] Example 11 is a method performed by an augmented reality (AR) device, comprising: generating map data of a real-world environment by processing images obtained from an image sensor and motion data from an IMU; detecting possible calibration errors by comparing the map data of the real-world environment with ground truth map data of the real-world environment; evaluating rules to determine to perform a recalibration process in response to detecting possible calibration errors; and performing the recalibration process by generating visual cues and displaying the visual cues via a display, the visual cues guiding the user of the AR device to traverse a path in the real-world environment during the recalibration process and to look in a specific direction while traversing the path.

[0092] In Example 12, the subject of Example 11 includes the fact that truth map data of a real-world environment was previously generated by an AR device operating in a real-world environment.

[0093] In Example 13, the subject of Examples 11 to 12 includes receiving, at the AR device, ground truth map data of the real-world environment captured by another computing device before comparing map data of the real-world environment with ground truth map data of the real-world environment.

[0094] In Example 14, the subjects of Examples 11 through 13 include, before generating map data of a real-world environment: determining that the AR device is operating in a state with a high probability of accurate factory calibration; and in response to determining that the AR device is operating in this state, i) generating truth map data of the real-world environment, and ii) storing the truth map data of the real-world environment.

[0095] In Example 15, the subject matter of Examples 11 through 14 includes determining that an AR device operates in a state with a high probability of accurate factory calibration by: evaluating usage logs to determine that the duration for which the AR device has been in active use since the AR device was factory calibrated is less than a threshold; evaluating usage logs to determine that the count of times the AR device has been activated since the AR device was factory calibrated is less than a threshold; evaluating configuration settings to determine that the duration for which the AR device has been first activated is less than a threshold; evaluating configuration settings to determine that the elapsed duration for which the AR device has been factory calibrated is less than a threshold; or any combination thereof.

[0096] In Example 16, the subject matter of Examples 11 to 15 includes, wherein the evaluation rule to determine the recalibration process to be performed includes: the evaluation rule to determine that the count of the number of possible calibration errors detected exceeds a threshold.

[0097] In Example 17, the subject matter of Examples 11 to 16 includes the following, wherein detecting possible calibration errors by comparing map data of a real-world environment with ground truth map data of a real-world environment includes: determining that the depth or location of a feature or object in the map data does not match the depth or location of a feature or object in the ground truth map data; determining that the scale or orientation of the map data does not match the scale or orientation of the ground truth map data; or a combination thereof.

[0098] In Example 18, the subject matter of Examples 11 through 17 includes the following: visual cues that prompt a user of an AR device to traverse a path in a real-world environment and look in a particular direction while traversing that path are derived by first determining an optimal path that allows for high observability of one or more parameters to be recalibrated, wherein the one or more parameters to be recalibrated are selected based on the types of possible calibration errors that have been detected.

[0099] In Example 19, the subject of Example 18 includes, wherein performing the recalibration process further includes, during the recalibration process and while the user is traversing a path guided by visual cues displayed via the AR device’s display, iteratively adjusting one or more calibration parameters of the AR device to reduce the difference between the newly generated map data and the ground truth map data.

[0100] In Example 20, the subject of Examples 11 through 19 includes, wherein performing the recalibration process further includes: during the recalibration process and while the user is traversing a path guided by visual cues displayed via a display of an AR device, in response to determining that the user has traversed the path in accordance with the displayed visual cues and is looking in various directions while traversing the path, providing visual feedback to the user via the display.

[0101] Example 21 is at least one machine-readable medium comprising instructions that, when executed by a processing circuitry system, cause the processing circuitry system to perform operations to implement any one of Examples 1 to 20.

[0102] Example 22 is a device that includes means for implementing any one of Examples 1 through 20.

[0103] Example 23 is a system for implementing any one of Examples 1 through 20.

[0104] Example 24 is a method for implementing any one of Examples 1 through 20.

[0105] Glossary

[0106] "Carrier signal" refers to any intangible medium, such as a medium capable of storing, encoding, or carrying instructions to be executed by a machine and including digital or analog communication signals, or other intangible medium that facilitates the transmission of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0107] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, multiple laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0108] "Communications network" refers to one or more parts of a network, such as an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling can enable any data transmission technology of various types, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0109] A “component” refers to, for example, a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies that implement specific processing or control functions. A component can be combined with other components via its interface to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components and part of a program that typically performs a related function. A component can constitute a software component (e.g., code contained on a machine-readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform certain operations. A hardware component can be a dedicated processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). Hardware components may also include programmable logic or circuitry systems temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a specific part of a machine) that is uniquely tailored to perform the configured functionality and is no longer a general-purpose processor. It will be appreciated that the decision to implement a hardware component mechanically in a dedicated and permanently configured circuitry system or in a temporarily configured (e.g., software-configured) circuitry system may be made for cost and time considerations. Therefore, the phrase “hardware component” (or “hardware-implemented component”) should be understood to include tangible entities, i.e., entities that are physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Consider examples of hardware components being temporarily configured (e.g., programmed), without needing to configure or instantiate each of the hardware components at any given time. For example, in the case where a hardware component includes a general-purpose processor configured by software to be a dedicated processor, the general-purpose processor may be configured at different times as its own different dedicated processors (e.g., including different hardware components). For example, software accordingly configures one or more specific processors to constitute a particular hardware component at one time and different hardware components at different times. Hardware components can provide information to other hardware components and can receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from the memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily (e.g., via software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be at least partially processor-implemented, where a particular processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or processor-implemented components, also referred to as “computer-implemented.” Furthermore, one or more processors can operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), where these operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not only residing within a single machine but also deployed across multiple machines. In some examples, the processor or processor-implemented component may reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other examples, the processor or processor-implemented component may be distributed across several geographic locations.

[0110] "Computer-readable storage medium" refers to both, for example, machine storage media and transmission media. Therefore, the term includes both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and may be used interchangeably in this disclosure.

[0111] A "brief message" is a message that is accessible for a limited time, such as a short period of time. Brief messages can be text, images, videos, etc. The access time for a brief message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is temporary.

[0112] "Machine storage medium" refers to one or more storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" refer to the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0113] "Non-transitory computer-readable storage medium" means, for example, a tangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine.

[0114] "Signal medium" means, for example, any intangible medium capable of storing, encoding, or carrying machine-executable instructions and including digital or analog communication signals, or other intangible media that facilitates the transmission of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0115] "User equipment" means, for example, a device that is accessed, controlled, or owned by a user and that the user interacts with to perform actions or interactions (including interactions with other users or computer systems).

Claims

1. An augmented reality (AR) device configured to perform a field-based recalibration process, the AR device comprising: monitor; processor; Two or more image sensors; Inertial Measurement Unit (IMU); as well as A memory storing instructions that, when executed by the processor, cause the AR device to perform operations, including: Map data of the real-world environment is generated by processing images obtained from the image sensor and motion data from the IMU. Possible calibration errors are detected by comparing the map data of the real-world environment with ground truth map data of the real-world environment (which was previously generated by the AR device). In response to the detection of the possible calibration error, an evaluation rule is used to determine whether a recalibration process should be performed; and The recalibration process is performed by generating visual cues and displaying them via the display, the visual cues prompting the user of the AR device to traverse a path in the real-world environment and look in a specific direction while traversing the path during the recalibration process.

2. The AR device according to claim 1, wherein, The truth map data of the real-world environment was previously generated by the AR device operating in the real-world environment.

3. The AR device according to claim 1, further comprising: Before comparing the map data of the real-world environment with the ground truth map data of the real-world environment: The AR device receives truth map data of the real-world environment captured by another computing device.

4. The AR device according to claim 1, wherein, The memory stores additional instructions thereon, which, when executed by the processor, cause the AR device to perform additional operations, including: Before generating the map data for the real-world environment: Determine that the AR device operates with a high probability of accurate factory calibration; and In response to determining that the AR device is operating in the state, i) generate the truth map data of the real-world environment, and ii) store the truth map data of the real-world environment.

5. The AR device according to claim 4, wherein, Determining that the AR device operates with a high probability of accurate factory calibration includes: The usage logs were evaluated to determine that the duration of active use of the AR device since it was factory-calibrated was less than a threshold. The evaluation logs determined that the count of the number of times the AR device had been activated since it was factory calibrated was less than a threshold. The configuration settings were evaluated to determine that the duration since the AR device was first activated was less than a threshold. Evaluate configuration settings to determine if the elapsed time since the AR device was factory calibrated is less than a threshold; or Any combination thereof.

6. The AR device according to claim 4, wherein, Determining that the AR device operates with a high probability of accurate factory calibration includes: The event has been determined to have not yet occurred, wherein the event is one of the following: An extreme acceleration event exceeding a threshold is detected via the IMU or dedicated accelerometer of the AR device, the extreme acceleration event indicating a physical impact or drop event; or A severed wire was detected within the frame of the AR device, indicating that the structural deformation exceeded a threshold.

7. The AR device according to claim 1, wherein, The evaluation rules for determining which recalibration process should be performed include: The rule is evaluated to determine if the count of the number of possible calibration errors detected exceeds a threshold.

8. The AR device according to claim 1, wherein, Detecting potential calibration errors by comparing the map data of the real-world environment with the ground truth map data of the real-world environment includes: It is determined that the depth or location of a feature or object in the map data does not match the depth or location of the feature or object in the ground truth map data; It is determined that the scale or orientation of the map data does not match the scale or orientation of the ground truth map data; or Its combination.

9. The AR device according to claim 1, wherein, The visual cues that prompt the user of the AR device to traverse the path in the real-world environment and look in a specific direction while traversing the path are derived by first determining an optimal path that allows for high observability of one or more parameters to be recalibrated, wherein the one or more parameters to be recalibrated are selected based on the types of possible calibration errors that have been detected.

10. The AR device according to claim 9, wherein, Performing the recalibration process also includes: During the recalibration process and as the user traverses a path guided by visual cues displayed via the AR device's display, one or more calibration parameters of the AR device are iteratively adjusted to reduce the discrepancy between the newly generated map data and the ground truth map data.

11. The AR device according to claim 1, wherein, Performing the recalibration process also includes: During the recalibration process and while the user is traversing a path guided by visual cues displayed via the display of the AR device, in response to determining that the user has traversed the path in accordance with the displayed visual cues and is looking in various directions while traversing the path, visual feedback is provided to the user via the display.

12. A method performed by an augmented reality (AR) device, the method comprising: Map data of the real-world environment is generated by processing images obtained from image sensors and motion data from IMUs. Possible calibration errors are detected by comparing the map data of the real-world environment with the ground truth map data of the real-world environment. In response to the detection of the possible calibration error, an evaluation rule is used to determine whether a recalibration process should be performed. as well as The recalibration process is performed by generating visual cues and displaying them via a display, the visual cues prompting the user of the AR device to traverse a path in the real-world environment during the recalibration process and to look in a specific direction while traversing the path.

13. The method according to claim 12, wherein, The truth map data of the real-world environment was previously generated by the AR device operating in the real-world environment.

14. The method of claim 12, further comprising: Before comparing the map data of the real-world environment with the ground truth map data of the real-world environment: The AR device receives truth map data of the real-world environment captured by another computing device.

15. The method of claim 12, further comprising: Before generating the map data for the real-world environment: The AR device is determined to operate with a high probability of accurate factory calibration. as well as In response to determining that the AR device is operating in the state, i) generate the truth map data of the real-world environment, and ii) store the truth map data of the real-world environment.

16. The method according to claim 15, wherein, Determining that the AR device operates with a high probability of accurate factory calibration includes: The usage logs were evaluated to determine that the duration of active use of the AR device since it was factory-calibrated was less than a threshold. The evaluation logs determined that the count of the number of times the AR device had been activated since it was factory calibrated was less than a threshold. The configuration settings were evaluated to determine that the duration since the AR device was first activated was less than a threshold. Evaluate configuration settings to determine if the elapsed time since the AR device was factory calibrated is less than a threshold; or Any combination thereof.

17. The method according to claim 15, wherein, Determining that the AR device operates with a high probability of accurate factory calibration includes: The event has been determined to have not yet occurred, wherein the event is one of the following: An extreme acceleration event exceeding a threshold is detected via the inertial measurement unit or a dedicated accelerometer of the AR device, the extreme acceleration event indicating a physical impact or drop event; or A severed wire was detected within the frame of the AR device, indicating that the structural deformation exceeded a threshold.

18. The method according to claim 12, wherein, The evaluation rules for determining which recalibration process should be performed include: The rule is evaluated to determine if the count of the number of possible calibration errors detected exceeds a threshold.

19. The method according to claim 12, wherein, Detecting potential calibration errors by comparing the map data of the real-world environment with the ground truth map data of the real-world environment includes: It is determined that the depth or location of a feature or object in the map data does not match the depth or location of the feature or object in the ground truth map data; It is determined that the scale or orientation of the map data does not match the scale or orientation of the ground truth map data; or Its combination.

20. The method according to claim 12, wherein, The visual cues that prompt the user of the AR device to traverse the path in the real-world environment and look in a specific direction while traversing the path are derived by first determining an optimal path that allows for high observability of one or more parameters to be recalibrated, wherein the one or more parameters to be recalibrated are selected based on the types of possible calibration errors that have been detected.

21. The method according to claim 20, wherein, Performing the recalibration process also includes: During the recalibration process and as the user traverses a path guided by visual cues displayed via the AR device's display, one or more calibration parameters of the AR device are iteratively adjusted to reduce the discrepancy between the newly generated map data and the ground truth map data.

22. The method according to claim 12, wherein, Performing the recalibration process also includes: During the recalibration process and while the user is traversing a path guided by visual cues displayed via the display of the AR device, in response to determining that the user has traversed the path in accordance with the displayed visual cues and is looking in various directions while traversing the path, visual feedback is provided to the user via the display.

Images