System and method for calibrating an augmented reality device

Abstract

A system and method for calibrating an augmented reality device. A method (60) of aligning virtual content generated by an AR device with a scene being viewed by a user in an operating environment using calibration data includes calibrating the AR device by pairing marker identifiers of 2D markers (50) affixed to objects (58) in a physical environment with 3D positions of those objects specified in a virtual environment containing 3D models of those objects and then generating a marker-to-model position pairing list that associates corresponding marker identifiers with corresponding 3D positions of the objects. The method (60) further includes using the pairing list to align the displayed virtual 3D content with objects appearing in the visualized physical environment. Position corrections are calculated at runtime based on a current AR device-to-marker offset calculated from an image of the 2D marker and the 3D position of the object retrieved from the pairing list.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for calibrating an augmented reality device. Background Art

[0002] Augmented reality (AR) is an interactive experience in a real-world environment in which objects residing in the real world are enhanced by computer-generated content and computer-processed sensory input. An augmented reality device allows a user to observe a real-world view while viewing virtual content superimposed on and aligned with items, images, objects, or environments in the field of view of the AR device or the user. Virtual content can include computer-generated images superimposed on a real-world view. Typical AR devices rely on calibrated sensors. If the sensors are not properly calibrated, the virtual content may appear misaligned with the real content.

[0003] Current AR systems often have difficulty obtaining accurate three-dimensional (3D) position information of an AR device relative to the physical environment in which the device is being used. These issues include: (1) position drift over time; (2) inability to obtain position information in areas with few features; and (3) inability to obtain correct position information in areas with repetitive features. Many AR systems use camera-based approaches to provide position information of visible features in the physical environment, but in some environments—particularly manufacturing-related use cases—the current level of 3D position accuracy of AR systems is insufficient to meet performance requirements. Summary of the invention

[0004] The subject matter disclosed in detail below relates to systems, methods, and apparatus for providing instant access to data to calibrate an augmented reality (AR) device relative to an operating environment, and then using the calibration data to align virtual content generated by the AR device with a scene being viewed by a user in the operating environment. A landmark-based reference system that uses two-dimensional (2D) markers as positional references that are mapped to a three-dimensional (3D) physical coordinate system defined for a specific target object (e.g., an aircraft) for an AR application. Passive 2D machine-readable code patterns on 2D markers (e.g., labels with symbols printed in a pattern) are used as positional landmarks to provide on-demand 3D position information to users.

[0005] As used herein, the term "position" includes a position in a fixed three-dimensional coordinate system and an orientation relative to the coordinate system. As used herein, a "physical coordinate system" is a coordinate system whose coordinates are ratio data scales and whose defined space is physical. As used herein, a "landmark-based reference system" is a reference system whose coordinates are ordinal data scales and whose defined space can be an abstract reference system. As used herein, the term "3D visualization environment" refers to a 3D display application running on a computer, which renders 3D content and displays it on a display device. The display system can be selected from at least one of a display device, a computer monitor, glasses, a head-mounted display, a tablet computer, a mobile phone, a smart phone, a wrist-mounted display, a projector, a head-mounted display (HMD), a holographic display system, a retinal display, and some other suitable display devices.

[0006] According to one embodiment, the method includes calibrating the AR device by pairing marker identifiers of 2D markers of objects affixed to the physical environment with 3D positions of those objects specified in a 3D model of those objects and then generating a marker-to-model position pairing list associating respective marker identifiers with respective 3D positions of the objects. The method further includes using the pairing list to align displayed virtual content with objects appearing in the visualized physical environment. The position calibration is derived at runtime based on a current AR device-to-marker offset derived from an image of the 2D marker and the 3D position of the object retrieved from the pairing list.

[0007] The present disclosure describes steps for implementing the concept of using passive 2D markers for 3D positioning and / or 3D positioning calibration (collectively referred to as "3D positioning" below) in AR applications. More specifically, the present disclosure describes a method for improving AR system position estimation for applications that require improved 3D positioning accuracy, such as building airplanes. The method proposed herein includes setting up a physical environment, obtaining reference position data from a 3D model of an object, and using the 3D position data at runtime to calibrate the position estimate of an existing positioning process of an AR device. The innovative features of the method include: (1) a process of establishing a physical environment and obtaining calibration data; and (2) a process of associating a 2D marker with a known 3D position of an object to which the marker is attached, using the association to generate real position information, and applying position calibration at runtime based on the current estimated position of the AR device and the 3D position of objects appearing in the scene being viewed.

[0008] Although various embodiments of systems, methods, and apparatus for aligning virtual content generated by an augmented reality (AR) application with a scene being viewed by a user in an operating environment are described in greater detail later herein, one or more of these embodiments may be characterized by one or more of the following aspects.

[0009] One aspect of the subject matter disclosed in detail below is a method for providing instant access to data for calibrating an AR device relative to an operating environment, the method comprising: (a) creating a plurality of 2D machine-readable code patterns on respective 2D markers using a first symbol representing a marker identifier that uniquely identifies the respective 2D markers and a second symbol representing a registration marker; (b) applying the respective 2D markers to selected locations on respective objects in a physical environment; (c) capturing image data representing the respective 2D markers in the physical environment; (d) processing the image data to obtain digital data representing the respective marker identifiers; (e) loading a correctly positioned 3D model representing the respective objects into a 3D visualization environment; (f) obtaining digital data representing the 3D location of the respective objects from the 3D model to which the 2D machine-readable code pattern is attached from the 3D virtual environment; (g) pairing the marker identifier of the respective 2D marker with the 3D location of the respective objects from the 3D model; (h) generating a data file containing pairing list data representing a pairing list of marker identifiers and 3D locations for use at runtime; and (i) converting the pairing list data into a form suitable for instant loading onto the AR device. In one proposed implementation, the 2D machine-readable code pattern is a QR code, the physical environment is an airplane, and the object is a window of the airplane.

[0010] According to one embodiment of the method described in the previous paragraph, the method further includes: (i) loading pairing list data into the AR device; (k) loading virtual content into the AR device, the virtual content including virtual features of an object with a 2D marker applied; (i) viewing the object in a physical environment on the AR device; (m) capturing image data representing the 2D marker applied to the object; (n) processing the image data to obtain data representing a marker identifier of the 2D marker applied to the object and a spatial position of a registration marker; (o) processing the spatial position of the registration marker to calculate an offset from the current AR device to the marker; (p) finding a 3D position of an object associated with the marker identifier of the 2D marker in the pairing list; (q) calculating a 3D position of the AR device in a physical environment reference system based on the offset from the current AR device to the marker and the 3D position of the object found in the pairing list; and (r) displaying the virtual content with a viewpoint based on the offset from the current AR device to the marker and the 3D position of the object found in the pairing list. According to one embodiment, step (i) includes creating a 2D machine-readable code pattern on a 2D marker using a symbol representing the pairing list data, and attaching the 2D marker to a structure in the physical environment; and step (i) includes optically reading the 2D machine-readable code pattern to obtain the pairing list data, and storing the obtained pairing list data in a non-transitory tangible computer-readable storage medium of the AR device. According to another embodiment, step (i) includes formatting the pairing list data for wireless transmission, and then transmitting a short wavelength ultra-high frequency radio wave modulated with information representing the pairing list data; and step (i) includes receiving an electromagnetic wave, demodulating the received electromagnetic wave to obtain the pairing list data, and storing the pairing list data in a non-transitory tangible computer-readable storage medium of the AR device.

[0011] Another aspect of the subject matter disclosed in detail below is a method for aligning virtual content generated by an augmented reality application with a scene being viewed by a user in an operating environment, the method comprising: (a) creating a plurality of 2D machine-readable code patterns on corresponding 2D markers using a first symbol representing a marker identifier that uniquely identifies the corresponding 2D marker and a second symbol representing a registration marker; (b) applying the corresponding 2D marker to a selected location on a corresponding object in a physical environment; (c) loading pairing list data into an AR device, wherein the pairing list data represents a pairing of the marker identifier on the corresponding 2D marker with a 3D location of the corresponding object to which the 2D marker is applied; and (d) loading virtual content into the AR device, the virtual content comprising a virtual image of the object to which the 2D marker is applied. (e) viewing an object in the physical environment on the AR device; (f) capturing image data representing a 2D marker applied on the object; (g) processing the image data to obtain data representing a marker identifier of the 2D marker applied on the object and a spatial position of a registration marker; (h) processing the spatial position of the registration marker to calculate an offset from the current AR device to the marker; (i) finding a 3D position of an object associated with the marker identifier of the 2D marker in a pairing list; (i) calculating a 3D position of the AR device in a reference frame of the physical environment based on the offset from the current AR device to the marker and the 3D position of the object found in the pairing list; and (k) displaying virtual content with a viewpoint based on the offset from the current AR device to the marker and the 3D position of the object found in the pairing list.

[0012] Another aspect of the subject matter disclosed in detail below is an AR device, comprising an imaging device, a display device, a non-transitory tangible computer-readable storage medium, and a computer system communicatively coupled to the imaging device, the display device, and the non-transitory tangible computer-readable storage medium, wherein: the non-transitory tangible computer-readable storage medium stores pairing list data and virtual content data, the pairing list data representing pairings of tag identifiers on 2D tags and 3D locations of objects in a physical environment to which the corresponding 2D tags are applied, the virtual content data representing virtual content including virtual features of the objects to which the corresponding 2D tags are applied; and the computer system is configured to perform the following operations: (a) process image data from an image captured by the imaging device to generate a 3D image of the object; and (b) process image data from an image captured by the imaging device to generate a 3D image of the object. Obtaining data representing a marker identifier of a 2D marker appearing in a captured image and a spatial position of a registration marker applied to an object; (b) processing the spatial position of the registration marker to calculate a current AR device-to-marker offset; (c) finding a 3D position of the object associated with the marker identifier of the 2D marker appearing in the captured image in a pairing list stored in a non-transitory tangible computer-readable storage medium; (d) calculating a 3D position of the AR device in a reference frame of the physical environment based on the current AR device-to-marker offset and the 3D position of the object found in the pairing list; and (e) displaying virtual content including virtual features of the object at a viewpoint based on the current AR device-to-marker offset and the 3D position of the object.

[0013] Other aspects of systems, methods, and apparatus for registering virtual content generated by an AR application with a scene being viewed by a user in an operating environment are disclosed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The features, functions, and advantages discussed in the foregoing sections may be implemented independently in various embodiments or may be combined in other embodiments. To illustrate the above and other aspects, various embodiments will be described below with reference to the accompanying drawings. All of the illustrations briefly described in this section are not drawn to scale.

[0015] Figure 1 is a diagram showing a QR code pattern mark attached to a window.

[0016] Figure 2 is a coordinate system conversion diagram showing a conversion sequence for converting the position of the AR device from the marker reference system to the reference system of the aircraft.

[0017] Figure 3 is a diagram showing the use of a mark (eg, a QR code pattern mark) adhered to a window inside a fuselage of an aircraft.

[0018] Figure 4is a diagram showing the pairing and registration of a 2D marker (eg, a QR code pattern marker) and a 3D model of an object (eg, an airplane window) for an augmented reality application.

[0019] Figure 5 is a diagram representing runtime use of an augmented reality system configured to correct interior 3D position estimates using 2D markers affixed to windows inside an aircraft and a list of 2D marker-to-3D object model pairings (hereinafter referred to as a “marker-to-model position pairing list”).

[0020] Figure 6 is a flow chart identifying steps of a method for providing instant access to data for calibrating an AR device relative to an operating environment, according to one embodiment.

[0021] Figure 7 is a flow chart identifying steps of a method for registering virtual content generated by an augmented reality application with a scene being viewed by a user in an operating environment, according to one embodiment.

[0022] Figure 8 is a block diagram identifying some components of an AR system capable of aligning computer-generated virtual features with a physical environment viewed using 2D markers as landmarks.

[0023] Fig. 9A is a diagram showing an isometric front view of a handheld imaging device that can be positioned using the methods disclosed herein.

[0024] Fig. 9B Yes means Fig. 9A Illustration of an isometric rear view of the depicted handheld imaging device.

[0025] Fig.10 is a diagram showing a schematic diagram of a field of view of a handheld imaging device projected onto a portion of a surface having a code pattern indicia in the form of a label applied thereto.

[0026] Fig.11 is a diagram showing an example of a label having a data matrix code pattern and three registration fiducial symbols.

[0027] Fig.12 is a diagram showing an example of a landmark-to-location mapping embodied as a QR code.

[0028] FIG. 13A to FIG. 13C are diagrams representing respective schematic diagrams of the field of view of a camera incorporated in an AR device projected onto respective areas of a surface having an array of code pattern markings applied thereon according to an optional operating mode.

[0029] Fig.14is a diagram representing an overlay of virtual content in the form of a texture map representing a code pattern on a view of a 2D marker having the code pattern printed thereon.

[0030] FIG. 15A to FIG. 15F is a diagram representing the superposition of a corresponding geometric symbol on a view of an object to which a 2D marker has been adhered (in this example, a window surrounded by a window frame).

[0031] Fig.16 is a block diagram identifying components of a system 100 suitable for hosting an augmented reality application that can communicate with a handheld or head-mounted imaging device.

[0032] Reference will hereinafter be made to the drawings, in which similar elements in different drawings have the same reference numerals. DETAILED DESCRIPTION

[0033] Illustrative embodiments of systems, methods, and devices for aligning virtual content generated by an augmented reality (AR) application with a scene being viewed by a user in an operating environment are described in more detail below. However, not all features of an actual implementation are described in this specification. Those skilled in the art will appreciate that in the development of any such implementation, many implementation-specific decisions must be made to achieve the developer's specific goals, such as compliance with system-related and business-related constraints, which will vary from one implementation to another. In addition, it should be understood that such development work may be complex and time-consuming, but it will still be routine work for those of ordinary skill in the art who benefit from the present disclosure.

[0034] For the purpose of illustration, a method for calibrating an AR device to enable accurate alignment of displayed virtual content with viewing objects in the physical environment of an aircraft interior will be described in detail below. However, the calibration and alignment process disclosed herein can also be used for AR applications of target objects other than aircraft.

[0035] An example of a structure or object that can use the present embodiment is an aircraft fuselage. Typically, the fuselage is designed using a detailed computer-aided design (CAD) system that is used to create a virtual environment consisting of a 3D model of the structure, wherein items are defined relative to a common Cartesian coordinate system. This coordinate system can then be used as a reference coordinate system for determining the absolute position of any element of the structure contained in a 3D model database. The CAD system can be used to create the contents of the 3D model database, which is then maintained and can be used to evaluate references in ongoing aircraft inspections and maintenance. The 3D model database provides a coordinate system and precise geometric information associated with the coordinate system, which is associated with the entire structure and all components contained therein. When maintenance is performed, information about repairs or other maintenance procedures can be stored in a maintenance and / or other database relative to the reference coordinate system, which is also associated with the reference coordinate system. Although a 3D model database is used for the example embodiment, any system that establishes a reference to the absolute position of a common coordinate system can be used.

[0036] Aircraft have many features that can be associated with the same features in the CAD database. This one-to-one mapping between the physical location of features on the real object and the measured Cartesian coordinates from the virtual model (hereinafter referred to as "landmark to location mapping") enables the process of tracking the motion of the imaging device in the physical environment.

[0037] The subject matter disclosed in detail below relates to systems and methods for calibrating an AR device by pairing coded two-dimensional (2D) markers affixed in a physical environment with 3D models of objects in the physical environment, and then using these pairings to align displayed virtual content with objects in the viewed physical environment. A landmark-based reference system using 2D markers as landmarks is mapped to a 3D physical coordinate system defined for a specific target object (e.g., an aircraft) for an AR application. Passive coded 2D markers (e.g., labels with printed code patterns) are used as positional landmarks to provide on-demand 3D position information to users.

[0038] The present disclosure describes steps for implementing the concept of using 2D markers for 3D positioning in AR applications. More specifically, the present disclosure describes a method for improving AR system position estimation for applications that require improved 3D positioning accuracy, such as building aircraft. The method proposed herein includes setting up a physical environment, obtaining reference position data from a 3D model of an object, and using the 3D position data at runtime to calibrate the position estimate of an existing positioning process of an AR device. The innovative features of the method include: (1) a process of setting up a physical environment and obtaining calibration data; and (2) a process of associating a 2D marker with a known 3D position of an object to which the marker is attached, using the association to generate real position information, and applying position correction at runtime based on the current estimated position of the AR device and the 3D position of objects appearing in the scene being viewed.

[0039] Passive 2D markers are used as positional landmarks to provide on-demand 3D position information to users in a landmark-based reference system, which can then be mapped to an underlying 3D physical coordinate system defined for a particular target object (e.g., an airplane). According to the methods disclosed herein, the passive 2D markers are in the form of attachable labels, such as adhesive labels with printed text and machine-readable data storage capabilities. The labels are applied to corresponding different locations on the target object (e.g., on some windows of an airplane, even if the labels are not in exactly the same location on every window).

[0040] The machine-readable data storage capability preferably takes the form of a 2D code pattern. As used herein, the term "code pattern" refers to a symbol system that represents machine-readable data. According to various embodiments, the code pattern includes an array of pixel-like elements in a pattern that is decoded by a scanning algorithm. Examples of suitable commercially available 2D code patterns include the following: barcodes, Quick Response (QR) codes, Data Matrix codes, Aztec codes, and MaxiCodes.

[0041] The methods disclosed herein provide a way for users to quickly, easily, and inexpensively detect an environment using location landmarks (e.g., tags or markers with printed code patterns) that can later be used to give discrete and continuous on-demand location tracking at a resolution level specified by the landmark spacing or density in the environment used by the AR application. The location landmarks are designed to be quickly and easily established and later removed.

[0042] According to some embodiments disclosed herein, the locations of landmarks exist in a landmark-based reference frame defined by the user based on unique descriptions that may be meaningful to the user in some use cases (e.g., "left window in row 1," "front restroom," etc.). These locations can then be mapped into an underlying physical 3D coordinate system that gives more precise Cartesian x, V, z coordinates that other tools can use to determine the location of an imaging device (e.g., a handheld scanner or a camera integrated into a head-mounted display). For example, an AR device can be configured to receive viewpoint information (e.g., the location of a virtual camera) from an integrated camera regarding multiple landmarks with known coordinates in the reference frame of a target object, and based on the user viewpoint information, set the viewpoint so that the virtual content simultaneously generated by the AR application is aligned / registered with the scene that the user is viewing in the physical world (e.g., inside an airplane) at the time.

[0043] As used herein, the term "viewpoint" is the apparent distance and direction at which a camera views and records an object. AR devices allow users to display virtual content from a viewpoint that can be characterized as the apparent location of a virtual camera. As used herein, the term "position" includes both position (e.g., x, v, z coordinates) and direction (e.g., the viewing direction vector of a virtual line of sight).

[0044] The AR device calibration process proposed in this paper uses landmarks in the form of 2D markers with code patterns printed on a substrate that is affixed to physical objects at known locations in the physical environment. An example of a suitable commercially available code pattern is a QR code. A QR code is a 2D barcode (also called a matrix barcode) with an integrated registration fiducial symbol (also called a registration mark). Figure 1 is a diagram showing a QR code pattern mark 50 pasted on a window 58. The QR code pattern mark 50 includes a flexible substrate on which a QR code pattern 48 is printed. The QR code pattern 48 includes a payload consisting of data identifying on which window 58 the QR code pattern mark 50 is pasted. In addition, the QR code pattern 48 includes a registration reference symbol 52a at the lower left corner of the code pattern, a registration reference symbol 52b at the upper left corner of the code pattern, and a registration reference symbol 52c at the upper right corner of the code pattern for tracking the AR device ( Figure 1 The position of (not shown).

[0045] As described above, during operational use, the absolute position coordinates of the AR device / camera defined in the reference frame of the physical environment (e.g., an aircraft) can be calculated using the current AR device-to-marker offset and the 3D position of the object to which the marker is attached in the physical environment reference frame. A 4×4 homogeneous transformation matrix is ​​used to define the spatial position (position and orientation) of one specific reference frame relative to another specific reference frame. A sequence of 4×4 homogeneous transformation matrix multiplications can be used to convert between coordinate frames.

[0046] Using a 3D model of an object (e.g., a window of an airplane) to which a QR code is attached (e.g., glued) and has a position defined in the reference frame of a physical environment (e.g., an airplane), the 3D position of the QR code in the reference frame of the physical environment is obtained. Then, the position of the camera (which is part of the AR device) relative to the QR code (the offset of the AR device to the marker) is calculated using a QR code-based process that solves the camera pose problem. Then, the position of the AR device / camera in the reference frame of the physical environment (e.g., in airplane coordinates) is calculated using the transformation matrix equation.

[0047] Figure 2 is a coordinate system conversion diagram showing the conversion sequence for converting the position of the AR device / camera from the reference frame of the QR code to the reference frame of the aircraft. The specific conversion sequence for converting the position of the AR device / camera from the reference frame of the QR code to the reference frame of the aircraft is described by the following equation:

[0048]

[0049] This equation calculates the transformation matrix (which is the position of the AR device / camera defined in reference frame {C} relative to the reference frame {A} of the aircraft) as the transformation matrix (which is the position of the QR code defined in reference frame {Q} relative to the aircraft's reference frame {A}) and the transformation matrix (which is the position of the AR device / camera defined in reference frame {C} relative to the position of the QR code defined in reference frame {Q}). T is a 4×4 homogeneous transformation matrix, and the leading subscripts and superscripts are as follows: {C} is the reference frame of the AR device / camera; {A} is the absolute reference frame of the aircraft; and {Q} is the reference frame of the QR code.

[0050] Figure 3is a diagram illustrating the use of QR code pattern markings 50a and 50b within an aircraft fuselage interior. Each of the QR code pattern markings 50a and 50b includes a symbol (QR code pattern 48a and 48b) that encodes corresponding unique reference location identification data identifying the reference location where the QR code pattern has been placed. In addition, each QR code pattern 48a and 48b includes a registration reference symbol at the lower left corner of the code pattern, a registration reference symbol at the upper left corner of the code pattern, and a registration reference symbol at the upper right corner of the code pattern, as previously described in Figure 2 As described in Figure 3 In the illustrated scenario, a QR code pattern mark 50a is applied to the center of the top of a window 58a surrounded by a window frame 46a; a QR code pattern mark 50b is applied to the center of the top of a window 58b surrounded by a window frame 46b. Each QR code pattern 48a and 48b is unique. The payload in each QR code pattern mark contains a unique identifier for a corresponding one of the windows. If desired, the QR code pattern marks 50a and 50b can be printed on both sides, so they are visible and usable from both the inside and outside of the aircraft. The text on each mark indicates which row of windows the mark should be installed on. If each window is covered with a protective film, the associated code pattern mark can be applied to the film and will subsequently be removed with the film during the assembly process without leaving a significant risk of foreign matter on the aircraft. In some embodiments, the marking pattern can be printed on or within the protective film itself.

[0051] The QR code patterns 48a and 48b contain registration marks of known size and orientation, such as fiducial symbols 52a to 52b, which allow a small amount of continuous relative position and orientation tracking when the QR code pattern marks are completely within the field of view of a handheld imaging device (e.g., a camera) capable of continuous scanning. If more than one QR code pattern mark is in the field of view of the camera, positions can be inserted between two QR code pattern marks, which enables continuous scanning over a larger area. The steps of scanning the QR code pattern marks and using the returned logical coordinates can be repeated as often as needed. Once the process is completed, the set of QR code pattern marks can be removed at that time, or they can be left on the plate for reuse until the window protection film itself is removed later in the assembly process.

[0052] In the embodiments disclosed herein, a QR code pattern is preferred because the registration marks are integrated into the format and because the QR code pattern is a standard type available in commercially available off-the-shelf products, which makes it more convenient for the user to print the label. However, alternative code patterns that do not have integrated registration marks can be employed. If the continuous relative position tracking aspect of the concept is not required or if it is a separate part of the label, Data Matrix Codes, MaxiCodes, Aztec Codes, and other types of 2D barcodes can be used.

[0053] According to an optional embodiment, a separate registration mark template (i.e., a background image) can be added to a code pattern that lacks integrated registration marks. The code pattern can be placed in the empty space between the registration marks to provide a continuous tracking function equivalent to that which can be achieved using a QR code. Any source that provides machine-readable landmark-based reference data can be employed, such as a UPC standard barcode, a Data Matrix (ECC 200) 2D matrix barcode, or a MaxiCode 2D matrix barcode (used by UPS public domain). As with the QR code pattern example, these optional forms of discrete data storage can be used with relative motion tracking to provide continuous tracking in an absolute coordinate system.

[0054] Figure 4 is a diagram showing the pairing and registration of 2D markers (e.g., QR code pattern markers 50a and 50b) and 3D models 44a and 44b of objects (e.g., airplane windows) for augmented reality (AR) applications. In this example, windows 58a and 58b have corresponding QR code pattern markers 50a and 50b adhered (pasted) thereon; window 58 has no marker. The payload of each QR code pattern marker 50a and 50b includes encoded data that uniquely identifies the marker. Figure 4 In the example shown, the QR code pattern mark 50a is uniquely identified by the reference "W32R", while the QR code pattern mark 50b is uniquely identified by the reference "W34R".

[0055] During setup, QR code pattern markers 50a and 50b are scanned and read by the system. The markers can be scanned using a separate handheld scanner or an imaging device (e.g., a camera) integrated with the AR device. The markers are read by processing the image data to extract the marker identifiers. Before or after scanning the markers, the 3D models 44a and 44b are loaded into the 3D visualization environment for use during the pairing and registration process. The corresponding marker identifiers are then associated (in a pairing list) with the corresponding positions of the physical objects represented by the corresponding 3D models. A data file containing a pairing list of marker identifiers and positions of objects represented by the 3D models is generated and saved (in a non-transitory tangible computer-readable storage medium) for use at runtime. For example, the pairing list can be in the form of a lookup table.

[0056] More generally, the technology disclosed herein includes a process for pairing two forms of acquired data: a unique alphanumeric payload of a 2D tag and corresponding 3D position data that specifies the location of the 2D tag in a reference frame of a physical environment (e.g., an aircraft interior). 2D tags are placed throughout the physical environment and associated with corresponding 3D locations to establish a pairing table during an initial setup phase. The setup phase involves using a 3D model of the physical environment that can be queried by a user to obtain a specific 3D location on a target object, and the 2D tag (e.g., a matrix barcode, such as a QR code) is read by a scanner (or by a camera and image processing application) to obtain a unique identifier payload encoded in the 2D tag. These two pieces of information are associated together in a pairing table, and then the user continues the acquisition process to obtain the next pair, and so on. When the data acquisition process is complete, the pairing table is saved and can be converted into another machine-readable form accessible to the user (e.g., another matrix barcode, such as a QR code or other electronic form). The pairing table will be scanned by the user's AR device at the beginning of runtime use. Because the use of the system is decentralized, multiple independent instances of the runtime application can be used simultaneously.

[0057] According to some embodiments, when a user (or an autonomous agent in the case of robotic applications) enters a physical environment, a unique list of marker-to-model location pairings may be automatically read by an imaging device integrated with the AR device. Figure 5 The scene is shown after the initial setup and after the user has entered the aircraft via the gate 55 and is walking through the aisle. A QR code pattern marker with a landmark to position mapping 54 printed on it is placed near the gate during setup and is used to retrieve the associated physical aircraft coordinates of the detected markers in the aircraft coordinate system. These physical aircraft coordinates can then be fed into the AR application to set the position and orientation of the virtual content displayed by the AR device to align with the physical objects that the user is viewing during runtime. According to other embodiments, the user (or agent) decides when to scan the pairing list. In some embodiments, the 3D position can include both position and orientation (such as a selected surface position with x, v, z coordinates and the orientation of the surface normal at that position). In other embodiments, only x, y, z coordinates may be required.

[0058] After initial setup, AR applications involve scanning the physical environment for 2D markers, looking up the corresponding 3D position associated with the 2D marker at runtime, applying that 3D position to the current AR device-to-marker offset, and then correcting the AR system's internal 3D position estimate. The AR system is capable of supporting multiple users simultaneously and is not limited to using any specific manufacturer's hardware.

[0059] Figure 5is a diagram representing runtime usage of an AR system configured to correct interior 3D position estimates using 2D markers affixed to windows inside an aircraft and a list of marker-to-model position pairs. Figure 5 In the example shown, the 2D markers include QR code pattern markers 50a to 50d affixed to windows 58a to 58d, respectively; some windows 58 have no 2D markers. Windows 58a to 58d are surrounded by window frames 46a to 46d, respectively. If multiple readable markers are in view, the positioning algorithm selects the marker that is closer, and if no fully readable marker is in view (which may include markers that are too blurry to read), the method does not provide a position correction to the AR system.

[0060] Figure 5 Two users 2 are shown walking on the floor 6 of an aircraft cabin. Each user 2 wears a respective head mounted display 4 as part of an AR device. The head mounted display 4 is configured to place images of both the physical world and virtual objects in the user's field of view 8. At least one camera ( Figure 5 (not shown) is mounted to or included in each head mounted display 4 device. In this figure, for each device, one of the cameras faces forward, wherein the corresponding field of view 8 is represented by Figure 5 The camera is used to capture images of the QR code pattern markers 50a to 50d, which are then processed to calculate the position data required to align the virtual content being displayed on the head mounted display 4 with the scene being viewed by the user.

[0061] One type of wearable AR device is sometimes called a perspective AR device, in which the user wears a head-mounted display and sees light from the real world and light from a computer-generated virtual environment. There are other forms of AR that use image data from a camera with computer-generated content as an overlay on the camera image data. The technology proposed in this article can be applied to both forms of AR. The virtual content displayed in these displays can be any form of computer-generated data. Some virtual content can be 3D solid models (such as 3D CAD models); other forms can be 3D lines, texture mapping, or even text placed in appropriate 3D positions. Not all hardware is capable of rendering complex 3D solid models (hereinafter referred to as "3D models"), and other hardware may only render a limited number of 3D geometries. In addition, in some cases, rendering a simpler representation may be better than having a complete 3D model. In all cases, it is still necessary to properly place (position) the computer-generated data (whether it is a 3D model or other virtual content) in the 3D physical environment (position and orientation), which is the core element of physical to virtual world alignment. Although one intended use for the system is for head-mounted display devices, other types of AR systems will also be able to use the techniques disclosed herein.

[0062] The initial setup process involves scanning 2D markers in a physical environment and associating (pairing) the 2D markers with corresponding 3D locations of physical objects that are part of a reference object (such as a window frame on an airplane). Each window frame can be a separate object with a corresponding 2D marker affixed to it, and each 3D model of a window frame has specific corresponding 3D location data that can be associated with a marker identifier.

[0063] After or before the pairing list is loaded into the AR device, virtual 3D content related to the object (e.g., 3D models, 3D lines, or other 3D forms) is also loaded into the AR device. In AR applications, complete CAD models are typically not used because some AR devices may not have sufficient rendering capabilities to display the complete model. Some embodiments of the techniques presented herein involve showing reference 3D content, such as a portion of a window or a border around a window or a square outline of a QR code adhered to a window. Once the limited 3D content is properly aligned through a calibration process (physical to virtual world registration), the user sees the 3D content as an overlay on the real-world scene.

[0064] At the beginning of the setup process, 3D virtual objects - e.g., 3D solid models, 3D line segments, or marker outlines defined in absolute (aircraft) coordinates - are loaded into the virtual environment display application on the AR device, but the unique contents of the marker's payload (e.g., a matrix barcode such as a QR code) have not yet been associated with the appropriate 3D virtual object. Thereafter, each unique marker identifier is associated with a corresponding unique position (position and orientation) of the corresponding 3D virtual object. The association of the marker with the position enables the process to determine the relative position of the marker with respect to the virtual 3D object it is paired with, and the computer can then calculate the absolute position of the marker in the physical environment (e.g., aircraft coordinate) reference system.

[0065] Using an AR device at runtime is a different process than how the AR device is used during the initial setup process, which involves the initial pairing of physically encoded 2D markers with corresponding marker locations. Construction of the marker-to-model location pairing table is done once per aircraft, and then the pairing table is available at runtime to any user who enters that aircraft and wishes to use their AR device and the tracking correction system proposed herein.

[0066] Figure 66 is a flowchart identifying steps of a method 60 for setting up and calibrating an AR system according to one embodiment, the AR system being configured to scan 2D markers placed on physical objects for the purpose of registering virtual content with an image of a physical environment. The initial setup of the system in a target environment involves the following steps. First, a plurality of 2D machine-readable code patterns are created on the corresponding 2D markers using a first symbol representing a marker identifier that uniquely identifies the corresponding 2D marker and a second symbol representing a registration marker (step 61). Then, the corresponding 2D marker is applied to a selected location on the corresponding object in the physical environment (step 62). After the marker installation is completed, image data representing the corresponding 2D marker in the physical environment is captured using an imaging device (e.g., a handheld imaging device or an imaging device incorporated in an AR device) (step 63). The image data is processed to obtain digital data representing the corresponding marker identifier (step 64). Before or after obtaining the image data, a 3D model representing the corresponding object with the marker affixed is loaded into a 3D visualization environment (step 65) and displayed on a laptop or tablet computer. Then, the user obtains digital data representing the 3D position of the corresponding object from the 3D model (step 66). The user then instructs the application to pair each marker identifier of the corresponding 2D marker with the 3D position of the corresponding object from the 3D model (step 67). Upon completion of the pairing of the marker identifiers with the 3D positions of the associated objects, a data file containing pairing list data representing a list of pairings of marker identifiers and 3D positions is generated for use at runtime (step 68). The pairing list data is then converted into a form suitable for immediate loading onto an augmented reality (AR) device (step 69), such as Fig.12 Examples in .

[0067] Physical objects (represented by 3D models) have known position coordinates in the reference frame of the physical environment, which means that the coordinates of the collocated marker can also be determined by referencing the pairing list. Subsequently, the coordinates of the AR device in the reference frame of the aircraft can be calculated based on the marker coordinates and the offset from the AR device to the marker measured as described below.

[0068] For the initial setup of the aircraft interior environment, the most efficient method of installing 2D markings on windows would be to have the window OEM print the unique 2D code pattern on a protective film placed on the window before delivery. However, this would likely require a level of planning and possibly impractical changes at the window manufacturer. As a more realistic alternative, a separate unique 2D code pattern marking could be placed on the protective film after the window is prepared by the OEM. The 2D code pattern marking could be placed on the window before or after the window is installed on the aircraft.

[0069] In addition to loading the pairing list data onto the AR device, Figure 6The steps shown are environment preparation tasks that occur only once, while loading data into an AR device is an event that occurs for an individual user and may occur multiple times. An AR device can be used on multiple aircraft (or multiple environments) rather than being assigned to one aircraft. A user might work on a particular aircraft for an hour or so, and then move to a different aircraft, and then move to another aircraft later (or perhaps back to the first aircraft). Dozens of other workers with their own AR devices may also need to load data. After the pairing list has been created, the data preparation for one aircraft will be used by many separate AR devices.

[0070] After the initial setup, the AR device can use 2D markers at runtime to achieve accurate positioning of virtual content relative to the scene being viewed by the user of the AR device. The runtime process needs to be scalable so that setup data from multiple environments (aircraft) can be easily applied to multiple devices - and in an instant manner (such as when the user walks onto the aircraft). To achieve scalability, the pairing list data is prepared in such a way that the data can be easily uploaded to the AR device when the user walks into the environment. There are several ways in which this upload can occur.

[0071] According to one embodiment, step 69 includes: creating a 2D machine-readable code pattern on a 2D marker using symbols representing the pairing list data; and attaching the 2D marker to a structure in the physical environment. For example, the pairing list data can be encoded as a large QR code (or multiple QR codes) attached to a location near the entry gate of the aircraft. When the user walks onto the aircraft to configure their AR device, the user scans the set QR code with the AR device. This is an optical process using something like a printable QR code (but something like an electronic ink display can also be used).

[0072] According to another embodiment, step 69 includes: formatting the pairing list data for wireless transmission; and transmitting short wavelength ultra-high frequency radio waves modulated with information representing the pairing list data. For example, the pairing list data may be uploaded using a short-range local wireless transmission process (such as Bluetooth) that provides pairing list data for a specific aircraft, which is accessible once the user steps on the aircraft, but is not accessible from other locations (such as on other aircraft) because mixing calibration data sets should be avoided. (Bluetooth is a wireless technology standard for exchanging data between fixed and mobile devices over short distances using short wavelength UHF radio waves in the industrial, scientific and medical radio band (from 2.400 GHz to 2.485 GHz) and establishing personal area networks.)

[0073] Figure 78 is a flowchart identifying steps of a method 88 for registering virtual content generated by an augmented reality application with a scene being viewed by a user in an operating environment, according to one embodiment. The method 88 for registering virtual content with a scene being viewed involves the following steps. Pairing list data is loaded into an AR device (step 70). Before or after step 70, virtual content including virtual features of an object with a 2D marker applied is loaded into the AR device (step 72). The virtual features have a fixed relationship with the virtual object corresponding to the object in the physical environment. The object in the physical environment is then viewed on a display device incorporated in the AR device (step 74). In addition, image data representing the 2D marker applied on the object is captured using an imaging device (e.g., a camera) incorporated in the AR device (step 76). The image data is processed to obtain data representing a marker identifier of the 2D marker applied on the object and a spatial position of a registration marker (step 78). The spatial position of the registration marker is then processed to calculate an offset from the current AR device to the marker (step 80). The process then searches the pairing list and finds the 3D position of the object associated with the marker identifier of the 2D marker applied on the object (step 82). Then, the 3D position (e.g., absolute position coordinates) of the AR device defined in the reference frame of the physical environment (e.g., the aircraft) is calculated based on the current AR device-to-marker offset and the 3D position of the object (collapsed with the marker) found in step 82 (step 84). Then, the virtual content is displayed with a viewpoint based on the current AR device-to-marker offset calculated in step 84 and the 3D position found in step 82 (step 86).

[0074] According to one embodiment, step 69 (see Figure 6 ) includes: preparing the pairing list data in a form suitable for immediate loading onto the AR device, for example, creating a 2D machine-readable code pattern on a 2D marker using symbols representing the pairing list data, and attaching the 2D marker to a structure in the physical environment. According to another embodiment, step 70 (see Figure 7 ) includes optically reading a 2D machine-readable code pattern to obtain pairing list data and storing the obtained pairing list data in a non-transitory tangible computer-readable storage medium of the AR device.

[0075] According to another embodiment, step 69 (see Figure 6 ) includes: formatting the pairing list data for wireless transmission; and transmitting a short wavelength ultra-high frequency radio wave modulated with information representing the pairing list data; and step 70 (see Figure 7 ) includes: receiving electromagnetic waves, demodulating the received electromagnetic waves to obtain pairing list data, and storing the pairing list data in a non-temporary tangible computer-readable storage medium of the AR device.

[0076] Figure 8 is a block diagram of some components of an AR system 90 that identifies the ability to properly align computer-generated virtual features with a physical environment viewed using 2D markers as landmarks, according to one embodiment. The AR system 90 includes an AR device 92, a computer-aided design (CAD) system 34, and one or more 2D markers 50 affixed to corresponding objects in the physical environment. The CAD system 34 includes a database that includes CAD model data representing models of objects that are components of a reference object (such as an airplane). The CAD model data includes 3D position data that specifies coordinates of corresponding 3D positions of the objects in a reference frame of the target object (e.g., an airplane). The AR system further includes a transceiver 36 operably coupled to the AR device 92 and a transceiver 40 operably coupled to the CAD system 34. The transceivers 36 and 40 are wirelessly coupled via respective antennas 38 and 42. The communication path enables the AR device 92 to receive the CAD model data from the CAD system 34.

[0077] The AR device 92 includes an imaging device 26 (e.g., a camera) having a field of view (FOV) that preferably surrounds the 2D marker 50; a display device 32 (e.g., Figure 5 The head mounted display 4 shown in the figure is capable of displaying virtual content on a transparent screen while the user is viewing a physical scene; a non-transitory tangible computer-readable storage medium (e.g., memory 94); and a computer system 24, communicatively coupled to the imaging device 26, the display device 32, and the memory 94. The memory 94 stores pairing list data and virtual content data, the pairing list data indicating the tag identifier on the 2D tag 50 and the object in the physical environment to which the corresponding 2D tag 50 is applied ( Figure 8 50 ), the virtual content data represents virtual content including virtual features of an object to which a corresponding 2D marker 50 is applied.

[0078] The computer system 24 is configured to perform the following operations. Image data from an image captured by the imaging device 26 is processed to obtain data representing the spatial position of the marker identifier of the 2D marker 50 applied to the object appearing in the captured image and the registration marker. The spatial position of the registration marker is then processed to calculate the offset of the current AR device to the marker. The computer system then searches the pairing list stored in the memory 94 to find the 3D position of the object associated with the marker identifier of the 2D marker 50 appearing in the captured image. The 3D position of the AR device 92 in the reference system of the physical environment is then calculated based on the offset of the current AR device to the marker and the 3D position of the object found in the pairing list. The computer system then controls the viewpoint of the display device 32 to display virtual content including virtual features of the object based on the offset of the current AR device to the marker and the 3D position of the object. Optionally, the imaging device 26 of the AR device 92 may also be used during the setup process.

[0079] According to other embodiments, a separate code scanning device (in Fig. 9A and Fig. 9B ) to obtain a digital tag code for the pairing step. The code scanning device can be a separate handheld device. The handheld device can be used during setup, but will not be used for the runtime portion. The handheld device can be configured to associate the encoded tag's embedded unique identifier data (e.g., "AA123") with the geometric model data of the 3D object (e.g., "w765.obi") and the spatial position of the associated 3D object, which includes the position and orientation of the 3D object relative to the aircraft coordinates (e.g., 4×4 transformation matrix or quaternion).

[0080] If a handheld device is used to associate a tag with a location, the device can be configured to communicate with a computer system ( Figure 8The 3D model may be loaded into and displayed in a 3D virtual environment running on a laptop or tablet PC into which the handheld device is inserted, for example. When the user scans the QR code (which is located on one of the windows), the data decoded by the scanner is received and stored on the PC. The user then also selects the appropriate 3D model in the 3D virtual environment (using a mouse or touch screen). In other words, the user instructs the paired application running on the PC that the scan data just acquired using the scanner will be associated with the 3D object just selected - or more specifically, that the 3D position of the selected 3D object will be associated with the scan data just acquired. It is determined by the person performing the task which 3D model in the 3D virtual environment is the appropriate 3D model to be selected. The person sees the virtual models on the screen and decides which model to select based on the most recently scanned markers in the physical environment.

[0081] According to one proposed implementation, the hardware includes a scanner and a PC / tablet computer; the software includes a scanning application hosted by the scanner and a 3D display application and a pairing application hosted by the PC / tablet computer. The PC / tablet computer receives the encoded tag's unique identifier data from the scanner and retrieves 3D position data associated with a selected 3D object in a virtual environment from a non-transitory tangible computer-readable storage medium. Optionally, the scanning application and the pairing application can be integrated with the 3D display application to make one unified software application that handles all of these steps.

[0082] According to an optional embodiment, instead of a handheld scanner, a head-mounted display with an integrated camera can be used for the pairing operation. The camera in the head-mounted display will be used to read the QR code (and the associated software will decode the QR code) to obtain the unique identifier data. The 3D model can be loaded and displayed in the head-mounted display instead of the tablet or laptop. Since the user does not have a mouse when using the head-mounted display, a different type of interaction will be used to select the 3D model that should be associated with the data from the QR code (acquired by the camera and image decoding application). For this selection aspect of the process, the head-mounted display typically has some type of gesture input; voice commands can also be used.

[0083] Fig. 9A and Fig. 9B 1 are isometric front and rear views of an exemplary handheld imaging device 18 that may be used during the setup process. The handheld imaging device 18 may be an iOS device (such as an iPod Touch manufactured by Apple Inc. of Cupertino, California). or ) or other "smartphones", tablet computers or other handheld computing devices. Fig. 9A and Fig. 9BIn the embodiment shown, the handheld imaging device 18 includes a display screen 12, a microprocessor 14 (in Fig. 9A ), a button 16 for activating image capture, and a line of sight 20 (indicated by a dotted line in Fig. 9B An integrated or attachable camera 28 (indicated by a dashed line in FIG. 1 ). In an alternative, image capture may be activated by pressing a virtual button displayed on the display screen 12.

[0084] The handheld imaging device 18 may also incorporate elements for providing wireless communication capabilities. Specifically, the handheld imaging device 18 may have onboard 3D graphics capabilities capable of running a 3D visualization application, or it may be coupled to a computer running a 3D visualization application (on a Fig. 9A and Fig. 9B The remote computer system has the ability to communicate with a 3D model database, such as Figure 8 The 3D model of the aircraft is stored in a non-transitory tangible computer-readable storage medium that is accessible by a computer system hosting a 3D visualization application. The 3D visualization application enables a graphical representation of the aircraft components in their physical form.

[0085] Fig.10 is a diagram showing a schematic diagram of the extent of the field of view 22 of a camera in a handheld imaging device 18 projected onto a portion of an interior surface 10 of a target object having a 2D (eg, rectangular) code pattern marking 50 applied thereon. Fig.10 In the embodiment shown, the code pattern mark 50 includes a flexible substrate on which a code pattern 48 is printed to form a landmark position for removable or semi-permanent placement on the inner surface 10 of the target object. The code pattern 48 includes three registration reference symbols 52a to 52c located at corresponding corners of the code pattern mark 50. The registration reference symbols 52a to 52c enable continuous relative position tracking. The flexible substrate of the code pattern mark 50 may include a sheet of paper, a film made of a polymer material, or other suitable printable flexible material. Alternatively, a portable inkjet printer (or similar device) may be used to print the code pattern 48 directly onto a substrate (not shown) that has been applied to the surface 10 (e.g., the inner surface of the window of the aircraft fuselage). The microprocessor 14 of the handheld imaging device 18 is configured to decode the code pattern 48 using the code pattern decoder software. The handheld imaging device 18 is also configured to communicate with the AR device remote computer system using a radio frequency signal 30.

[0086] Fig.11is a diagram showing an example of an alternative type of rectangular code pattern mark 50 having a data matrix code pattern 56 and three registration fiducial symbols 52a to 52c at respective corners of the code pattern mark 50. The registration fiducial symbols 52a to 52c are located on the mark label, but outside the data matrix portion of the code pattern 56, and can be used in the same manner as the registration fiducial symbols 52a to 52c of the QR code pattern mark. More specifically, the fiducial symbols 52a to 52c do not overlap with the data matrix portion of the code pattern 56, so that a blank space is defined between the fiducial symbols 52a to 52c and the data matrix portion of the code pattern 56.

[0087] Once the code pattern indicia 50 has been installed, the user can read each code pattern indicia 50 using a handheld imaging device 18 configured to read and decode the code. The logical coordinates returned by the decoding are then mapped into the correct aircraft coordinates, taking into account factors that may include the primary and secondary aircraft models and applicable assembly offsets. This landmark to location mapping (also referred to herein as a "pairing list") is created once for each set of factors required and prior to first use on an applicable aircraft.

[0088] Fig.12 is a diagram showing an example of a landmark-to-position map 54 incorporated into the payload of a QR code pattern. A marker having a landmark-to-position map 54 printed thereon can be placed anywhere on a reference object or separate from the reference object (including portable). The landmark-to-position map 54 is used to retrieve the associated physical aircraft coordinates of the detected marker in an aircraft coordinate system. These physical aircraft coordinates can then be fed into an AR application to set the position and orientation of the virtual content displayed by the AR device to align with the physical object being viewed by the user.

[0089] For landmark-to-position mapping 54, the QR code registration symbols in the corresponding corners of the markers are not important for this method and will be ignored when reading the mapping because the landmark-to-position mapping is not used for continuous tracking when reading the mapping. If a data matrix code pattern is used for this process instead of a QR code pattern, then Fig.11 The registration fiducial symbols 52a to 52c shown would not need to be included in the marking of the landmark-to-position mapping.

[0090] The pairing list embodied by the landmark-to-position map 54 associates the marker identifier data decoded from the imaged QR code pattern with the 3D position of the 3D object (selected from the virtual environment). Typically, the 3D position of the 3D object selected from the virtual environment corresponds to the position of the QR code pattern marker, for example, if the marker is placed at the center of the corresponding window, but in some embodiments, there may be a known offset between the center of the window and the center of the QR code pattern marker. If this is the case, the position stored in the pairing list will be adjusted by the offset.

[0091] According to one embodiment, the system uses a coded marker in the form of a QR code. A QR code consists of an array of pixel-like elements in a pattern on a white background, which can be read by an imaging device such as a camera. At runtime, the position of the QR code relative to the camera (which is part of the AR device) is obtained by solving the camera pose problem using well-known QR code-based AR processing. The process uses a "position" marker and an "alignment" marker embedded in each QR code. These markers are parts of the QR code marker that do not change, including (for example, in version 10 of the QR code) three relatively large "position" squares (also called "tracking boxes") in the corresponding corners of the square marker and six smaller "alignment" squares (also called "tracking boxes") arranged in a grid pattern on the marker. By using this alignment technique, the user can determine the relative position (position and orientation) of the camera of the AR device relative to the QR code marker. The user can then use the position of the marker relative to the reference system of the physical environment (e.g., aircraft coordinates) to obtain the absolute coordinates of the camera (AR device) defined in the reference system.

[0092] The imaging device 26 (see Figure 8 ) can be used to read the QR code information and pass that information on to other software components for additional processing. The other software components retrieve the absolute 3D position data from the pairing table (using the unique identifier acquired during the QR code scan), and calculate the relative position of the AR device 92 relative to the QR code pattern mark 50 (calculated by the software using the position and alignment mark data from the QR code scan), and then perform the aforementioned matrix multiplication to calculate the correct absolute position - which is then used to correct the estimate of the absolute position of the AR device 92.

[0093] In order to determine the position (position and orientation) of the AR device 92 relative to the target object, it is necessary to obtain the initial reference position information in the absolute coordinate system of the target object (such as the aircraft coordinates). Figure 1An array of QR code pattern markers 50 of the type shown enables establishing the relative position of the AR device 92 relative to the surface of the target object. The relative tracking function enabled by the registration fiducial symbols 52a to 52c in each code pattern 48 allows the distance and relative angle to the AR device 92 to be calculated. The code pattern 48 is within the field of view 8 of the camera of the AR device 92 (see Figure 5 ), and the data representing the unique landmark reference embedded in the code pattern 48 will be the reference position for subsequent movement. The relative tracking function enabled by the registration fiducial symbols 52a to 52c in the code pattern allows the distance to the AR device 92 and the relative angle of the line of sight of the camera incorporated in the AR device 92 to be calculated as the AR device 92 moves.

[0094] The described tracking functionality is particularly useful in situations where the AR device 92 has only discrete (i.e., discontinuous) access to the QR code data on the QR code pattern marker. In other words, the tracking method proposed herein does not require the QR code to be viewed all the time in order to be able to continuously track the AR device position. When the QR code is not in view, the internal AR device tracking handles the positioning. More specifically, the system of the preferred method continuously processes the input image data from the camera to find the QR code in the environment, but it does not require the QR code to be in view all the time to allow the AR device to track the position. The AR device only needs to intermittently view the QR code to correct for drift that occurs in the local positioning process (such as a process based on simultaneous localization and mapping (SLAM)) running on the AR device. In contrast, if at least one QR code is continuously visible (and readable) all the time, the system can use those code pattern markers for tracking, but the process returns to the default "discrete update" correction method proposed elsewhere in this article.

[0095] FIG. 13A to FIG. 13Cis a diagram showing a corresponding schematic diagram of the field of view of the AR device 92 projected onto a corresponding area of ​​the surface 10 having an array of QR code pattern markers 50a to 50d applied thereon according to an optional operation, in which the code pattern markers are continuously visible. Each code pattern has three registration fiducial symbols at corresponding corners to enable continuous determination of changes in the relative position and orientation of the AR device 92 as the AR device 92 moves throughout the environment. The algorithm for position and orientation tracking and processing can be similar to the algorithm disclosed in the ARToolKit supported by the Human Interface Technology Laboratory (HITLab) of the University of Washington, HIT Lab NZ of the University of Canterbury in New Zealand, and ARToolworks, Inc. in Seattle, Washington, or similar position and orientation determination software. ARToolKit uses computer vision techniques to calculate the real camera position and orientation relative to the code pattern marker. The camera of the AR device 92 captures a video of the target object. The software searches for any QR code position and alignment elements in each video frame. If a sufficient number of positions and alignment elements are found, the software algorithm uses mathematical methods to calculate the position of the camera relative to the pattern. ARToolKit gives the position of the code pattern marker in the camera coordinate system. For the embodiments described herein, software routines that employ these types of position tracking algorithms operate in the computer system 24 (e.g., microprocessor) of the AR device 92 (see Figure 8 ).

[0096] Three registration reference symbols 52a to 52c ( Figure 1 The relative position and orientation of the AR device 92 (as shown) relative to the AR device 92 provides data for determining the camera pose relative to the surface 10 of the target object. The camera pose can be represented by a coordinate transformation matrix. This continuously updated relative positioning information and the unique reference location identification data contained in the data payload area of ​​the code pattern read by the AR device 92 are used to provide continuous tracking in the absolute coordinate system of the target object (e.g., an aircraft). This tracking function allows positioning at any location inside the target object as long as at least one code pattern remains within the field of view 8 of the AR device 92 and at a distance that can be resolved by the AR device 92. Given the absolute position and orientation of the AR device 92 relative to the target object, data from an onboard or off-board database associated with the location of interest can be retrieved. The data can be in a variety of forms: text, icons, photos, non-destructive inspection scans, videos, and 3D models, which can then be displayed on the display device 32 of the AR device 92 (see Figure 8 ).

[0097] Reference again FIG. 13A to FIG. 13C, using a plurality of code pattern marks 50a to 50d spaced apart along the inner surface 10 of a target object (such as an aircraft fuselage) allows the field of view of the AR device 92 to move along the target object, wherein the associated virtual content presented on the display device 32 is adjusted to indicate the changing position of the moving AR device 92. The code pattern marks 50a to 50d can be placed on the inner surface 10 of the target object at intervals to form an array that allows the camera ( FIG. 13A to FIG. 13C (not shown) when the AR device 92 moves or translates along the inner surface 10 to view the code pattern for position correction. Fig.13A In the initial position of the AR device 92 shown, the code pattern marker 50b is within the field of view 8a of the camera of the AR device 92. Based on the unique landmark reference information incorporated in the code pattern marker 50b, the tracking function disclosed herein is used to determine the relative positioning (i.e., position and orientation) of the AR device 92 relative to the absolute coordinate system of the inner surface 10 of the target object. Thereafter, when the AR device 92 moves to Fig. 13B In the second position shown, the code pattern mark 50c enters the field of view 8b of the camera of the AR device 92, thereby allowing continuous real-time updating of the relative position and orientation of the AR device 92. Similarly, when the AR device 92 is moved to Fig. 13C In the third position shown, the code pattern mark 50d enters the field of view 8c of the camera of the AR device 92. If more than one QR code pattern mark is in the field of view 8 of the camera, one or another QR code pattern mark can be selected, or a position can be inserted between two QR code pattern marks, which enables continuous scanning over a larger area.

[0098] As previously mentioned, the virtual content displayed by the AR device 92 can be a 3D solid model (such as a 3D CAD model) or other forms, such as 3D lines, texture mapping, or even text placed at appropriate 3D locations. According to some embodiments, the virtual content includes virtual features of objects with 2D markers applied thereto.

[0099] A simplified approach that does not require a 3D model is to use a texture map of a 2D marker scaled to the size of the physical marker and placed at the appropriate location in the virtual 3D environment. The texture map image or a portion thereof can be made transparent or semi-transparent in the visualization display to allow the physical QR code pattern to be partially visible through the AR image overlay. In the visualization environment of the AR device, the texture map will overlap the physical marker to help the user confirm that the system is correctly aligned. Fig.14is a diagram showing an overlay of virtual content in the form of a texture map 51 representing a 2D machine-readable code pattern on a 2D marker applied to an object in a physical environment, which texture map 51 can be superimposed on a view of a 2D marker 50 having the machine-readable code pattern printed thereon. In other words, the virtual code pattern will be superimposed on the real code pattern visible in the user's field of view. Any misalignment of the virtual code pattern relative to the visible code pattern indicates to the user that the AR device may not be properly calibrated and needs to be recalibrated.

[0100] Another option is to display virtual content in the form of geometric symbols that have a unique relationship to the boundaries of the 2D markers or to the boundaries of the objects to which the 2D markers are applied when the AR device is properly calibrated relative to the coordinate system of the physical environment. FIG. 15A to FIG. 15F 4 is a diagram showing the superposition of the corresponding geometric symbols on the view of the object to which the 2D marker has been attached (in this example, the window 58 surrounded by the window frame 46). Each geometric symbol is scaled and placed in the appropriate position in the virtual 3D environment. In the visualization environment of the AR device, the geometric symbols will help the user confirm that the AR system is properly aligned.

[0101] according to Fig.15A In the first embodiment shown, the virtual feature is a geometric symbol 96a that is shaped and scaled to coincide with the inner boundary of the window frame 46 when the virtual content is properly aligned with the physical environment. Fig. 15B In the second embodiment shown, the virtual feature is a geometric symbol 96b that is shaped (e.g., rectangular) and scaled to coincide with the boundaries of the 2D marker applied to the window 58 when the virtual content is properly aligned with the physical environment. Fig. 15C In the third embodiment shown, the virtual feature is a geometric symbol 96c that is shaped (eg, a diamond) and scaled to be inscribed within the inner boundaries of the window frame 46 when the virtual content is properly aligned with the physical environment.

[0102] Any variation of this virtual content can be used together. Fig.15D In the fourth embodiment shown, the virtual features include a geometric symbol 96a shaped and scaled to coincide with the inner boundary of the window frame 46 and a geometric symbol 98a shaped and scaled to coincide with the outer boundary of the window frame 46 when the virtual content is properly aligned with the physical environment. Fig.15E In the fifth embodiment shown, the virtual features include a geometric symbol 96b shaped (e.g., rectangular) and scaled to coincide with the boundaries of the 2D marker applied to the window 58 and a geometric symbol 98b shaped (e.g., rectangular) and scaled to allow the outer boundaries of the window frame 46 to be inscribed within the geometric symbol 98b when the virtual content is properly aligned with the physical environment. Fig.15FIn the sixth embodiment shown, the virtual features include a geometric symbol 96c shaped (e.g., a diamond) and scaled to be inscribed within the inner boundary of the window frame 46 and a geometric symbol 98b shaped (e.g., a rectangle) and scaled to allow the outer boundary of the window frame 46 to be inscribed within the geometric symbol 98b when the virtual content is properly aligned with the physical environment.

[0103] The AR application disclosed herein runs on a computer system that is configured to receive a list of marker identifier to 3D position pairings and convert that information into a data string that specifies the absolute position of the AR device (i.e., specifies the viewpoint). The command message is sent to the AR application, which uses the absolute position of the AR device 92 to display virtual content that is positioned and oriented to align with objects in the physical environment that the user is viewing. The virtual content shown in these displays can be any form of computer-generated data. Some virtual content can be 3D solid models (such as 3D CAD models); other forms can be 3D lines, texture maps, or even text placed at appropriate 3D locations.

[0104] AR applications can run on many different types of computer systems. As used in the present disclosure, the term "computer system" should be broadly interpreted as including a digital data processing system having at least one computer or processor, and the digital data processing system may have multiple computers or processors that communicate with each other directly through a network or bus. As used in the previous sentence, the terms "computer" and "processor" both refer to a device including a processing unit (e.g., a central processing unit, an integrated circuit, or an arithmetic logic unit). Examples of suitable computer systems include smartphones, tablet computers, laptop computers, desktop computers, workstation computers, smart watches, head-mounted displays, or other wearable displays. The computer system may include at least two processors connected to each other, one of which is a graphics processor.

[0105] Fig.16 1 is a block diagram identifying components of a system 100 suitable for hosting an augmented reality application that can communicate with a handheld imaging device of the type described above. Processor 102 is generally any computer hardware capable of executing a computer program that processes data. Processor 102 may include electronic circuits, some of which may be packaged as an integrated circuit (e.g., a chip) or multiple interconnected integrated circuits. Processor 102 may be configured to execute a computer program, which may be stored as computer-readable program code on the processor or otherwise stored in memory 106. In alternative embodiments, processor 102 may be embodied as or otherwise include one or more application specific integrated circuits, field programmable gate arrays, and the like.

[0106] Storage device 104 is generally any hardware capable of permanently storing digital data and computer programs (e.g., computer readable program code and 3D model files). Such storage device may take the form of a hard drive, solid-state drive, optical disk, magnetic tape, or some combination thereof.

[0107] Memory 106 is generally any computer hardware capable of temporarily storing digital data and computer programs (e.g., computer readable program code and 3D model files). Memory 106 may include volatile and non-volatile memory and may be fixed or removable. Examples of suitable memory include random access memory, read-only memory, hard drive, flash memory, thumb drive, SD card, removable computer disk, optical disk, or some combination thereof.

[0108] In various cases, storage device 104 and memory 106 may be referred to as tangible computer-readable storage media. Tangible computer-readable storage media are non-transitory devices capable of storing information and are distinguished from computer-readable transmission media such as electronic transient signals that can carry information from one location to another.

[0109] The system 100 further includes one or more input devices 112 (such as Fig. 9A and Fig. 9B The handheld imaging device 18 is shown. The input device 112 can be wired or wireless and can be configured to receive information from a user into the device, such as for processing, storage and / or display. Examples of suitable user input devices include a mouse, a microphone, an image or video capture device, a keyboard or keypad, a joystick, a touch-sensitive surface (separate from or integrated into a touch screen), a biometric sensor, etc.

[0110] In response to input of an augmented reality activation command from a user via input device 112, processor 102 retrieves a file containing virtual content from storage device 104 and temporarily stores it in memory 106. Processor 102 selectively processes the virtual content stored in memory 106 according to other user input received via input device 112 or according to instructions read from augmented reality application software stored in memory 106.

[0111] In addition to the storage device 104 and the memory 106, the processor 102 is also connected to a graphics processor 108, which is in turn connected to a display device 110. The graphics processor 108 is configured to render a digital 3D model of a structural product composed of multiple parts and having corresponding viewpoints for display in response to receiving a coordinate transformation matrix from the processor 102. More specifically, the graphics processor 108 processes the 3D model data and outputs the pixel data to a digital image or raster graphics image file for display on a display device. In an alternative solution, high-end systems typically have a central processing unit (CPU) and a graphics processing unit (GPU), but some systems (such as low-end computing systems or smart phones) may have a single processor with an embedded graphics processing chip (commonly referred to as "integrated" graphics) or even "software" rendering that uses the CPU to render images.

[0112] The processor 102 may also be connected to a network interface 114, such as for sending information to and / or receiving information from other devices, networks, etc. The network interface 114 may be configured to send and / or receive information via a physical (wired) and / or wireless communication link. Examples of suitable communication interfaces include a network interface controller (NIC), a wireless NIC, etc.

[0113] The display device 110 may be configured to present or otherwise display information to a user. Suitable examples include liquid crystal displays, light emitting diode displays, plasma display panels, laser-based displays (including retinal displays), and the like.

[0114] Although the present disclosure is primarily focused on tasks related to aircraft, the use of the technology described herein is not limited to this particular field. Other manufacturing, architecture, and construction applications using augmented reality can take advantage of the concept.

[0115] Although systems, methods, and devices for aligning virtual content generated by an AR application with a scene being viewed by a user in an operating environment have been described with reference to various embodiments, those skilled in the art will appreciate that various changes may be made and equivalents may be substituted for elements thereof without departing from the teachings herein. In addition, many modifications may be made to adapt the concepts and reductions of the practices disclosed herein to particular circumstances. Therefore, the subject matter intended to be covered by the claims is not limited to the disclosed embodiments.

[0116] As used herein, the term "computer system" should be interpreted broadly to include a system having at least one computer or processor, and the system may have multiple computers or processors communicatively coupled via a network or bus. As used in the preceding sentences, both the terms "computer" and "processor" refer to a device that includes a processing unit (e.g., a central processing unit) and some form of memory (e.g., a non-transitory tangible computer-readable storage medium) for storing programs (e.g., encoded instructions) that can be read by the processing unit.

[0117] The methods described herein may be encoded as executable instructions embodied in a non-transitory tangible computer-readable storage medium, including but not limited to a storage device and / or a memory device. When executed by a processor or a computer, such instructions cause the processor or computer to perform at least a portion of the methods described herein.

[0118] The method claims set forth below should not be interpreted as requiring that the steps described therein be performed in alphabetical order (any alphabetical order in the claims is used only for the purpose of referencing previously described steps) or in the order in which they are recited, unless the claim language expressly specifies or states a condition indicating a particular order in which some or all of those steps are performed. Unless the claim language expressly states a condition excluding such an interpretation, a method claim should not be interpreted to exclude any portion of two or more steps from being performed simultaneously or alternately.

[0119] Furthermore, the present disclosure includes embodiments according to the following items:

[0120] Item 1. A method for providing instant access to data for calibrating an augmented reality device relative to an operating environment, the method comprising:

[0121] (a) creating a plurality of two-dimensional (2D) machine-readable code patterns on respective 2D tags using a first symbol representing a tag identifier that uniquely identifies the respective 2D tags and a second symbol representing a registration mark;

[0122] (b) applying a corresponding 2D marker to a selected location on a corresponding object in the physical environment;

[0123] (c) capturing image data representing corresponding 2D markers in the physical environment;

[0124] (d) processing the image data to obtain digital data representing the corresponding tag identifier;

[0125] (e) loading a three-dimensional (3D) model representing the correct positioning of the corresponding object into a 3D visualization environment;

[0126] (f) acquiring digital data representing the 3D position of a corresponding object from the 3D model having the 2D machine-readable code pattern attached thereto from the 3D virtual environment;

[0127] (g) pairing the marker identifiers of the corresponding 2D markers with the 3D positions of the corresponding objects from the 3D model;

[0128] (h) generating a data file containing pair list data representing a list of pairs of marker identifiers and 3D positions for use at runtime; and

[0129] (i) Converting the pairing list data into a form suitable for instant loading onto an augmented reality (AR) device.

[0130] Item 2. The method according to item 1, wherein step (i) comprises:

[0131] creating a 2D machine-readable code pattern on the 2D mark using symbols representing the paired list data; and

[0132] Attach 2D markers to structures in the physical environment.

[0133] Item 3. The method according to item 1, wherein step (i) comprises:

[0134] formatting the pairing list data for wireless transmission; and

[0135] Transmits short wavelength ultra-high frequency radio waves modulated with information representing the pairing list data.

[0136] Item 4. The method of Item 1, wherein the 2D machine-readable code pattern is a QR code.

[0137] Item 5. The method of Item 1, wherein the physical environment is an aircraft and the object is a window of the aircraft.

[0138] Item 6. The method according to Item 1, further comprising:

[0139] (i) loading the pairing list data into the AR device;

[0140] (k) loading virtual content into the AR device, the virtual content including virtual features of the object having the 2D marker applied thereto;

[0141] (1) Viewing objects in the physical environment on an AR device;

[0142] (m) capturing image data representing a 2D marker applied to an object;

[0143] (n) processing the image data to obtain data representing the spatial positions of the marker identifiers and the registration markers of the 2D markers applied on the object;

[0144] (o) processing the spatial position of the registration marker to calculate the offset from the current AR device to the marker;

[0145] (p) finding the 3D position of the object associated with the tag identifier of the 2D tag in the pairing list;

[0146] (q) calculating the 3D position of the AR device in the reference frame of the physical environment based on the current AR device-to-marker offset and the 3D position of the object found in the pairing list; and

[0147] (r) Display virtual content with a viewpoint based on the current AR device-to-marker offset and the 3D position of the object found in the pairing list.

[0148] Item 7. A method according to Item 6, wherein step (q) includes correcting the position estimate of the positioning process of the AR device.

[0149] Item 8. The method according to Item 6, wherein:

[0150] Step (i) comprises creating a 2D machine-readable code pattern on a 2D tag using a symbol representing the paired list data, and attaching the 2D tag to a structure in a physical environment; and

[0151] Step (i) includes optically reading the 2D machine-readable code pattern to acquire pairing list data, and storing the acquired pairing list data in a non-transitory tangible computer-readable storage medium of the AR device.

[0152] Item 9. The method according to Item 6, wherein:

[0153] Step (i) comprises formatting the pairing list data for wireless transmission and then transmitting a short wavelength ultra-high frequency radio wave modulated with information representing the pairing list data; and

[0154] Step (i) includes receiving electromagnetic waves, demodulating the received electromagnetic waves to obtain pairing list data, and storing the pairing list data in a non-transitory tangible computer-readable storage medium of the AR device.

[0155] Item 10. The method of Item 6, wherein the virtual feature is a scaled texture map representing a 2D machine-readable code pattern applied to a 2D marking on the object.

[0156] Item 11. A method according to Item 6, wherein the virtual feature is a geometric symbol shaped and scaled to coincide with the boundaries of a 2D marker applied to the object or the boundaries of the object.

[0157] Item 12. A method according to Item 6, wherein the virtual feature is a geometric symbol shaped and scaled to be inscribed within the boundaries of the object.

[0158] Item 13. A method according to Item 6, wherein the virtual feature is a geometric symbol that is shaped and scaled to allow the boundary of the object to be inscribed within the geometric symbol.

[0159] Item 14. A method for registering virtual content generated by an augmented reality application with a scene being viewed by a user in an operating environment, the method comprising:

[0160] (a) creating a plurality of 2D machine-readable code patterns on respective 2D tags using a first symbol representing a tag identifier that uniquely identifies the respective 2D tags and a second symbol representing a registration mark;

[0161] (b) applying a corresponding 2D marker to a selected location on a corresponding object in the physical environment;

[0162] (c) loading pairing list data into an augmented reality (AR) device, wherein the pairing list data represents pairings of marker identifiers on corresponding 2D markers and 3D positions of corresponding objects to which the 2D markers are applied;

[0163] (d) loading virtual content into the AR device, the virtual content including virtual features of the object having the 2D marker applied thereto;

[0164] (e) viewing objects in the physical environment on the AR device;

[0165] (f) capturing image data representing a 2D marker applied to an object;

[0166] (g) processing the image data to obtain data representing the spatial positions of the marker identifiers and the registration markers of the 2D markers applied on the object;

[0167] (h) processing the spatial position of the registration marker to calculate the offset from the current AR device to the marker;

[0168] (i) finding the 3D position of the object associated with the marker identifier of the 2D marker in the pairing list;

[0169] (i) calculating the 3D position of the AR device in the reference frame of the physical environment based on the current AR device-to-marker offset and the 3D position of the object found in the pairing list; and

[0170] (k) Displaying virtual content with a viewpoint based on the current AR device-to-marker offset and the 3D position of the object found in the pairing list.

[0171] Item 15. A method according to Item 14, wherein step (i) includes correcting the position estimate of the positioning process of the AR device.

[0172] Item 16. The method according to Item 14, further comprising:

[0173] creating a 2D machine-readable code pattern on at least one 2D marking using symbols representing the paired list data; and

[0174] Attach 2D markers to structures in the physical environment,

[0175] Wherein, step (c) includes optically reading the 2D machine-readable code pattern to obtain the pairing list data, and storing the obtained pairing list data in a non-transitory tangible computer-readable storage medium of the AR device.

[0176] Item 17. The method according to Item 14, further comprising:

[0177] formatting the pairing list data for wireless transmission; and

[0178] Transmitting short-wavelength ultra-high frequency radio waves modulated with information representing the pairing list data,

[0179] Wherein, step (c) includes receiving electromagnetic waves, demodulating the received electromagnetic waves to obtain pairing list data, and storing the pairing list data in a non-transitory tangible computer-readable storage medium of the AR device.

[0180] Item 18. The method of Item 14, wherein the virtual feature is a scaled texture map representing a 2D machine-readable code pattern on a 2D marking applied to the object.

[0181] Item 19. The method of Item 14, wherein the virtual feature is a geometric symbol selected from the group consisting of:

[0182] A geometric symbol that is shaped and scaled to conform to the boundaries of a 2D marker applied to the object;

[0183] A geometric symbol that is shaped and scaled to conform to the object's boundaries;

[0184] A geometric symbol shaped and scaled to be inscribed within the object's boundaries; and

[0185] A geometric symbol that is shaped and scaled to allow the bounds of an object to be inscribed within the geometric symbol.

[0186] Item 20. An augmented reality (AR) device, comprising an imaging device, a display device, a non-transitory tangible computer-readable storage medium, and a computer system communicatively coupled to the imaging device, the display device, and the non-transitory tangible computer-readable storage medium, wherein:

[0187] a non-transitory tangible computer-readable storage medium storing pairing list data representing pairings of tag identifiers on 2D tags with 3D locations of objects in a physical environment to which the corresponding 2D tags are applied, and virtual content data representing virtual content including virtual features of the objects to which the corresponding 2D tags are applied; and

[0188] The computer system is configured to perform operations including:

[0189] (a) processing image data from an image captured by an imaging device to obtain data representing marker identifiers of 2D markers applied on an object that appear in the captured image and spatial locations of registration markers;

[0190] (b) processing the spatial position of the registration marker to calculate the offset from the current AR device to the marker;

[0191] (c) finding the 3D position of the object associated with the marker identifier of the 2D marker appearing in the captured image in a pairing list stored in a non-transitory tangible computer readable storage medium;

[0192] (d) calculating the 3D position of the AR device in the physical environment reference frame based on the current AR device-to-marker offset and the 3D position of the object found in the pairing list; and

[0193] (e) Displaying virtual content including virtual features of the object at a viewpoint based on the current AR device-to-marker offset and the 3D position of the object.

[0194] Item 21. A system according to Item 20, wherein the virtual feature is a scaled texture map representing a 2D machine-readable code pattern on a 2D tag applied to the object.

[0195] Item 22. The system of Item 20, wherein the virtual feature is a geometric symbol selected from the group consisting of:

[0196] A geometric symbol that is shaped and scaled to conform to the boundaries of a 2D marker applied to the object;

[0197] A geometric symbol that is shaped and scaled to conform to the object's boundaries;

[0198] A geometric symbol shaped and scaled to be inscribed within the object's boundaries; and

[0199] A geometric symbol that is shaped and scaled to allow the bounds of an object to be inscribed within the geometric symbol.

[0200] As used in this disclosure, the term "properly positioned" refers to matching the position of an object in the physical environment. This is important because corresponding 3D models in the virtual environment may not always be placed where they are in the fully assembled object.

Claims

1. A method for providing instant access to data for calibrating an augmented reality device relative to an operating environment, the method comprising: (a) creating a plurality of two-dimensional machine-readable code patterns on the corresponding two-dimensional marks using a first symbol representing a mark identifier that uniquely identifies the corresponding two-dimensional mark and a second symbol representing a registration mark, wherein the two-dimensional machine-readable code pattern is a QR code; (b) applying the respective two-dimensional marking to a selected location on the respective object in the physical environment; (c) capturing image data representing the corresponding two-dimensional marker in the physical environment; (d) processing the image data to obtain digital data representing a corresponding tag identifier; (e) loading a suitably positioned three-dimensional model representing the respective object into the three-dimensional visualization environment; (f) acquiring, from the three-dimensional visualization environment, digital data representing the three-dimensional position of the corresponding object from the three-dimensional model to which the two-dimensional machine-readable code pattern is attached; (g) pairing a marker identifier of the corresponding two-dimensional marker with a three-dimensional position of the corresponding object from the three-dimensional model; (h) generating a data file containing pairing list data representing a pairing list of the marker identifiers and the three-dimensional positions for use at runtime; (i) converting the pairing list data into a form suitable for immediate loading onto the augmented reality device; (j) loading the pairing list data into the augmented reality device; (k) loading virtual content into the augmented reality device, the virtual content including virtual features of the object to which the two-dimensional marker is applied; (l) viewing the object in the physical environment on the augmented reality device; (m) capturing image data representing the two-dimensional marking applied to the object; (n) processing the image data to obtain data representing the spatial positions of the marker identifiers and registration markers of the two-dimensional marker applied on the object; (o) processing the spatial position of the registration marker to calculate the offset from the current augmented reality device to the marker; (p) finding in the pairing list the three-dimensional position of the object associated with the tag identifier of the two-dimensional tag; (q) calculating a three-dimensional position of the augmented reality device in a reference frame of the physical environment based on an offset of the current augmented reality device to the marker and the three-dimensional position of the object found in the pairing list; and (r) displaying the virtual content with a viewpoint based on an offset of the current augmented reality device to the marker and a three-dimensional position of the object found in the pairing list.

2. The method according to claim 1, wherein: Step (i) comprises: creating the two-dimensional machine-readable code pattern on the two-dimensional mark using symbols representing the pairing list data; and The two-dimensional marker is attached to a structure in the physical environment.

3. The method according to claim 1, wherein: The physical environment is an airplane, and the object is a window of the airplane.

4. The method according to claim 1, wherein: Step (i) comprises creating the two-dimensional machine-readable code pattern on the two-dimensional tag using symbols representing the pairing list data, and attaching the two-dimensional tag to a structure in the physical environment; and Step (i) includes optically reading the two-dimensional machine-readable code pattern to acquire the pairing list data, and storing the acquired pairing list data in a non-transitory tangible computer-readable storage medium of the augmented reality device.

5. The method according to claim 1, wherein: Step (i) comprises formatting the pairing list data for wireless transmission and then transmitting a short wavelength ultra-high frequency radio wave modulated with information representing the pairing list data; and Step (i) includes receiving electromagnetic waves, demodulating the received electromagnetic waves to obtain the pairing list data, and storing the pairing list data in a non-transitory tangible computer-readable storage medium of the augmented reality device.

6. The method according to claim 1, wherein: The virtual feature is a scaled texture map representing the two-dimensional machine-readable code pattern applied on the two-dimensional marking on the object.

7. The method according to claim 1, wherein: The virtual feature is a geometric symbol that is shaped and scaled to coincide with the boundaries of the two-dimensional mark applied to the object or the boundaries of the object.

8. The method according to claim 1, wherein: The virtual feature is a geometric symbol shaped and scaled to be inscribed within the boundary of the object, or the virtual feature is a geometric symbol shaped and scaled to allow the boundary of the object to be inscribed within the geometric symbol.

9. An augmented reality system comprising an imaging device, a display device, a non-transitory tangible computer-readable storage medium, and a computer system communicatively coupled to the imaging device, the display device, and the non-transitory tangible computer-readable storage medium, wherein: The non-transitory tangible computer-readable storage medium stores pairing list data representing pairings of tag identifiers on two-dimensional tags and three-dimensional locations of objects in a physical environment to which the corresponding two-dimensional tags are applied, and virtual content data representing virtual content including virtual features of the objects to which the corresponding two-dimensional tags are applied; and The augmented reality system is configured to perform operations including: (a) creating a plurality of two-dimensional machine-readable code patterns on the corresponding two-dimensional marks using a first symbol representing a mark identifier that uniquely identifies the corresponding two-dimensional mark and a second symbol representing a registration mark, wherein the two-dimensional machine-readable code pattern is a QR code; (b) applying the respective two-dimensional marking to a selected location on the respective object in the physical environment; (c) capturing image data representing the corresponding two-dimensional marker in the physical environment; (d) processing the image data to obtain digital data representing a corresponding tag identifier; (e) loading a suitably positioned three-dimensional model representing the respective object into the three-dimensional visualization environment; (f) acquiring, from the three-dimensional visualization environment, digital data representing the three-dimensional position of the corresponding object from the three-dimensional model to which the two-dimensional machine-readable code pattern is attached; (g) pairing a marker identifier of the corresponding two-dimensional marker with a three-dimensional position of the corresponding object from the three-dimensional model; (h) generating a data file containing pairing list data representing a pairing list of the marker identifiers and the three-dimensional positions for use at runtime; (i) converting the pairing list data into a form suitable for instant loading onto an augmented reality device; (i) loading the pairing list data into the augmented reality device; (k) loading virtual content into the augmented reality device, the virtual content including virtual features of the object to which the two-dimensional marker is applied; (l) viewing the object in the physical environment on the augmented reality device; (m) capturing image data representing the two-dimensional marking applied to the object; (n) processing image data from an image captured by the imaging device to obtain data representing the spatial locations of marker identifiers and registration markers of the two-dimensional markers applied on the object that appear in the captured image; (o) processing the spatial position of the registration marker to calculate the offset from the current augmented reality device to the marker; (p) finding the three-dimensional position of the object associated with the marker identifier of the two-dimensional marker appearing in the captured image in a pairing list stored in the non-transitory tangible computer-readable storage medium; (q) calculating a three-dimensional position of the augmented reality device in a reference frame of the physical environment based on an offset of the current augmented reality device to the marker and the three-dimensional position of the object found in the pairing list; and (r) displaying virtual content including a virtual feature of the object at a viewpoint based on the offset of the current augmented reality device to the marker and the three-dimensional position of the object, wherein the virtual feature is a scaled texture map representing a two-dimensional machine-readable code pattern on the two-dimensional marker applied to the object.

Images