Spatial localization using augmented reality

Abstract

Spatial localization using augmented reality. Methods and system for locating an event on an aircraft using augmented reality content, a variety of ultrasonic devices configured in mesh topology, and deep learning.

Description

1 / 19 Spatial localization using augmented reality. Cross-reference to related applications.

[001] None. DECLARATION REGARDING RESEARCH OR DEVELOPMENT SPONSORED BY THE FEDERAL GOVERNMENT

[002] None. FIELD OF THE INVENTION

[003] The non-limiting technology herein relates to the use of an ultrasound sensor network (hereinafter referred to as Ultrasound) to automatically detect the user's location coordinates in an indoor Augmented Reality (AR) environment, including mobile devices, enabling the user to receive real-time location information; and the use of Deep Learning methods (hereinafter referred to as DL) to improve the aircraft maintenance process by guiding technicians to quickly, accurately, and precisely locate the elements that need to be accessed during maintenance. In other respects, the technology herein relates to methods and systems for locating and navigating an environment using ultrasound, in some modalities in combination with Deep Learning / artificial intelligence image pattern recognition, to assist a user in using augmented reality content on a display device. CONTEXT

[004] Currently, the aircraft maintenance business is facing difficulties in obtaining maintenance technicians to perform the necessary tasks. In many Petition 870200133836, dated 10 / 23 / 2020, page 14 / 143 In 2 out of 19 cases, technicians have inadequate knowledge about the aircraft that need to be repaired; this makes the maintenance task more expensive, as it may take longer or require more than one mechanic to perform the task.

[005] To perform maintenance tasks, the mechanic should normally first check if there is any available information about the aircraft requiring maintenance action. If there is no advanced information about the need for aircraft maintenance, the mechanic should check the aircraft logbook to see if there is any pending maintenance to be performed. In situations where there is a problem with the aircraft, the first approach is to try to correct the problem. In the case of problems that are not simple, it is possible to check the dispatch of the activity using the Minimum Equipment List (MEL), used to determine the aircraft's continued airworthiness. If the defective equipment does not appear on the MEL, it is necessary to perform troubleshooting, i.e., try to resolve the problem through a systematic search for the root cause of the component problem or its replacement.

[006] This troubleshooting procedure should generally be performed until the problem can be resolved and the aircraft is operational again. If there is a problem with the aircraft, in which the mechanic must perform maintenance procedures, it is necessary to know the location of specific aircraft components that may have failed or require attention. Petition 870200133836, dated 10 / 23 / 2020, page 15 / 143 3 / 19 In addition to obtaining the parts, equipment, and materials necessary to perform this activity. The necessary maintenance information can be found in the aircraft maintenance manual (maintenance manuals may contain, for example, removal / installation procedures, troubleshooting, activation / deactivation, etc.) and can be accessed by the mechanic through a printed document and / or digital file. The mechanic should normally follow all maintenance steps described in the aircraft maintenance manual to ensure proper equipment maintenance.

[007] From the above, it becomes apparent that prior knowledge of the aircraft is necessary in order to quickly locate and service the component that will need maintenance. In many cases, the components that need to be located are not easily found, as they may be hidden inside or behind other aircraft components, such as panel covers. The faster the maintenance location is found, the faster maintenance activities can begin.

[008] The difficulty of locating a component can be reduced by coupling a geolocation system between the equipment and the mechanic. Geographic location is used for various purposes, for example: to navigate in unfamiliar environments, locate objects and identify places, among other things. However, there is no technology suitable for aircraft maintenance that offers high positioning accuracy simultaneously in indoor and outdoor environments. Or Petition 870200133836, dated 10 / 23 / 2020, page 16 / 143 4 / 19 In other words, there is no solution that is reliable, effective and efficient for such purposes.

[009] Currently, many technologies exist that enable location tracking. Most often, the solution used is based on GPS systems that provide highly accurate absolute position geocoordinates. However, GPS has limitations when used indoors or under obstructions, such as in the cargo compartment of an aircraft or under the fuselage of an aircraft. For GPS to function properly, the receiver needs to be within the line of sight of the GPS satellites. Thus, to overcome the shortcomings of GPS systems, in recent decades, several approaches have emerged proposing hardware and software solutions adapted for indoor location tracking. Meanwhile, high location accuracy is useful for correctly locating the item being searched for and for not directing the maintenance user to the wrong location, so as not to delay the execution of a maintenance task. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The following detailed description of exemplary, non-limiting illustrative embodiments should be read in conjunction with the drawings of which:

[0011] FIG. 1 illustrates examples of non-limiting components (hardware, software, server) in a prior art AR-based system.

[0012] FIG. 2 shows a non-limiting embodiment of a basic non-limiting AR processing flow example.

[0013] FIGS. 3A, 3B and 3C show examples not Petition 870200133836, dated 10 / 23 / 2020, page 17 / 143 5 / 19 limitations on how events are captured and how they are subsequently used in non-limiting technology.

[0014] FIG. 4 shows an example of a non-limiting flowchart of the methodology used in Figures 3A, 3B and 3C.

[0015] FIGS. 5A and 5B are non-limiting examples where non-limiting technology is applied.

[0016] FIG. 6 illustrates a non-limiting embodiment of a sensor array. DETAILED DESCRIPTION OF PREFERRED MODALITIES

[0017] Examples of non-limiting modalities find application in improving the aircraft maintenance process and in quickly guiding technicians to locate the elements that need to be accessed during maintenance, with accuracy and precision.

[0018] The proposed technology is suitable for providing maintenance for a variety of environments and vehicles, not limited to: indoor environments, complex structures, aircraft, automobiles, and vessels. More specifically, the preferred modalities of the technology in a potential application are incorporated into an aircraft maintenance system.

[0019] Ultrasound technology can be used, among other applications, to locate users in an environment with great precision. As is known, the operating principle is based on the response time of the propagation of ultra-high frequency sound waves emitted by transceiver devices. The sensors measure the time of flight or the time of arrival (absolute or differential) of the emitted ultrasonic pulses. The distance can be calculated for each path between an emitter(s) and the Petition 870200133836, dated 10 / 23 / 2020, page 18 / 143 6 / 19 (s) sensor(s) based on the speed of sound. The user's position can be obtained through trilateration of signals from transceiver devices installed in the environment and on the user or on a device the user carries. While this ultrasound technology has been used in the past to determine the position of a user's body part (e.g., a hand) relative to a series of sensors and / or emitters (see the Nintendo Power Glove manufactured by Mattel as an example), current non-limiting technology extends these approaches to make it useful in aircraft maintenance contexts.

[0020] Augmented Reality (AR) is a real-time representation that combines the real physical world and the virtual world into a common user interface. Generally, virtual information is processed in the real physical world using an augmented reality device, which can consist of a display device such as a portable monitor, goggles, or glasses. One AR approach uses transparent lenses to allow the user to see the real world through the lenses while displaying virtual objects in real-world scenes. Another AR approach uses a camera to capture the real-world scene and combines the captured camera video stream with one or more virtual artificial display objects that appear to the user as part of the real-world scene. Yet another AR approach models the real world using 3D graphics techniques and displays scenes defined by a model of the world, plus additional virtual objects that are not part of the world (this approach Petition 870200133836, dated 10 / 23 / 2020, page 19 / 143 7 / 19 is sometimes called mixed reality.

[0021] There are two interrelated elements in most AR-based systems. The first element comprises the hardware and software components used to implement the AR-based system. The second element comprises ways in which AR is implemented in a real-world environment. Three components used in many AR-based systems include: Hardware (100), Software (200), and Server (300). Figure 1 explains the relationship between the three components and how they make AR-based systems work.

[0022] The hardware (100) in an AR device typically uses modules such as Display Devices (102) (e.g., head-mounted display (HMD), smartphone screen, glasses, etc.), Input Devices (104) (e.g., sensors such as: camera, infrared sensor, depth sensor, GPS, gyroscope, accelerometer, etc.) and a Processor (106). The sensors (105) detect the position and orientation of the display device (102) in or relative to an environment.

[0023] The software aspect (110) of an AR device is used to process and inject virtual images into the real world. The software (110) (e.g., StudioMax, Cinema4D, AutoCAD 3D, etc.) is typically tasked with generating virtual images used to overlay or combine live images.

[0024] As an AR device requests certain virtual images, the server (120) (e.g., web, cloud, etc.) retrieves and sends the virtual images to the AR devices and is also normally Petition 870200133836, dated 10 / 23 / 2020, page 20 / 143 8 / 19 capable of storing virtual images for later use. Some AR systems do not use a server (120), but generate virtual images locally or with a point-to-point or other computer or network architecture.

[0025] The way an AR device interacts with the real world generally depends on the availability of the environment. Marker-based and location-based AR interfaces are two main means of associating the real world with the virtual world. Marker-based AR generally has prior knowledge of the environment the device sees, while location-based AR generally does not. Location-based AR works by locating a reference point (e.g., absolute or relative position) in the environment where the user is.

[0026] In a non-limiting implementation example, the basic AR processing flow (200) begins with an image captured by a camera or CMOS image sensor (202). This video is divided into frames (204). Each image / frame is processed (206) to detect a marker (220), which in turn is used to determine the identification of a position or pose, for example, in relation to a marker. When this marker is detected, the camera's position (and in some cases also its orientation) is calculated considering its intrinsic parameters, for example, in relation to the marker and therefore to the environment. Once this position / orientation / pose is defined, the virtual object(s) (208) are processed in the same image / frame and the translation, rotation and perspective angles of the virtual content are Petition 870200133836, dated 10 / 23 / 2020, page 21 / 143 9 / 19 applied for display (210). This technology is used as an example in the context of video games, see AR Cards and game applications associated with the Nintendo 3DS 3D handheld video game system.

[0027] The fiducial marker (220) is a technique frequently used to enable the AR system to accurately position a virtual object in the real physical world. The fiducial marker (220) is typically a two-dimensional or multidimensional object that is positioned in the scene to be captured, detected by the camera, and processed to identify the object's position, as exemplified in Figure 2. In one embodiment, the fiducial marker (220) comprises a sticker or card with a special pattern that is (a) easily recognizable by an image decoder or pattern recognizer, (b) distinguishable from other fiducial markers also placed in the scene (e.g., encoded with a unique optically recognizable identifier), and (c) in some cases may be measurable to allow an optical detector to deduce its pose (e.g., position and / or orientation or aspects thereof) based on the detected optical pattern.Other fiducial markers may comprise infrared patterns and / or emitters, energy-emitting beacons, or other arrangements as known in the art.

[0028] It is also possible to use AR without artificial fiducial markers or other such elements (220). In this case, a device such as a camera can capture position, orientation and / or pose by detecting natural features in the real physical world. An example of this Petition 870200133836, dated 10 / 23 / 2020, page 22 / 143 10 / 19 technique is the identification by edges or textures of an object based on the characteristics of a corresponding 3D model. Edge and / or texture matching allows the natural object itself to serve as a marker, without the need to use a dedicated artificial fiducial marking object (220) that is not part of the real, physical, natural world.

[0029] Many advances in this technology are enabling the use of AR in the aeronautical industry. The 3D models created during aircraft design can be reused, allowing the use of AR in aircraft manufacturing, training, inspection, and maintenance. The use of an AR-based device provides accessibility, displaying virtual information in the real, physical world.

[0030] The first step in AR tracking is the detection of a known target in an incoming video stream using a detection algorithm, producing the pose (e.g., position and orientation in 6 degrees of freedom) of the camera relative to the target. The detection procedure involves finding a set of matches between the received images and the reference images, but robust and adequate object detection for AR is still a challenging problem. Deep Learning techniques will be used to solve these target detection problems, since deep convolutional neural networks can be trained to detect targets for augmented reality tracking. The target image is processed to create many synthetic views from different angles and under different lighting conditions. Therefore, in addition to Petition 870200133836, dated 10 / 23 / 2020, p. 23 / 143 11 / 19 To accelerate the classification of the quality or condition of aircraft components in the fault and defect identification process, Deep Learning allows these processes to be performed by technicians with low levels of specialization, making them cheaper and allowing for fewer interventions.

[0031] The non-limiting technology described herein refers to systems and methods for spatial localization of three-dimensional points using combinations of augmented reality, ultrasound (or other active sensor geolocation system), computer vision, and deep learning. The exemplary system uses environment reconstruction techniques, along with ultrasound, to perform spatial localization of a three-dimensional point. Once the desired point is located, its accuracy and precision are enhanced by detecting and processing the region of interest in the images captured by the camera using computational visualization and deep learning techniques.

[0032] Through information obtained by ultrasound sensors or other components, which are fixedly distributed in the environment (for example, mounted on an aircraft fuselage), the space in which the user is located is reconstructed or reconstituted. The user also has an ultrasound sensor or emitter that is used to triangulate between the user's location point and the fixed sensors and / or detectors in the environment.

[0033] The example methodology begins with the selection of a physical reference that will serve as the origin. This reference is used to calibrate the virtual coordinate systems of the ultrasound and reconstruction. Petition 870200133836, dated 10 / 23 / 2020, page 24 / 143 12 / 19 3D. After calibrating the respective virtual coordinate systems of the ultrasound sensor system and the virtual 3D environment to the same origin (for example, transforming the virtual environment into world space as defined by the ultrasound coordinate system), sites of interest to the user, such as sites that need to be monitored in future inspections, so-called events, are also recorded. The recorded information is the spatial coordinates of the chosen sites and photos of these sites, as shown in Figure 3-A, similar to geomapping commonly used for online mapping of locations on GPS, such as tourist attractions.

[0034] With this information in the system, a different user (e.g., the mechanic) (or the same user at a later time) can open the application interface, which will display visual information, such as an arrow in Augmented Reality, to guide the user to the spatial location of the event that was recorded in the previous step (see Figure 3-B). During the user's path to the marked event (from Figure 3-A), the user's position (pose) is determined with high precision. Computational visualization techniques associated with Deep Learning algorithms are used to enhance the accuracy of the positioning, so that the application interface can show the user exactly where the event recorded by the initial user is located (see Figure 3-C).

[0035] The goal of computational visualization techniques and Deep Learning algorithms is to capture Petition 870200133836, dated 10 / 23 / 2020, page 25 / 143 13 / 19 images generated by an input sensor (e.g., a camera) are used to accurately segment and recognize the area of ​​interest, and then compared with images recorded in the system. A non-limiting flowchart example of this methodology is shown in Figure 4.

[0036] As mentioned above, for the operation of a proposed non-limiting system, ultrasound sensors or other components are installed in the environment of interest (502). This ultrasound system is based on the propagation of ultrasonic frequency sound through the air, using, for example, a piezoelectric device to generate sound pulses (generally above the range of human hearing, for example, at 20 kHz or more), allowing the sensors or other components to communicate. These sensors or other components, called anchors, are fixed at locations in the environment and are configured in a mesh topology that can actively track a mobile sensor and / or emitter held or worn by the user. This set of sensors or other components uses electrical energy to perform the transmission and reception of the signal and allow triangulation of the user's spatial positioning in the anchor sensor mesh.In general, a single emitter-sensor pair allows for distance detection, two emitter-sensor pairs (e.g., one emitter and two sensors or one sensor and two emitters) allow for the determination of a two-dimensional distance vector, and three emitter-sensor pairs allow for the detection of three-dimensional position coordinates. Further enhancements are possible (e.g., two sensors mounted close to each other in a handheld device). Petition 870200133836, dated 10 / 23 / 2020, page 26 / 143 14 / 19 can be used to detect aspects of orientation and allow pose detection in 4 degrees of freedom.

[0037] 3D reconstruction can use cameras, infrared sensors and / or depth sensors (e.g., RADAR and / or LIDAR, or systems like Microsoft's Kinect 3D sensor) to virtually reconstruct the real environment and thus identify the user's spatial position (e.g., as detected by the ultrasonic system) in relation to the virtual environment (504). The accuracy of the user's position is enhanced by combining the information obtained by the 3D cameras and the ultrasonic sensor system. To ensure even higher accuracy in the location of an event, computational visualization techniques associated with Deep Learning (DL) algorithms are applied.

[0038] Once the user's position is known, the Augmented Reality system displays the spatial coordinates of the recorded event and guides the user to the desired event via arrows or other indicators on the mobile device screen (and / or may provide audible instructions to the user, such as walk 10 steps ahead, locate the access panel identified as Do Not Step and rotate the lever 90 degrees counterclockwise to release the access panel's retention mechanism). This system and method for spatial localization of three-dimensional points using the combination of these technologies provides better spatial signal coverage, which translates into at least the following advantages: shorter time to locate the component to perform the maintenance task, high accuracy. Petition 870200133836, dated 10 / 23 / 2020, page 27 / 143 15 / 19 in the location of recorded events and greater tolerance to moving obstacles (e.g., pedestrian traffic, cars, and objects in general). Furthermore, the use of Augmented Reality can guide the user to the location of interest efficiently and accurately.

[0039] In the exemplary embodiment presented below, the expression "events" can mean structural failures (delamination, detachment, cracks, corrosion, etc.) of electrical / hydraulic equipment (avionics, connectors, cables, sensors, pipes, etc.). This embodiment is exemplary and not limiting to other applications in the aeronautical, avionics, or other industries that, for example, direct the user to items of interest that are not events.

[0040] FIG. 6 illustrates a non-limiting embodiment of an array of sensors or other ultrasonic devices, embedded in an avionics compartment 600, configured in a mesh topology, used to determine the location of a display device. In the system described herein, the sensors or other devices (a, b, e, d ...) are electronic components capable of transmitting and / or receiving signals in order to determine the location of the display device within an avionics compartment 600.

[0041] In a non-limiting embodiment, the embedded array of sensors (a, b, e, d...) or other devices function as ultrasonic sensors configured to detect a signal emitted from a user-carried display device via an emitting component that is part of the device. Petition 870200133836, dated 10 / 23 / 2020, page 28 / 143 16 / 19 display or a module attachment to the display device or device that the user uses. In this example, the display device emits an ultrasonic sound that the system uses to determine the 3D coordinates of the display device's location (pose) in avionics bay 600.

[0042] The emitter of the display device and the array of ultrasonic sensors are operationally coupled to an aircraft computing system. The computing system controls when the emitter emits an ultrasonic pulse and / or is notified when the emitter emits a pulse. The computing device (or hardware operationally connected to the computing device) times how long it takes for the emitted signal to reach each sensor in an array of ultrasonic sensors. The computing device uses this timing information to calculate the location of the display device. In one embodiment, the emitter is part of a user's display device and the anchors are the array of ultrasonic sensors embedded in known locations on the aircraft fuselage.

[0043] Another preferred non-limiting embodiment, the display device is equipped with an ultrasonic sensor or other receiver component that is part of the display device or a module accessory to the display device. This receiver is configured to detect signals emitted by a set of emitters embedded in the aircraft's avionics compartment 600. The emitters can be controlled to emit pulses in a known sequence and / or using conventional techniques of Petition 870200133836, dated 10 / 23 / 2020, page 29 / 143 17 / 19 signal tagging (so that the sensor can distinguish the ultrasonic pulses emitted by the various emitters and match a received pulse with a known location of the emitter).

[0044] In yet another non-limiting embodiment, the display device and / or aircraft anchors are electronic components with transceiver properties. This embodiment is configured so that a transceiver emits a pulse that bounces / reflects off a target and is received by the same or a different transceiver anchored in avionics compartment 600 or maintained by the user. Thus, some embodiments may have an active device anchored only within the environment, other embodiments may have an active device mounted or carried by the user, and still other embodiments may have active devices in the environment and on the user.

[0045] All the non-limiting modalities described above can be supplemented with an image sensor, which is part of the display device or a module attachment to the display device, where a user determines a physical reference. Furthermore, Deep Learning image processing techniques are used to determine the location of the display device on the aircraft more precisely, in addition to the modalities described above. These Deep Learning neural networks can be trained using a sequence of known images to, for example, recognize environmental features such as the fuselage of a particular aircraft.

[0046] Example of a Use Case Petition 870200133836, dated 10 / 23 / 2020, page 30 / 143 18 / 19

[0047] In this example of non-limiting embodiments, the proposed method and system are used for the spatial localization of the three-dimensional location of an electrical connector (reference name: P0813) that is in the aircraft's avionics compartment (see Figure 5-A).

[0048] Consider a hypothetical situation described below: a) The aircraft monitoring system reports a fault message; b) The error message has an associated troubleshooting procedure; c) The troubleshooting procedure (see Figure 5-B) calls for checking an electrical connector (P0813) to correct this fault. This electrical connector is located in the aircraft's avionics compartment; (d) Non-limiting technology guides the mechanic to the requested event, so that the mechanic can quickly, accurately, and precisely locate the P081 3 electrical connector located in the aircraft's avionics compartment; this allows the mechanic to perform the tasks required by the troubleshooting procedure.

[0049] Any patents and publications cited above are incorporated by reference.

[0050] Although the non-limiting technology has been described in connection with what are currently considered the most practical and preferred embodiments, it should be understood that the invention is not to be limited to the disclosed embodiments, but, on the contrary, is intended to Petition 870200133836, dated 10 / 23 / 2020, page 31 / 143 19 / 19 covers various modifications and equivalent provisions included in the standard and within the scope of the attached applications. Petition 870200133836, dated 10 / 23 / 2020, page 32 / 143

Claims

1 / 2 CLAIMS 1. A method for locating an event using augmented reality, CHARACTERIZED by comprising: a) emitting a signal to an environment including an aircraft; b) detecting the emitted signal; c) processing the detected emitted signal to determine the spatial location of a display device in the environment; d) using Deep Learning to detect a physical reference point in the environment; e) displaying, on the display device in response to the determined location of the display device and the detected physical reference point, augmented reality content to guide the display device to an event on or within the aircraft.

2. The method, according to claim 1, is characterized by further including the use of the detected physical reference point to calibrate with an environmental coordinate system.

3. The method, according to claim 1, is characterized by further including the recording of a spatial coordinate of the event and the orientation of the display device toward the recorded event based on the spatial coordinate.

4. The method, according to claim 1, is characterized by further including tracking of the display device, determining the time of flight between at least one ultrasonic emitter and at least one ultrasonic sensor.

5. The method according to claim 4, characterized by at least one ultrasonic emitter or at least one ultrasonic sensor being deposited in the display device.

6. The method according to claim 4, characterized by further including the determination of the position of the display device.

7. The method according to claim 4, characterized in that at least one emitter or at least one sensor comprises a mesh topology.

8. The method, according to claim 1, is characterized by the augmented reality content generated and displayed by the display device, comprising arrows, text boxes, virtual thermography, and other figures to point to the event.

9. An aircraft system configured to locate an event using mixed reality content, CHARACTERIZED by comprising: a) an array of ultrasonic devices, at least some of which are incorporated into an aircraft, configured to determine the dynamic location of a display device as it moves relative to an environment containing an aircraft; b) an image sensor configured to capture an image of the environment; and c) a processor coupled to the image sensor, the processor using the captured image to determine a reference position in the environment and using the determined reference position and the determined dynamic location to generate a mixed reality image for display on the display device, the mixed reality image including an indication of at least one part of the aircraft to be maintained.

10. The system, according to claim 10, is characterized by the processor calibrating a coordinate system using the determined reference position and the determined dynamic location.

11. The system, according to claim 10, is characterized by the processor being coupled to the array of ultrasonic devices and tracking the display device by measuring the time-of-arrival (TOA) change of the ultrasonic signals exchanged between the ultrasonic devices.

12. The system, according to claim 10, CHARACTERIZED in that the display device includes at least one ultrasonic device.

13. The system, according to claim 10, is characterized by the mixed reality image comprising arrows, text boxes, virtual thermography, and other figures to point to an event.

14. The system, according to claim 15, is characterized by the mixed reality image comprising instructions for aircraft maintenance.

15. The system for locating an event using augmented reality content, CHARACTERIZED by comprising: a) an array of ultrasonic devices, capable of receiving and transmitting ultrasonic signals, configured to determine spatial coordinates by measuring the time of arrival (TOA) of ultrasonic pulses; b) an additional sensor configured to measure a three-dimensional feature of an environment including an aircraft; and c) at least one processor coupled to the array of ultrasonic devices and the additional sensor, at least one processor being configured to use the determined spatial coordinates and the measured three-dimensional feature to position the virtual content within a 3D model of the environment.

16. The system, according to claim 17, is characterized by at least one processor being further configured to record an image of an event relative to the 3D model.

17. The system, according to claim 17, is characterized by at least one processor being further configured to use a deep learning neural network to measure the three-dimensional feature.

18. The system, according to claim 17, is characterized by further including a mobile display device, at least one processor being further configured to generate augmented reality display content on the mobile display device. Petition 870260077729, dated 04 / 08 / 2026, pp. 12 / 13

Images