Spatial positioning using augmented reality

Abstract

This invention relates to spatial localization using augmented reality. It also includes methods and systems for localizing events related to aircraft using augmented reality content, an array of ultrasonic devices configured in a grid topology, and deep learning.

Description

[0001] Cross-references to related applications

[0002] none

[0003] Statement regarding federally sponsored research or development

[0004] none Technical Field

[0005] The non-limiting techniques described herein relate to the use of ultrasonic sensor networks (referred to herein as "ultrasound") to automatically detect the location coordinates of a user in an indoor environment with augmented reality (referred to herein as "AR") including mobile devices, enabling the user to receive location information in real time; and the use of deep learning (referred to herein as "DL") methods to guide technicians to quickly locate components that need to be accessed during maintenance, with accuracy and precision, for the purpose of improving aircraft maintenance processes. In some other aspects, the techniques described herein relate to methods and systems for using ultrasound to locate and navigate environments, and in some embodiments, to combine deep learning / artificial intelligence image pattern recognition to assist users in using augmented reality content on display devices. Background Technology

[0006] Currently, aircraft maintenance operations face difficulties in obtaining maintenance technicians to perform the tasks they require. In many cases, technicians lack sufficient knowledge of the aircraft requiring maintenance, leading to higher costs for maintenance tasks, as they may take longer or require more than one mechanic to perform the task.

[0007] To perform maintenance tasks, mechanics typically must first check for any available information about the aircraft requiring maintenance. If no advanced information about the aircraft needs maintenance, the mechanic should check the aircraft's flight log to see if there are any pending maintenance requests. In cases where the aircraft has a problem, the first approach is to try to correct the problem. If the problem is not simple, the assignment of activities can be verified using a Minimal Equipment List (MEL), which is used to determine the aircraft's continued airworthiness. If faulty equipment is not listed in the MEL, it is necessary to perform troubleshooting, i.e., to try to resolve the problem by systematically searching for the root cause of the component problem or its replacement.

[0008] This troubleshooting process must typically be performed until the problem is resolved and the aircraft is operational again. If an aircraft malfunctions, the mechanic must perform a maintenance procedure, which, in addition to obtaining the necessary parts, equipment, and materials, also requires knowledge of the location of specific components on the aircraft that may be malfunctioning or require attention. The necessary maintenance information can be found in the aircraft maintenance manual (which may contain information such as disassembly / installation procedures, troubleshooting, activation / deactivation, etc.) and is accessible to the mechanic via printed documents and / or digital files. The mechanic must generally follow all maintenance steps outlined in the aircraft maintenance manual to ensure proper maintenance of the equipment.

[0009] Based on the above, prior knowledge about the aircraft needs to become readily apparent in order to quickly locate and maintain the components that require maintenance. In many cases, locating the components that need to be located is not easy because they may be hidden inside, beside, or behind other components of the aircraft, such as panel covers. The faster the maintenance location is found, the faster maintenance activities can be initiated.

[0010] The complexity of positioning components can be reduced by coupling geolocation systems between equipment and mechanics. Geolocation is used for many different purposes, such as navigation in unknown environments, object location, and site identification. However, there is no technology suitable for aircraft maintenance that provides high positioning accuracy in both indoor and outdoor environments. In other words, there is no reliable, effective, and efficient solution for this purpose.

[0011] Currently, many technologies exist that enable positioning. In most cases, the solutions used are based on the GPS system, which provides high-precision absolute geographic coordinates. However, GPS has limitations when used indoors or under obstacles (such as in an aircraft cargo hold or under the fuselage). For GPS to function properly, the receiver needs to be within the line of sight of GPS satellites. Therefore, to overcome the limitations of the GPS system, several approaches have emerged in recent decades, proposing hardware and software solutions for positioning in enclosed environments. High positioning accuracy helps to correctly locate the object being searched and avoids directing maintenance users to the wrong location, thus preventing delays in the execution of maintenance tasks. Attached Figure Description

[0012] The following detailed description of exemplary, non-limiting, exemplary embodiments will be read in conjunction with the accompanying drawings, in which:

[0013] Figure 1 Exemplary non-limiting components (hardware, software, servers) in prior art AR-based systems are shown.

[0014] Figure 2A non-limiting embodiment of an example non-limiting basic AR processing flow is shown.

[0015] Figure 3A , 3B The 3C and 3C examples demonstrate how to capture events and how to use them in non-restrictive techniques.

[0016] Figure 4 It shows Figure 3A , 3B Example non-restricted flowcharts of methods used in 3C.

[0017] Figure 5A and 5B This is a non-restrictive example in which non-restrictive techniques are applied.

[0018] Figure 6 A non-limiting embodiment of the sensor array is shown. Detailed Implementation

[0019] The example non-limiting embodiments are applied to improve the aircraft maintenance process and quickly guide technicians to accurately and precisely locate the components that need to be accessed during maintenance.

[0020] The proposed technology is suitable for providing maintenance for various environments and vehicles, including but not limited to: indoor environments, complex structures, aircraft, automobiles, and ships. More specifically, a preferred embodiment of the technology in a potential application is incorporated into an aircraft maintenance system.

[0021] In other applications, ultrasonic technology can be used to locate users in an environment with high precision. It is well known that the working principle is based on the response time of the propagation of ultra-high frequency sound waves emitted by a transceiver device. Sensors measure the time of flight or arrival time (absolute or differential) of the emitted ultrasonic pulse. The distance can be calculated based on the speed of sound for each path between (one or more) transmitters and (one or more) sensors. The user's position can be obtained through trilateration of signals from transceiver devices installed in the environment and signals about the user or the device carried by the user. While this ultrasonic technology has been used in the past to determine the posture of a user's body parts (e.g., hands) relative to an array of sensors and / or transmitters (see, for example, the Nintendo Power Glove manufactured by Mattel), this non-limiting technique extends such a method to make it applicable in aircraft maintenance environments.

[0022] Augmented reality (“AR”) is a real-time representation that combines the real, physical, and virtual worlds within a public user interface. Typically, augmented reality devices, which can be configured as display devices such as handheld displays, goggles, or glasses, render virtual information based on the real physical world. One AR approach uses transparent lenses to allow users, with permission, to view the real world while virtual objects are displayed within the real-world scene. Another AR approach uses a camera to capture a real-world scene and combines the captured camera video stream with one or more virtual artificial display objects that appear to the user as part of the real-world scene. Yet another AR approach uses 3D graphics technology to simulate the real world and displays a scene defined by a model of the world and one or more other virtual objects that are not part of the world (this approach is sometimes referred to as “mixed reality”).

[0023] Most AR-based systems have two interrelated elements. The first element includes the hardware and software components used to implement the AR-based system. The second element includes the way AR is implemented in the real-world environment. Many AR-based systems use three components: hardware (100), software (200), and a server (300). Figure 1 This explains the relationship between the three components and how they enable the AR-based system to function.

[0024] The hardware (100) in an AR device typically uses modules such as a display device (102) (e.g., a head-mounted display (HMD), smartphone screen, glasses, etc.), an input device (104) (e.g., sensors such as cameras, infrared sensors, depth sensors, GPS, gyroscopes, accelerometers, etc.), and a processor (106). The sensors (105) detect the position and orientation of the display device (102) in or relative to the environment.

[0025] The software (110) aspect of AR devices is used to render virtual images and inject virtual images into the real world. The task of the software (110) (e.g., StudioMax, Cinema4D, AutoCAD3D, etc.) is typically to generate virtual images for overlay on or in combination with real-time images.

[0026] When an AR device requests certain virtual images, a server (120) (e.g., a network, cloud, etc.) retrieves the virtual images and sends them to the AR device, and is often also able to store the virtual images for later use. Some AR systems do not use a server (120), but instead generate virtual images locally or using peer-to-peer or other computer or network architectures.

[0027] The way AR devices interact with the real world typically depends on the availability of the environment. Tag-based and location-based AR interfaces are two main ways to associate the real world with the virtual world. Tag-based AR usually has prior knowledge about the environment the device "sees," while location-based AR usually does not. Location-based AR works by locating a reference point (e.g., absolute or relative position) in the user's environment.

[0028] In one exemplary, non-limiting implementation, the basic AR processing flow (200) begins with an image captured by a camera or CMOS image sensor (202). The video is divided into frames (204). Each image / frame is processed (206) to detect a marker (220), which is then used to determine, for example, the position or pose relative to the marker. When the marker is detected, the camera's inherent parameters are taken into account, and the camera's position (and in some cases its orientation) relative to the marker and therefore relative to the environment is calculated. Once the position / orientation / pose is defined, one or more virtual objects (208) are rendered in the same image / frame, and translation, rotation, and viewpoint for the virtual content are applied for display (210). Such techniques are used as examples in the context of video games; see “AR Cards” and related game applications for the Nintendo 3DS handheld 3D video game system.

[0029] Reference markers (220) are a technique commonly used to allow AR systems to accurately locate virtual objects in the real physical world. Reference markers (220) are typically two-dimensional or multi-dimensional objects that are placed in a scene to be captured and detected by a camera, and then processed to identify the object's position, such as... Figure 2 As shown. In one embodiment, the reference marker (220) comprises a sticker or card with a special pattern that (a) is easily identifiable by an image decoder or pattern recognizer, (b) is distinguishable from other reference markers also placed in the scene (e.g., encoded using a unique, optically identifiable identifier), and (c) may in some cases be measurable to allow an optical detector to infer its attitude (e.g., its position and / or orientation or phase) based on the optically detected pattern. Other reference markers may include infrared patterns and / or transmitter beacons emitting energy or other arrangements known in the art.

[0030] AR can also be used without any artificial reference markers or other such elements (220). In this case, a device such as a camera can capture position, orientation, and / or pose by detecting natural features in the real physical world. One example of this technique is identification based on features of a corresponding 3D model, through the edges or textures of objects. The correspondence between edges and / or textures allows the natural object itself to be used as a marker without the need for a dedicated artificial reference marker object that is not part of the real natural physical world (220).

[0031] This significant advancement in technology has made the use of AR in the aerospace industry possible. 3D models created during aircraft design can be reused, allowing AR to be used in aircraft manufacturing, training, inspection, and maintenance. AR-based devices provide accessibility by displaying virtual information within the real physical world.

[0032] The first step in AR tracking is to use detection algorithms to detect known targets in the input video stream, thereby generating the camera's "pose" (e.g., position and orientation in six degrees of freedom) relative to the target. This detection process involves finding a set of matches between the input image and one or more reference images, but robust and appropriate detection of objects for AR remains a challenging problem. Deep learning techniques will be used to solve these target detection problems because deep convolutional neural networks can be trained to detect targets for augmented reality tracking. Target images are rendered to create numerous synthetic views from different angles and under different lighting conditions. Therefore, in addition to accelerating the classification of the quality or condition of aircraft components in the process of identifying faults and defects, deep learning also allows these processes to be performed by less skilled technicians, making them cheaper and allowing for less intervention.

[0033] The non-limiting techniques described herein relate to systems and methods for spatially locating three-dimensional points using a combination of augmented reality, ultrasound (or other geolocation systems via active sensors), computer vision, and deep learning. An exemplary system uses environment reconstruction techniques and ultrasound to achieve spatial localization of three-dimensional points. Once the desired point is located, regions of interest in one or more images captured by a camera can be detected and processed using computer vision techniques and deep learning, improving accuracy and precision.

[0034] Information obtained from ultrasonic sensors or other components permanently distributed in the environment (e.g., mounted on an aircraft fuselage) can reconstruct or reconstruct the space in which a user is located. The user also has ultrasonic sensors or transmitters used to perform triangulation between the user's location and fixed sensors and / or detectors in the environment.

[0035] An exemplary method begins by selecting a physical reference to be used as the origin. This reference is used to calibrate the virtual coordinate systems for ultrasound and 3D reconstruction. After calibrating the respective virtual coordinate systems of the ultrasound sensor system and the virtual 3D environment against the same origin (e.g., by converting the virtual environment into world space defined by the ultrasound coordinate system), user records of locations of interest, such as locations to be monitored in future examinations (referred to as “events”), are also recorded. The recorded information includes the spatial coordinates of the selected locations and photographs of these locations, such as those obtained by [unclear text - likely a website or organization]. Figure 3A As shown, this is similar to geomapping, which is typically used to map GPS locations (such as tourist attractions) online.

[0036] Using this information in the system, different users (e.g., a mechanic) (or the same user at different, later times) can open the application interface, which will display visual information, such as augmented reality arrows, to guide the user's spatial orientation to the event recorded in the previous step (see [link]). Figure 3B During the path from the user to the flagged event (from...) Figure 3A This allows for high-precision determination of the user's location (pose). Using computational vision techniques associated with deep learning algorithms improves positioning accuracy, enabling the application interface to accurately display the location of the initially recorded user-generated events to the user (see...). Figure 3C ).

[0037] The goal of computer vision techniques and deep learning algorithms is to capture images generated by input sensors (e.g., cameras) in order to segment and accurately identify regions of interest and compare them with images recorded in the system. In the following text, Figure 4 An example non-restrictive flowchart of the method is shown below.

[0038] As described above, to operate a proposed non-limiting system, ultrasonic sensors or other components are mounted in the environment of interest (502). This ultrasonic system is based on the propagation of ultrasonic sound in the air at ultrasonic frequencies, using, for example, one or more piezoelectric devices to generate sound pulses (typically above human hearing range, e.g., 20 kHz or higher), allowing the sensors or other components to communicate with each other. These sensors or other components, referred to as “anchors,” are fixed in place in the environment and configured in a grid topology that actively tracks the movement of handheld or worn sensors and / or transmitters. This set of sensors or other components uses electrical energy to perform transmission and signal reception, and enables triangulation of the user’s spatial positioning within the anchored sensor grid. Generally, a single transmitter-sensor pair allows for distance detection, two transmitter-sensor pairs (e.g., one transmitter and two sensors or one sensor and two transmitters) enable the determination of a two-dimensional distance vector, while three transmitter-sensor pairs enable the detection of three-dimensional position coordinates. Additional enhancements (e.g., two sensors mounted close to each other on a handheld device) can be used to detect directional phase to achieve four degrees of freedom of attitude sensing.

[0039] 3D reconstruction can use cameras, infrared sensors, and / or depth sensors (e.g., RADAR and / or LIDAR, or systems such as Microsoft's Kinect 3D sensor) to virtually reconstruct the real-world environment and thus identify (e.g., as detected by an ultrasound system) the user's spatial location relative to the virtual environment (504). The accuracy of user location can be improved by combining information obtained from 3D cameras and ultrasound sensor systems. To further ensure high accuracy in event localization, computer vision techniques associated with deep learning (DL) algorithms are applied.

[0040] Once the user's location is known, the augmented reality system displays the spatial coordinates of the recorded event and guides the user to the desired event via arrows or other indicators on the mobile device's display (and / or can deliver audible instructions to the user, such as "Walk forward 10 steps to find the access panel marked 'Do Not Step On,' then rotate the handle 90 degrees counterclockwise to release the access panel retaining mechanism"). Systems and methods for spatially locating 3D points using this combination of technologies provide better spatial signal coverage, which translates to at least the following advantages: shorter time to locate components for performing maintenance tasks, higher accuracy in locating recorded events, and greater tolerance for moving obstacles (e.g., pedestrian traffic, general vehicles, and objects). Additionally, augmented reality can efficiently and accurately guide users to locations of interest.

[0041] In the exemplary embodiments presented below, the term "event" can refer to a structural failure (delamination, debonding, cracking, corrosion, etc.) of electrical / hydraulic equipment (avionics equipment, connectors, cables, sensors, pipes, etc.). This embodiment is an exemplary and non-limiting application in the aircraft industry, avionics equipment, or other environments, such as guiding a user to other items of interest besides "events".

[0042] Figure 6 A non-limiting embodiment of a sensor or other ultrasonic device array is illustrated, which is embedded in an avionics bay 600 in a grid topology configuration and used to determine the location of a display device. In the system described herein, the sensor or other device (a, b, c, d…) is an electronic component capable of transmitting and / or receiving signals to determine the location of the display device within the avionics bay 600.

[0043] In a non-limiting embodiment, the embedded array of sensors (a, b, c, d…) or other devices serves as an ultrasonic sensor configured to sense signals emitted from a user-carried display device via a transmitter assembly, which is part of the display device or a module accessory of the display device or the user-worn device. In this example, the display device emits ultrasound, which the system uses to determine the 3D coordinates of the display device's position (attitude) within the avionics bay 600.

[0044] The transmitter of the display device and the ultrasonic sensor array are operatively coupled to the aircraft's computing system. The computing system controls when the transmitter emits ultrasonic pulses and / or is informed when the transmitter emits pulses. The computing device (or hardware operatively connected to the computing device) times the time it takes for the emitted signal to reach each sensor in the ultrasonic sensor array. The computing device uses this timing information to calculate the position of the display device. In one embodiment, the transmitter is part of the user's display device, and the anchor is the ultrasonic sensor array embedded at a known location on the aircraft fuselage.

[0045] In another preferred, non-limiting embodiment, the display device is equipped with a sensor or other ultrasonic receiver assembly that is part of the display device or a module accessory thereof. The receiver is configured to sense signals emitted by a transmitter array embedded in the avionics bay 600 of the aircraft. The transmitters can be controlled to emit pulses in a known order and / or using conventional signal marking techniques (therefore, the sensor can distinguish ultrasonic pulses emitted by various transmitters and match the received pulses with known transmitter locations).

[0046] In yet another non-limiting embodiment, the display device and / or the aircraft's anchor is an electronic component with transceiver characteristics. This embodiment is configured such that the transceiver emits pulses that bounce / reflect from a target and are received by the same or different transceivers anchored to the avionics bay 600 or held by the user. Thus, some embodiments may have one or more active devices anchored only within the environment, other embodiments may have one or more active devices mounted on or carried by the user, and still other embodiments may have active devices both within the environment and on the user.

[0047] All the previously described non-limiting embodiments may be supplemented with an image sensor that is part of or a module accessory of the display device, wherein the user determines a physical reference. Furthermore, in addition to the previously described embodiments, deep learning image processing techniques have been used to more accurately determine the location of the display device within the aircraft. Such a deep learning neural network can be trained using a series of known images to, for example, identify features of the environment, such as the fuselage of a particular aircraft.

[0048] Example usage

[0049] In this example of a non-limiting embodiment, the proposed method and system are used to spatially locate the three-dimensional position of an electrical connector (label name: P0813) in an aircraft avionics bay (see [reference]). Figure 5A ).

[0050] Consider the following assumptions:

[0051] a) The aircraft monitoring system notifies the aircraft of a malfunction;

[0052] b) The fault message includes a related troubleshooting process;

[0053] c) Troubleshooting process (see...) Figure 5B The electrical connector (P0813) is required to be inspected to correct the fault. This connector is located in the aircraft's avionics bay.

[0054] d) Non-restrictive techniques guide the mechanic to the requested event, enabling the mechanic to quickly, accurately, and precisely locate electrical connector P0813 in the aircraft's avionics bay; this allows the mechanic to perform the tasks required for the troubleshooting process.

[0055] Any patents and publications cited above are incorporated herein by reference.

[0056] Although non-limiting techniques have been described in conjunction with embodiments that are currently considered to be the most practical and preferred, it should be understood that the invention is not limited to the disclosed embodiments, but rather is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for locating events using augmented reality, comprising: a. To transmit signals into the environment, including aircraft; b. Detect the emitted signal; c. Processing the detected emitted signals to determine the spatial location of the display device in the environment, wherein environmental reconstruction techniques and an ultrasonic system are used to determine the spatial location; d. After determining the spatial location, the accuracy and precision of the spatial location are improved by using computer vision technology and deep learning to detect and process physical reference points in the environment; as well as e. An event in which augmented reality content is displayed on the display device in response to a determined display device location and a detected physical reference point, thereby guiding the display device onto or within the aircraft.

2. The method of claim 1, further comprising using the detected physical reference point to calibrate with the coordinate system of the environment.

3. The method of claim 1, further comprising recording the spatial coordinates of the event, and guiding the display device to the recorded event based on the spatial coordinates.

4. The method of claim 1, wherein the ultrasound system comprises at least one ultrasound transmitter and at least one ultrasound sensor, the method further comprising tracking the display device by determining the time of flight between the at least one ultrasound transmitter and the at least one ultrasound sensor.

5. The method according to claim 4, wherein, The at least one ultrasonic transmitter or the at least one ultrasonic sensor is disposed on the display device.

6. The method of claim 4, further comprising determining the orientation of the display device.

7. The method according to claim 4, wherein, The at least one ultrasonic transmitter or the at least one ultrasonic sensor comprises a grid topology.

8. The method according to claim 1, wherein, The generated augmented reality content displayed by the display device includes arrows, text boxes, virtual thermal images, and other graphics used to point to the event.

9. The method according to claim 7, wherein, The generated augmented reality content displayed on the display device provides instructions or other maintenance information.

10. An aircraft system configured to use mixed reality content to locate events, the system comprising: a. An array of ultrasonic devices, wherein at least some ultrasonic devices are embedded in an aircraft, the array of ultrasonic devices being configured to determine the dynamic position of the display device, together with an environment reconstruction technique, as the display device moves relative to the environment containing the aircraft; b. An image sensor configured to capture an image of the environment; and c. A processor coupled to the image sensor, the processor using the captured image to determine a reference position in the environment, and using the determined reference position and a determined dynamic position to generate a mixed reality image for display on the display device, the mixed reality image including indications of at least a portion of the aircraft to be maintained. Wherein, after determining the dynamic position, the system is configured to improve the accuracy and precision of the dynamic position by using computer vision technology and deep learning to detect and process the reference position in the environment.

11. The system according to claim 10, wherein, The processor uses the determined reference position and the determined dynamic position to calibrate the coordinate system.

12. The system according to claim 11, wherein, The processor is configured to implement a deep learning neural network to determine the reference position.

13. The system according to claim 10, wherein, The processor is coupled to the array of ultrasound devices and tracks the display device by measuring the change in the time of arrival (TOA) of the ultrasound signals exchanged between the ultrasound devices.

14. The system according to claim 10, wherein, The display device includes at least one of the ultrasound devices.

15. The system according to claim 10, wherein, The mixed reality images include arrows, text boxes, virtual thermal images, and other graphics used to point to events.

16. The system according to claim 15, wherein, The mixed reality imagery includes instructions for maintaining the aircraft.

Images