Systems, methods, and apparatus for correlating orientation of real-world object with digital world

Abstract

The invention relates to a system, method and apparatus for correlating the orientation of a real-world object with a digital world, and discloses a portable electronic device for obtaining information about a region of interest of an object. A portable electronic device may include a display, an image capture device, a measurement device, and a processor. The processor may be configured to determine one or more positions of the portable electronic device relative to an orientation of the one or more items of the object, and determine an orientation of a region of interest of the object relative to the one or more positions of the portable electronic device. The processor may also be configured to identify electronic or digital information associated with the orientation of the region of interest of the object and transmit the electronic or digital information or the orientation of the region of interest to a remote computing device.

Description

Technical Field

[0001] The present disclosure generally relates to data processing systems, and more particularly to systems and methods for determining the orientation of regions of interest of physical objects (such as vehicles) in a real-world environment and obtaining electronic or digital information (such as digital models, virtual representations, schematic diagrams, etc.) related to the regions of interest of the physical objects. Background Art

[0002] The purpose of providing this background description is to generally present the context of the present disclosure. Unless otherwise indicated herein, the materials described in this section are neither expressly nor implicitly admitted to be prior art to the present disclosure or the appended claims.

[0003] Many complex objects and structures (such as large transport aircraft) are composed of multiple elements and systems, and each system may contain a large number of components and / or parts. These complex objects are often inspected by on-site support personnel (such as maintenance personnel, inspectors, etc.) in order to repair the object and fix its defects (such as damage, malfunction, etc.). To analyze a region of the object, it may be desirable to determine the orientation and measurement information of the defect of the object and obtain information related to the defect (such as schematic diagrams, drawings, maintenance records, etc.). However, obtaining information related to the defects of large and complex objects can be a daunting task. For example, support personnel may need to obtain and review a large number of technical manuals and documents to find the appropriate information for troubleshooting and fixing the defects of the object.

[0004] In some cases, on-site support personnel can communicate with on-site and / or off-site analysts or experts (such as maintenance engineers, repair engineers, etc.) regarding the defects of an object (such as a vehicle). Generally, on-site support personnel can use a camera to take photos of the defects (such as damage) of the object and record the location of the defects from the orientation of the on-site support personnel. For example, a maintenance person can take photos of the damage of a vehicle (such as an aircraft), record the orientation of the damage, and create a damage report by drawing a diagram of the damaged area on the vehicle. Figure 1 A photo showing a lightning strike report is presented, which indicates the orientation of the damage relative to known items and the orientation on the skin or aircraft. On-site support personnel can provide the photos of the damage, the damage report, and the details of the damage orientation to on-site and off-site analysts. On-site and / or off-site analysts (such as structural engineers) can use this information to perform structural analysis, conduct engineering verification, complete compliance inspection records, and / or submit the records to the Federal Aviation Administration (FAA) for review.

[0005] After the orientation information, photos, and damage reports are received, on-site and / or off-site analysts can attempt to determine the orientation of the object damage by visually comparing the photos with available documents (such as drawings, technical manuals, etc.). However, in areas where there are few uniquely identifiable landmark features, it may be difficult to determine the exact orientation of the damage. Additionally, on-site and off-site analysts may only be able to obtain a subjective indication of the orientation of the object damage. As a result, the assessment or analysis of the damage may be performed with a high probability of orientation error. Moreover, human analysis of a large number of defects (such as damage) may be prone to error.

[0006] In addition, photos or images taken by the cameras of support personnel and sent to on-site and / or off-site analysts may contain embedded metadata, such as GPS orientation data and camera setting information. The embedded metadata from the images can be used by on-site and / or off-site analysts to estimate the orientation of the camera relative to the object. However, without additional context regarding the orientation (i.e., position and orientation) of the object relative to the camera's reference frame, the metadata may be insufficient to determine the orientation of the camera relative to the object. Additionally, since the orientation of the camera relative to the object may not be accurately determined, it may be difficult to use the image to identify the orientation of the defect of the object.

[0007] On-site and / or off-site analysts may also have difficulty obtaining accurate measurements of the orientation of the defect of the object from the photos or images. Typically, an item of known size (such as a measuring tape) is inserted into the image to provide a size reference for the analyst. However, even with the use of a reference scale, it may be difficult to use the image to locate and determine the exact position of the defect of the object in the object's coordinate system. For at least these reasons, it would be advantageous to develop systems and methods capable of determining the orientation of an area of interest (such as a defect, damage, etc.) of a physical object in a real-world environment and obtaining electronic or digital information related to the area of interest of the physical object. Summary of the Invention

[0008] This application is directed to embodiments related to systems, methods, and devices for determining the orientation or position of an area of interest of a physical object (such as a vehicle, aircraft, system, etc.) in a real-world environment. Embodiments can use the orientation of the area of interest to obtain electronic or digital information (such as a 3D digital model, virtual representation, schematic diagram, engineering drawing, etc.) related to the area of interest of the physical object. The area of interest can correspond to an anomaly, defect, damage, fault, component, part, object, item, and / or condition of the physical object.

[0009] Embodiments can be used to inspect, troubleshoot, and / or repair physical objects. For example, support personnel can use these embodiments to perform repairs and inspections of physical objects. Embodiments can quickly and accurately determine the physical orientation or position of a region of interest relative to a physical object. For example, embodiments can determine the coordinates of a region of interest (e.g., anomaly, defect, damage, etc.) of a physical object in a coordinate system of the object (e.g., a coordinate system centered on the object). In some examples, embodiments can calculate the physical orientation of an object of interest on or in a vehicle (e.g., an aircraft). As a result, the likelihood of human error in accurately determining the orientation of a region of interest of a physical object can be reduced.

[0010] Embodiments can convert the physical orientation of a region of interest of a physical object to a position in an object coordinate system (e.g., a coordinate system centered on a vehicle). Embodiments can obtain electronic or digital information about a region of interest of a physical object based on the orientation of the region of interest in the object coordinate system. For example, embodiments can use the orientation of a region of interest of a physical object in an object coordinate system to identify and retrieve electronic information and records related to the region of interest of the physical object (e.g., a vehicle, an aircraft, etc.). The electronic information can be organized or classified using an object coordinate system (e.g., a vehicle coordinate system) and can include CAD / CAM electronic records, 3D digital models, virtual representations, schematics, engineering drawings, technical documents, maintenance reports, trouble reports, maintenance bulletins, system reports, structural analyses, operating documents, and other information related to the region of interest of a physical object.

[0011] Embodiments can be configured to search for electronic or digital records and reports (e.g., damage report records, maintenance requests, etc.) in one or more databases for information related to a region of interest of a physical object. For example, embodiments can search for electronic reports and records for problems related to the damage orientation of a particular type of vehicle or aircraft. Embodiments can identify relevant electronic or digital records and display the relevant electronic or digital records to support or maintenance personnel. Embodiments can also store or record any current damage reports of a physical object as new records or reports in a database. For example, embodiments can create and store a maintenance request or problem report (including a description and type of damage, object or vehicle type, photos and sketches of the damage, etc.) in a database (e.g., an airline database) based on the physical orientation of the damage.

[0012] Embodiments can send retrieved electronic and digital reports and records related to an area of interest of a physical object to the devices of on-site and off-site analysts or experts. For example, embodiments can send graphical and virtual representations of an area of interest of a physical object (e.g., a virtual representation of an aircraft cockpit) to on-site and / or off-site analysts. The electronic and digital information can assist on-site and off-site analysts in analyzing the area of interest of the physical object. For example, embodiments can enable on-site and off-site analysts to view electronic or digital information related to the area of interest of the physical object. Thus, on-site and off-site analysts may not need to access and inspect the physical object and can remotely determine the repair of any defects (e.g., anomalies, damages, conditions, etc.) associated with the area of interest.

[0013] By enabling efficient and reliable determination of an area of interest of a physical object (e.g., a vehicle, a system, etc.) in a real-world environment, embodiments can improve the process of troubleshooting and repairing physical objects. For example, embodiments can reduce the time required to identify and repair defects (e.g., flaws, damages, cracks, conditions, malfunctions, etc.) of a physical object and reduce the time necessary to obtain electronic and digital information related to the defects. Embodiments can also automate and accelerate the process of repairing a physical object (e.g., a vehicle), improve the accuracy of locating defects, reduce errors in determining the orientation of defects, and create new records for future repairs. Additionally, embodiments can advantageously improve the reliability, safety, repairability, and usability of a physical object (e.g., a vehicle), thereby resulting in improved performance and operational capabilities of the physical object. Further, in the aircraft industry, embodiments can reduce the number of flights delayed or cancelled due to maintenance or repair issues.

[0014] In one aspect, a portable electronic device for obtaining information about an area of interest of an object is disclosed. The portable electronic device can include a display, an image capture device, a measurement device, and a processor. The processor can be configured to determine one or more positions of the orientation of the portable electronic device relative to one or more items of the object and determine the orientation of the area of interest of the object relative to one or more positions of the portable electronic device. The processor can also be configured to identify electronic or digital information associated with the orientation of the area of interest of the object and send the electronic or digital information or the orientation of the area of interest to a remote computing device.

[0015] In another aspect, a method for obtaining information about an object's region of interest is disclosed. The method may include determining one or more positions of the orientation of a portable electronic device relative to one or more items of an object, and determining the orientation of the object's region of interest relative to one or more positions of the portable electronic device. The method may also include identifying electronic or digital information associated with the orientation of the object's region of interest, and sending the electronic or digital information or the orientation of the region of interest to a remote computing device.

[0016] In yet another aspect, a non-transitory computer-readable medium storing instructions is disclosed, which when executed by one or more processors, cause the one or more processors to perform operations for obtaining information about an object's region of interest. The operations may include determining one or more positions of the orientation of a portable electronic device relative to one or more items of an object, and determining the orientation of the object's region of interest relative to one or more positions of the portable electronic device. The operations may also include identifying electronic or digital information associated with the orientation of the object's region of interest, and sending the electronic or digital information or the orientation of the region of interest to a remote computing device.

[0017] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, additional aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] A more complete understanding of the embodiments of the present application can be obtained by reference to the detailed description and the claims when considered in conjunction with the following drawings, in which like reference numerals throughout the drawings refer to like elements. These drawings are provided to facilitate an understanding of the present disclosure and do not limit the breadth, scope, scale, or applicability of the present disclosure. The drawings are not necessarily drawn to scale.

[0019] Figure 1 A photograph showing an aircraft damage report, which details the orientation of the damage relative to other parts of the aircraft;

[0020] Figure 2 is a diagram of an exemplary aircraft;

[0021] Figure 3 is a schematic illustration of a system for obtaining information related to a region of interest of a physical object;

[0022] Figure 4A is Figure 2 a front view of a portable electronic device of the system, which shows an image and digital representation of a part of the cockpit of an aircraft;

[0023] Figure 4BYes Figure 3 Rear view of a portable electronic device;

[0024] Figure 5 Yes Figure 3 Schematic illustration of components of a portable electronic device;

[0025] Figure 6 Schematic illustration of a technique for determining the orientation of a region of interest of a physical object;

[0026] Figures 7A - 7E Illustrates another technique for determining the orientation of a region of interest of a physical object; and

[0027] Figure 8 Is an exemplary flowchart showing a method for obtaining information related to a region of interest of a physical object. Detailed Description

[0028] The accompanying drawings and the following description illustrate specific exemplary embodiments. It should be understood that those skilled in the art will be able to design various arrangements which, although not explicitly shown herein, embody the principles described herein and are included within the scope of the claims accompanying this specification. In addition, any examples described herein are intended to assist in understanding the principles of the disclosure and should be construed as having no limitation. Accordingly, the present disclosure is not limited to the specific embodiments or examples described below, but is limited by the claims and their equivalents.

[0029] Specific embodiments are described herein with reference to the accompanying drawings. In the specification, common features are denoted by common reference numerals throughout the drawings. In some of the drawings, multiple instances of a particular type of feature may be used. Although these features are physically and / or logically distinct, the same reference numeral may be used for each feature, and different instances are distinguished by adding letters to the reference numeral.

[0030] As used herein, various terms are for the purpose of describing particular embodiments only and are not intended to be limiting. For example, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. Additionally, the terms "comprise", "comprises", and "comprising" are used interchangeably with "include", "includes", or "including". Further, the term "wherein" is used interchangeably with the term "where". As used herein, "exemplary" indicates an example, embodiment, and / or aspect and should not be construed as limiting or indicating a preference or preferred embodiment. As used herein, ordinal numbers (such as "first", "second", "third", etc.) used to modify elements (such as structures, components, operations, etc.) do not themselves indicate any priority or order of the element relative to another element, but merely distinguish the element from another element having the same name (if no ordinal number is used). As used herein, the term "set" refers to a grouping of one or more elements, and the term "plurality" refers to a number of elements.

[0031] This application is directed to embodiments related to systems, methods, and apparatuses for determining the orientation or position of a region of interest of a physical object (e.g., a vehicle, an aircraft, a system, etc.) in a real-world environment. Embodiments can use the orientation of the region of interest to obtain electronic or digital information related to the region of interest of the physical object (e.g., a 3D digital model, a virtual representation, a schematic diagram, an engineering drawing, etc.). The region of interest can correspond to an anomaly, defect, damage, malfunction, component, part, object, item, and / or condition of the physical object.

[0032] Embodiments can be used to inspect, troubleshoot, and / or repair a physical object. For example, support personnel can use these embodiments to perform maintenance and inspection of a physical object. Embodiments can quickly and accurately determine the physical orientation or position of the region of interest relative to the physical object. For example, embodiments can determine the coordinates of the region of interest (e.g., an anomaly, a defect, damage, etc.) of a physical object in the coordinate system of the object (e.g., a coordinate system centered on the object). In some examples, embodiments can calculate the physical orientation of an object of interest on or in a vehicle (e.g., an aircraft). Thus, the possibility of human error in accurately determining the orientation of the region of interest of a physical object can be reduced.

[0033] Embodiments can convert the physical orientation of a region of interest of a physical object into a position in an object coordinate system (e.g., a vehicle-centered coordinate system). Embodiments can obtain electronic or digital information about the region of interest of a physical object based on the orientation of the region of interest in the object coordinate system. For example, embodiments can use the orientation of the region of interest of a physical object in the object coordinate system to identify and retrieve electronic information and records related to the region of interest of the physical object (e.g., a vehicle, an aircraft, etc.). The electronic information can be organized or classified using the object coordinate system (e.g., vehicle coordinate system), and can include CAD / CAM electronic records, 3D digital models, virtual representations, schematics, engineering drawings, technical documents, maintenance reports, defect reports, maintenance bulletins, system reports, structural analyses, operating documents, and other information related to the region of interest of the physical object.

[0034] Embodiments can be configured to search one or more databases for electronic or digital records and reports (e.g., defect report records, maintenance requests, etc.) related to the region of interest of a physical object. For example, embodiments can search for electronic reports and records for problems related to the defect orientation of a particular type of vehicle or aircraft. Embodiments can identify the relevant electronic or digital records and display the relevant electronic or digital records to support or maintenance personnel. Embodiments can also store or record any current defect reports of the physical object as new records or reports in the database. For example, embodiments can create and store a maintenance request or problem report (including a description and type of the defect, object or vehicle type, photos and sketches of the defect, etc.) in the database (e.g., an airline database) based on the physical orientation of the defect.

[0035] Embodiments can send the retrieved electronic and digital reports and records related to the region of interest of a physical object to the devices of on-site and off-site analysts or experts. For example, embodiments can send graphical and virtual representations (e.g., a virtual representation of an aircraft cockpit) of the region of interest of a physical object to on-site and / or off-site analysts. The electronic and digital information can assist on-site and off-site analysts in analyzing the region of interest of the physical object. For example, embodiments can enable on-site and off-site analysts to view electronic or digital information related to the region of interest of the physical object. Thus, on-site and off-site analysts may not need to access and inspect the physical object, and can remotely determine the repair of any defects (such as anomalies, damages, conditions, etc.) associated with the region of interest.

[0036] By efficiently and reliably determining regions of interest of physical objects (such as vehicles, systems, etc.) in a real-world environment, embodiments can improve the process of troubleshooting and repairing physical objects. For example, embodiments can reduce the time required to identify and repair defects (such as flaws, damages, cracks, conditions, malfunctions, etc.) of physical objects, and reduce the time necessary to obtain electronic and digital information related to the defects. Embodiments can also automate and accelerate the process of repairing physical objects (such as vehicles), improve the accuracy of locating defects, reduce errors in determining the orientation of defects, and create new records to facilitate future repairs. In addition, embodiments can advantageously improve the reliability, safety, repairability, and usability of physical objects (such as vehicles), thereby resulting in improved performance and operating capabilities of physical objects. Moreover, in the aircraft industry, embodiments can reduce the number of flights delayed or cancelled due to maintenance or repair issues.

[0037] As Figure 2 shown, an example embodiment of an aircraft 100 is illustrated, in which embodiments of systems and methods for obtaining electronic or digital information (such as 3D digital models, schematics, drawings, etc.) related to regions of interest of a physical object (such as the aircraft 100) can be implemented. As Figure 2 shown, the aircraft 100 includes a fuselage 102 having a left side 104, a right side 106, a nose end 108, and a tail end 110. A first wing 112 is coupled to the left side 104 of the fuselage 102. A second wing 114 is coupled to the right side 106 of the fuselage 102. In the illustrated example, the aircraft 100 includes a door 116 disposed on the left side 104 of the fuselage 102. Passengers and / or crew can enter (such as board) and / or leave (such as disembark) the aircraft 100 via the door 116. Figure 2 The aircraft 100 is only an example, and thus, without departing from the scope of the present disclosure, the embodiments disclosed herein can be used with other aircraft or vehicles.

[0038] The orientation or position of the structure and systems of the aircraft 100 can be specified relative to the local coordinates of the aircraft 100. The coordinate system of the aircraft 100 is indicated in Figure 2 where the x-axis indicates the front-to-back direction (such as the front to the back of the aircraft 100), the y-axis indicates the port-starboard direction (such as the left and right of the center of the aircraft 100), and the z-axis indicates the up-down direction (such as the bottom to the top of the aircraft 100). The x, y, and z coordinates relative to the aircraft 100 can also be referred to as stations (or fuselage stations), butt lines, and water lines, respectively.

[0039] Figure 3A system 200 for obtaining electronic or digital information about an area of interest 202 of a physical object 204 in a real-world environment 206 according to an exemplary embodiment is shown. As Figure 3 shown, the system 200 includes a portable electronic device 208, a communication network 210, a database 212, and communication devices 214 and 216. As shown, the portable electronic device 208 communicates with the communication network 210. The communication network 210 of the system 200 can be used to provide a communication link between various devices and computers connected together within the system 200. The communication network 210 can include connectors such as wires, wireless communication links, or fiber optic cables.

[0040] The portable electronic device 208 of the system 200 can be configured to display a real-time view of the physical object 204 in the real-world environment 206 (or a portion thereof) such that a user can view a representation of the physical object 204 in real time. The portable electronic device 208 can also be configured to determine the location or orientation of the area of interest 202 of the physical object 204 in the real-world environment 206, such as Figure 2 the orientation of an anomaly (e.g., defect, fault, condition, damage, etc.) of the aircraft 100. Additionally, the portable electronic device 208 can be configured to obtain electronic or digital information (e.g., 3D digital model, virtual representation, technical document, schematic, maintenance report, system data, photographic record, silhouette image, etc.) related to the area of interest 202 of the physical object 204.

[0041] The physical object 204 of the system 200 can include a vehicle, building, structure, system, subsystem, power plant, ship, spacecraft, surface or skin of a vehicle, component, part, and / or any other suitable physical object or item in the real-world environment 206. The real-world environment 206 can be any type of environment in the physical world (such as a workspace). In the example shown, the real-world environment 206 can be within the physical object 204, such as Figure 2 the fuselage of the aircraft 100. In other examples, the real-world environment 206 can be, but is not limited to, a maintenance environment, manufacturing environment, production environment, design environment, installation environment, and / or any other suitable environment.

[0042] As Figure 3 shown, the communication devices 214 and 216 of the system 200 can be connected to the communication network 210. The communication devices 214 and 216 can be, for example, wireless or computing devices operated by various personnel (such as maintenance personnel, mechanics, technicians, analysts, engineers, etc.). The communication devices 214 and 216 can be located within the physical object 204 (e.g., within an aircraft) or at a facility (e.g., a maintenance facility) located away from the physical object 204.

[0043] The database 212 of the system 200 can be connected to the communication network 210. The database 212 can store information related to the physical object 204. For example, the database 212 can include electronic or digital information about the architecture and structure of the physical object 204 (e.g., 3D digital models, virtual representations, schematics, specifications, designs, photographic records, silhouette images, etc.). The electronic or digital information can also include maintenance information about the physical object 204 (e.g., maintenance actions and messages, component installation and removal, etc.). In addition, the electronic or digital information can include information about the systems, subsystems, components, parts, etc. of the physical object 204. The electronic or digital information stored in the database 212 can be accessed and / or retrieved based on the object's coordinate system and / or its correlation with the position or orientation (e.g., coordinates) of the physical object 204. The electronic or digital information stored in the database 212 can also contain information in a graph database format that stores data as nodes (entities) and the relationships between them. These relationships are represented as edges that can have various attributes.

[0044] The portable electronic device 208 of the system 200 can be configured to capture physical or sensor data about the physical object 204 and determine the earth-centered orientation data (e.g., GPS orientation information) of the portable electronic device 208 within the real-world environment 206 and the object-centered relative orientation data (e.g., aircraft-centered). Based on the sensor data and / or orientation data, the portable electronic device 208 can be configured to determine the orientation of the portable electronic device 208 within the real-world environment and relative to the physical object 204. The portable electronic device 208 can also be configured to measure the distance or range from the portable electronic device 208 to the region of interest 202 (e.g., anomaly, defect, fault, component, part, damage, condition, etc.) of the physical object 204. In some examples, the portable electronic device 208 can use a measuring device to project a laser or light beam 218 onto the physical object to illuminate the region of interest 202 of the physical object 204.

[0045] Once the portable electronic device 208 determines the distance to the region of interest 202 of the physical object 204, the portable electronic device 208 can determine the orientation or position of the region of interest 202 of the physical object 204 in the physical coordinate system. For example, the portable electronic device 208 can determine the coordinates of the region of interest 202 in the real-world environment or the physical coordinate system of the portable electronic device 208. The portable electronic device 208 can convert the orientation of the region of interest 202 in the physical coordinate system into that of the physical object (e.g., Figure 2the position in the coordinate system of the aircraft 100), as further described below. For example, the portable electronic device 208 can transform or convert the orientation of the region of interest 202 of the physical object 204 in the physical coordinate system to a corresponding position in the coordinate system of the physical object (e.g., the virtual coordinate system of the digital model of the physical object 204).

[0046] After determining the orientation of the region of interest 202 of the physical object 204 in the coordinate system of the object, the portable electronic device 208 can identify and obtain electronic or digital information associated with or related to the region of interest 202 of the physical object 204 (e.g., digital model, virtual representation, schematic diagram, maintenance report, specifications, design, installation drawing, digital twin, etc.). The portable electronic device 208 can retrieve the electronic or digital information from the database 212 of the system 200 or from the memory of the portable electronic device 208. To obtain the electronic or digital information, the portable electronic device 208 can generate a query based on the orientation of the region of interest 202 of the physical object 204 in the coordinate system of the object. The portable electronic device 208 can use the query to retrieve the electronic or digital information from the database 212 and / or the memory of the portable electronic device 208. For example, the portable electronic device 208 can retrieve electronic or digital information that matches or is related to the region of interest 202 of the physical object 204. In some examples, the portable electronic device 208 can retrieve a 3-D digital model of the physical object (e.g., a digital model of an aircraft) or a 3-D model corresponding to the region of interest of the physical object (e.g., a digital model of the structure, system, or component of an aircraft).

[0047] The portable electronic device 208 can display the electronic or digital information obtained from the memory or the database 212 (e.g., virtual representation, schematic diagram, specifications, etc.) and / or send the electronic or digital information to other communication devices, such as the communication devices 214 and 216. For example, the portable electronic device 208 can display a virtual representation of the region of interest 202 of the physical object 204 and send the virtual representation to the communication devices 214 and 216. The communication devices 214 and 216 can be configured to display the electronic or digital information received from the portable electronic device 208. In some examples, the portable electronic device 208 can send the orientation of the region of interest 202 of the physical object 204 in the coordinate system of the object to the communication devices 214 and 216. The communication devices 214 and 216 can use the orientation of the region of interest 202 to access electronic or digital information from the database 212 or the local memory. For example, on-site and / or off-site analysts can use the communication devices 214 and 216 to access and retrieve electronic or digital information from the database 212 based on the position or orientation of the region of interest 202 of the physical object 204 in the coordinate system of the object.

[0048] AsFigure 4A and Figure 4B As shown, the portable electronic device 208 of the system 200 can be a compact device that can be handled or carried by a user or a maintenance person. The portable electronic device 208 can include a smart phone, a tablet computer, a laptop computer, or any other suitable device. As Figure 4A shown, the portable electronic device 208 can display an image of an area of interest of a physical object along with a graphical or virtual representation of the area of interest. The portable electronic device 208 can correlate and match a low-fidelity digital image (model-based) with a high-fidelity digital image (photo database-based). For example, once the portable electronic device 208 determines the orientation of the area of interest or object (in both the digital world and the physical world), the portable electronic device 208 can match the low-fidelity digital image (model-based) with the high-fidelity digital image (photo database-based). As an example, a user or mechanic performing an assessment to verify that wiring is placed in the correct orientation can view the real or physical world (e.g., inside an aircraft) as well as digital photos from a database (e.g., the digital world) that show where the wires should be. In some examples, the portable electronic device 208 can include a wearable device (e.g., a pair of augmented reality glasses). For example, the wearable device can provide a real-time view of the real-world environment by overlaying a representation of the real-world environment on a transparent or semi-transparent display that functions similarly to eyeglass lenses, enabling the user to view the real-world environment through the display.

[0049] Now referring to Figure 5 , a schematic illustration of components of the portable electronic device 208 is shown. The portable electronic device 208 can include a communication unit 520, a sensor system 522, a storage device 524 (e.g., a memory or a database), a processing unit 526, a user interface 528, and a display device 530. In other examples, the portable electronic device 208 can include additional components, hardware, or functionality. A bus 532 can couple the communication unit 520, the sensor system 522, the storage device 524, the processing unit 526, the user interface 528, and the display device 530 together to enable communication between them. Although only one bus is depicted, the portable electronic device 208 can include multiple buses or other types of communication paths between any of its elements or components.

[0050] The communication unit 520 of the portable electronic device 208 can be configured to connect to a communication network (e.g., Figure 3 the communication network 210). The communication unit 520 can receive data from other devices within the communication network, such as telecommunications and / or computing devices (e.g Figure 3The communication devices 214 and 216)) receive and send data / communications to the other devices. The communication unit 520 enables the portable electronic device 208 to communicate with other devices via a wireless channel or a wired communication link. For example, the communication unit 520 may enable the portable electronic device 208 to wirelessly transmit the location or orientation of the region of interest of a physical object (e.g., Figure 3 the physical object 204) to other devices. The communication unit 520 may also enable the portable electronic device 208 to wirelessly transmit electronic or digital information (e.g., virtual representations, schematics, repair reports, drawings, component information, etc.) retrieved from the storage device 524 to other devices.

[0051] The communication unit 520 may include a wireless connection, a wired connection, a cable connection, an optical fiber connection, etc., and may communicate via a wide area network (WAN), a local area network (LAN), a cellular network, a peer-to-peer communication network, or any other suitable network. The communication unit 520 may also operate to interface with a communication network using any type of communication protocol, such as, for example, Wi-Fi (e.g., 802.xx protocol), radio frequency (RF) protocol (e.g., 900 MHz, 1.4 GHz, and 5.6 GHz), cellular communication protocol (e.g., 2G, 3G, 4G, 5G, etc.), or any other communication protocol.

[0052] The sensor system 522 of the portable electronic device 208 may be configured to capture and collect physical or sensor data (e.g., image data, range, distance, location information, etc.) about one or more physical objects in the real-world environment. The sensor system 522 may include various types of sensors, such as a GPS sensor, an inertial measurement unit (IMU) or sensor, a measurement sensor, an image sensor (e.g., an image capture device for capturing images of physical objects in the real-world environment), or any other suitable sensor. The sensor system 522 may send the sensor data to the processing unit 526. The sensor data may be processed by the processing unit 526 to determine the location and orientation of the portable electronic device 208 in the real-world environment and / or the location and orientation relative to the physical objects in the real-world environment, as further described below.

[0053] The GPS sensor of the sensor system 522 can be configured to provide information about the location or orientation (e.g., orientation coordinates) of the portable electronic device 208 in the real-world environment. The IMU of the sensor system 522 can sense changes in the position and orientation of the portable electronic device 208 based on inertial acceleration. For example, the IMU can detect the pitch and yaw of the portable electronic device 208 when the portable electronic device 208 is stationary or in motion. The IMU can include one or more accelerometers that generate accelerometer sensor data. The one or more accelerometers can be used to measure static acceleration (such as the tilt of the portable electronic device 208 relative to gravity) and dynamic acceleration caused by the movement of the portable electronic device 208. The IMU can also include one or more gyroscopes that are configured to generate sensor data indicating the current orientation or orientation of the portable electronic device 208.

[0054] The image capture device or sensor of the sensor system 522 can be configured to capture image data of the real-world environment within its field of view. The image capture device can add geographic orientation data to the metadata field of the captured image data. The captured image data can be used to determine the orientation or position of the portable electronic device 208 relative to physical objects in the real-world environment. The captured image data can also be displayed on the display device 530 of the portable electronic device 208. In some examples, the image capture device can be a camera that includes three-dimensional capabilities. As Figure 4B shown, the imaging capture device can be positioned at the rear or back of the portable electronic device 208.

[0055] The measurement sensor of the sensor system 522 can be configured to measure the range and / or distance to an area of interest of a physical object in the real-world environment from the portable electronic device 208. For example, the portable electronic device 208 can measure the distance to an anomaly (such as damage, defect, etc.) of a physical object (such as an aircraft) in the real-world environment from the portable electronic device 208. In some examples, the measurement device can be configured to project a laser or light beam onto the physical object to illuminate the area of interest of the physical object. The measurement sensor can include a light-emitting device, a laser device, an optical device, or any other suitable measurement sensor.

[0056] Still referring to Figure 5, the storage device 524 of the portable electronic device 208 may store physical or sensor data captured by the sensors of the sensor system 522. The storage device 524 may also store information related to one or more physical objects. For example, the storage device 524 may include electronic or digital information about a physical object (e.g., digital model, virtual representation, schematic, specifications, design, installation drawing, system information, etc.). Additionally, the storage device 524 may store a digital model and / or virtual representation of a physical object. The digital model may include representations of the structure, components, systems, and subsystems of the physical object. In some examples, the storage device 524 may store a digital model representing the fuselage of an aircraft. The storage device 524 may also store mapping or location data representing a map of physical objects in a real-world environment based on spatial or physical coordinates.

[0057] The storage device 524 of the portable electronic device 208 may also store program instructions executed or implemented by the processing unit 526 of the portable electronic device 208. The storage device 524 may include a physical, non-transitory, computer-readable memory that stores data on a temporary or permanent basis for use by the processing unit 526. The memory may include one or more volatile and / or non-volatile memory devices, such as random access memory (RAM), static random access memory (SRAM), dynamic RAM (DRAM), read-only memory (ROM), flash memory, or any other suitable medium or memory that can be used to store desired information (e.g., system information, virtual models, mapping information, etc.).

[0058] The processing unit 526 of the portable electronic device 208 may communicate with various components of the portable electronic device 208. The processing unit 526 may include one or more processors. For example, the processing unit 526 may include one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more digital signal processors (DSPs), one or more peripheral interface controllers (PICs), or another type of microprocessor.

[0059] The processing unit 526 may be configured to identify and select a digital model of a physical object in a real-world environment. The processing unit 526 may retrieve the digital model of the physical object from the storage device 524 or a remote database (such as Figure 3 the database 212). In some examples, the processing unit 526 may select and / or retrieve a digital model representing the fuselage of an aircraft. In other examples, the processing unit 526 may retrieve a digital model representing a building, vehicle, industrial facility, power plant, ship, spacecraft, submarine, or any other suitable object based on the physical object.

[0060] A digital model can represent a 3-D model or representation of a physical object, which includes the systems and structures of the physical object. In some examples, the digital model can be a computer-aided design (CAD) model, and a coordinate system can be used to identify the location of the spatial or virtual content of the digital model of the physical object. The digital model can be based on the design, testing, manufacturing, installation, and / or operation phases of the physical object.

[0061] Once the processing unit 526 selects and retrieves the digital model of the physical object, the user can use the portable electronic device 208 to identify the location or orientation of the region of interest of the physical object in the real-world environment. When the user uses the portable electronic device 208, the processing unit 526 can receive and collect physical or sensor data associated with the physical object in the real-world environment 206 from the sensor system 522. For example, the processing unit 526 can capture image data about the physical object in the real-world environment. Based on the sensor data, the processing unit 526 can be configured to determine the orientation of the portable electronic device 208 in the real-world environment and the orientation or location of the physical object in the real-world environment. For example, the processing unit 526 can use GPS data to determine the orientation of the portable electronic device 208 in the real-world environment (e.g., earth-centered or aircraft-centered (relative to the aircraft)), and can use the image data to determine the position and orientation of the portable electronic device 208 in the real-world environment (e.g., earth-centered or aircraft-centered (relative to the aircraft)). In addition, the processing unit 526 can be configured to determine the distance from the portable electronic device 208 to the physical object and / or an item or marker on the physical object in the real-world environment. In addition, the processing unit 526 can be configured to determine the orientation of the portable electronic device 208 relative to the physical object and the item or marker of the physical object.

[0062] The processing unit 526 can also be configured to map the sensor data of the real-world environment in a physical coordinate system or reference frame. For example, the processing unit 526 can generate mapping or location data representing a map based on spatial or physical coordinates of the real-world environment. The processing unit 526 can map the real-world environment to establish a relationship between the location or orientation of the portable electronic device 208 and the location of the physical object within the real-world environment 206, such that after mapping the real-world environment, specific position coordinates within the physical coordinate system are assigned to the physical object. The physical coordinate system can be based on the location of the portable electronic device 208 within the real-world environment 206. In some examples, the physical coordinate system can be three-dimensional and include three mutually perpendicular axes.

[0063] The processing unit 526 can also be configured to track the location and orientation of the portable electronic device 208 in a real-world environment. For example, the processing unit 526 can process sensor or physical data received from the sensor system 522 to determine the location and orientation of the portable electronic device 208 relative to physical objects in the real-world environment. As the portable electronic device 208 moves within a real-world environment (e.g., the fuselage of an aircraft), the processing unit 526 can be configured to track physical objects in the real-world environment 206 to determine the location and orientation of the portable electronic device 208 relative to physical objects and / or items or markers of the physical objects in the real-world environment. For example, the processing unit 526 can track changes in the proximity and angle of the portable electronic device 208 relative to a physical object or an item of a physical object in the real-world environment. In some examples, the item can include a fiducial marker. Based on the perceived changes in the real-world environment around the portable electronic device 208, the processing unit 526 can calculate the movement (e.g., translation and / or rotation) of the portable electronic device 208 and determine the current location and orientation of the portable electronic device 208 relative to physical objects in the real-world environment 206.

[0064] The processing unit 526 can be configured to align or spatially register a physical object in the real-world environment 206 with a digital model or representation of the physical object. The processing unit 526 can use a transformation or transfer function to align the physical object defined in a physical coordinate system with the digital or virtual content of the digital model of the physical object defined in a local or object coordinate system. Once the physical object in the real-world environment is aligned with the digital model of the object, the portable electronic device 208 can be configured to determine the location of the region of interest of the physical object in the object coordinate system.

[0065] As Figure 6 shown, the portable electronic device 208 can use triangulation techniques to determine the location or orientation of the region of interest 602 of the physical object 604 in the real-world environment 606. In other embodiments, the portable electronic device 208 can use trilateration techniques, multilateration techniques, or any other suitable technique to determine the orientation of the region of interest 602 of the physical object 604 in the real-world environment. As Figure 6 shown, the portable electronic device 208 can determine a first position (P1) of the portable electronic device at a first time. From the first position (P1), the portable electronic device 208 can determine the distance or length from the first position (P1) to the region of interest 602 of the physical object 604. At a second time, the portable electronic device 208 can determine a second position (P2) of the portable electronic device 208 and the distance or length from the second position (P2) to the region of interest 602 of the physical object 604.

[0066] After the portable electronic device 208 determines the orientations of the first position and the second position, the portable electronic device 208 can calculate the distance (D) between the first position (P1) and the second position (P2). Based on these calculations, the portable electronic device 208 can calculate the orientation of the region of interest 602 of the physical object 604 based on the following expression:

[0067] tan(∠1) = d / x (1)

[0068] tan(∠2) = d / (D - x) (2)

[0069] d = x*tan(∠1) (3)

[0070] d = (D - x)*tan(∠2) (4)

[0071] x*tan(∠1) = (D - x)*tan(∠2) (5)

[0072] x*tan(∠1) = D*tan(∠2) - x*tan(∠2) (6)

[0073] x*tan(∠1) + x*tan(∠2) = D*tan(∠2) (7)

[0074] x*[tan(∠1) + tan(∠2)] = D*tan(∠2) (8)

[0075] x = D*tan(∠2) / [tan(∠1) + tan(∠2)] (9)

[0076] The portable electronic device 208 can use other techniques to determine the location or orientation of the region of interest 602 of the physical object 604 in the real-world environment 606. As Figures 7A - 7E shown, the portable electronic device can use trilateration with three (3) known orientations of the portable electronic device relative to the object to determine the location or orientation of the electronic device. Initially, the portable electronic device can be geolocated at three (3) positions relative to three (3) known and / or fixed features or items (P1, P2, and P3) of the object, as Figures 7A - 7C shown. For example, for an aircraft, the portable electronic device can determine its orientation within the aircraft based on three known and fixed items within the aircraft (such as RFID chips, QR codes at known reference points, and door hinges). The portable electronic device can determine the first orientation or position (PA) of the portable electronic device relative to the three known items (P1, P2, and P3) at a first time, as Figure 7AAs shown. The portable electronic device can also determine a second orientation (PB) of the portable electronic device relative to three known items (P1, P2, and P3) at a second time, as Figure 7B shown. In addition, the portable electronic device can determine a third orientation (PC) of the portable electronic device relative to three known items (P1, P2, and P3) at a third time, as Figure 7C shown.

[0077] After the portable electronic device determines three orientations of the portable electronic device relative to three known items, the portable electronic device can determine the position or orientation of the region of interest of the physical object in the real world based on the first position (PA), the second position (PB), and the third position (PC) of the portable electronic device, as Figure 7D shown. The portable electronic device can calculate the distance from the first position (PA) region to the region of interest (PX), calculate the distance from the second position (PB) to the region of interest (PX), and calculate the distance from the third position (PC) to the region of interest (PX). Based on these calculations, the portable electronic device 208 can calculate the orientation of the region of interest (PX) (e.g., the region of interest of the object) in three dimensions based at least on the following expressions, as Figure 7D – Figure 7E shown:

[0078] (D z – A z ) 2 + (D y – A y ) 2 - |AD| 2 = 0 (10)

[0079] (D z – B z ) 2 + (D y – B y ) 2 - |BD| 2 = 0 (11)

[0080] (D z – C z ) 2 + (D y – C y ) 2 - |CD| 2 = 0 (12)

[0081] D z =(((-A x ) 2 +(Bx ) 2 -(A y ) 2 +(B y ) 2 -|BD| 2 +|AD| 2 ) / 2)-(D y (B y -A y )) / (B x -A x ) (13)

[0083] where A(A z ,A y ) is the orientation of PA, where B(B z ,B y ) is the orientation of PB, where C(C z ,C y ) is the orientation of PC, where D(D z ,D y ) is the orientation of PX, and where the distances |AD|, |BD|, |CD| are known.

[0084] Referring again to Figure 6 , the portable electronic device 208 can utilize any suitable coordinate system to determine the position of the portable electronic device 208 within a real-world environment and / or the position of the region of interest 602 of the physical object 604. For example, the portable electronic device 208 can use a geodetic coordinate system, where the orientation on the Earth is specified by longitude (e.g., in degrees east or west of the prime meridian) and latitude (e.g., in degrees north or south of the equator), and altitude is specified by the height above mean sea level (MSL). This coordinate system provides spherical coordinates (e.g., approximating the shape of the Earth).

[0085] The portable electronic device 208 can also use a local coordinate system of east, north, and up (ENU). In this coordinate system, the orientation is specified by units east and north of the origin of the coordinate system (e.g., by geodetic surveying located on the Earth), and altitude is specified by the height above mean sea level (MSL). Additionally, the portable electronic device 208 can use a local coordinate system of north, east, and down (NED). This coordinate system is similar to the ENU system, where the x-component is the same as the east component of the ENU, the y-component is the same as the up component of the ENU, and the z-component is the negative of the north component of the ENU. The NED coordinate system is similar to the Earth-centered, Earth-fixed (ECEF) coordinate system. The relationship between the NED coordinate system and the ECEF coordinate system is given by the following expression:

[0086] P NED =RT (P ECEF – P Ref )

[0087] where P NED is a 3D position in the NED frame, P ECEF is the corresponding ECEF position, P Ref is a reference ECEF position (where the local tangent plane originates), and where R is a rotation matrix with columns on the north, east, and down axes and can be defined as follows in terms of the latitude φ and longitude λ corresponding to P Ref :

[0088]

[0089] In some embodiments, the portable electronic device 208 may use GPS orientation information to determine the orientation of an area of interest 602 of a physical object 604 (e.g., an aircraft) in the real-world environment (e.g., earth-centered or object / aircraft-centered (relative to the object / aircraft)). For example, the portable electronic device 208 may determine the GPS orientation of the area of interest 602 and the GPS orientation of a known fixed item or marker of the physical object 604 (e.g., an aircraft). The portable electronic device 208 may then calculate the object-centered orientation (e.g., aircraft-centered orientation) of the area of interest 602 relative to the known fixed item or marker of the object (e.g., the aircraft). The portable electronic device 208 may also use GPS orientation information to determine the orientation of the portable electronic device 208 in the real-world environment (e.g., earth-centered or object-centered (relative to the object)).

[0090] Referring again to Figure 5 , once the processing unit 526 of the portable electronic device 208 determines the orientation of an area of interest of a physical object in the real-world environment defined in a physical coordinate system, the processing unit 526 may be configured to determine the orientation of the area of interest of the physical object in terms of the coordinates of the coordinate system of the physical object. The processing unit 526 may transform or convert the coordinates of the orientation of the area of interest of the physical object in the real-world environment to the coordinates of the coordinate system of the physical object using a transformation or transfer function. In some examples, the coordinates of the coordinate system of the physical object may be three-dimensional coordinates defined along three mutually perpendicular axes within the coordinate system of the object.

[0091] Once the coordinates of the region of interest of the physical object are determined in the coordinate system of the object, the processing unit 526 can retrieve electronic or digital information about the region of interest of the physical object from the storage device 524 or a database. The processing unit 526 can be configured to search for electronic or digital records and reports (e.g., damage report records, maintenance requests, etc.) in one or more databases for information related to the region of interest of the physical object. For example, the processing unit 526 can search for electronic reports and records for issues related to the damage orientation of a particular type of vehicle or aircraft. The processing unit 526 can identify the relevant electronic or digital records and cause the relevant electronic or digital records to be displayed to the support or maintenance personnel. For example, the processing unit 526 can be configured to cause electronic or digital information (e.g., a graphical or virtual representation of the region of interest) to be displayed on the display device 530 of the portable electronic device 208. Additionally, the processing unit 526 can send the electronic or digital information to other communication or computing devices to enable remote personnel to analyze the region of interest of the physical object. Further, the processing unit 526 can also store or record any current damage report of the physical object as a new record or report in the database. For example, the processing unit 526 can create and store a maintenance request or problem report (including a description and type of the damage, object or vehicle type, photos and sketches of the damage, etc.) in a database (e.g., an airline database) based on the physical orientation of the damage.

[0092] Still referring to Figure 5 , the user interface 528 of the portable electronic device 208 can allow a user or a maintenance person to interact with the portable electronic device 208. The user interface 528 can include an interactive touch screen. In other examples, the user interface 528 can include a keyboard, a mouse, a microphone, or any other suitable input / output device. The user interface 528 of the portable electronic device 208 can be configured to receive inputs and / or user selections to enable the portable electronic device 208 to select one or more digital models representing the physical object. For example, a user can input information related to a physical object (e.g., an aircraft) in the real-world environment, a system of the physical object (e.g., the electrical wiring system of an aircraft), components of the system, and / or other information related to the physical object in the user interface 528. After receiving the information, the portable electronic device 208 can display graphical or virtual content related to the region of interest of the physical object on the user interface 528 or the display device 530, as further described below.

[0093] The display device 530 of the portable electronic device 208 can be configured to present visual, audio, and / or tactile information to a user or a maintenance person. The display device 530 can include a screen or any other suitable type of display. In other examples, the display device 530 can be integrated into a transparent or translucent face mask of an optical see-through augmented reality (AR) imaging device and can be viewed by a user or a technician wearing the AR imaging device.

[0094] The display device 530 of the portable electronic device 208 can be configured to display a live view of the real-world environment such that a user can view a representation of a physical object in the real-world environment in real time. In some examples, the display device 530 can display a representation of the fuselage (or a part thereof) of an aircraft. The display device 530 can also display augmented reality (AR) content, such as graphics or virtual content. The display device 530 of the portable electronic device 208 can also be configured to display a graphical or virtual representation of an area of interest of a physical object and / or an image of the area of interest, as Figure 4A shown.

[0095] Figure 8 FIG. 800 is a flowchart of a method for obtaining electronic or digital information (such as a 3D digital model, a virtual representation, a schematic diagram, a drawing, etc.) related to an area of interest of a physical object (such as an aircraft) in a real-world environment according to an exemplary embodiment. The method can be executed or implemented entirely or partially by a portable electronic device or an AR device (such as Figure 3 the portable electronic device 208).

[0096] The portable electronic device can identify and capture data of a physical object in the real-world environment. The physical object can be a vehicle, an aircraft, a building, an industrial facility, a power plant, a ship, a spacecraft, a submarine, or any other physical object. The portable electronic device can be configured to retrieve a digital model of the physical object from a memory or a database. The digital model can be a computer-aided design (CAD) model. The digital model of the physical object can include the structure and systems of the physical object. In some examples, the digital model can represent one or more systems of an aircraft. For example, the systems can include a hydraulic system, an air trim system, an environmental system, a flight management system, a navigation system, a communication system, a sensor system, a propulsion system, a flight control system, an electrical system, a pneumatic system, a guidance system, a radar system, an air conditioning system, a blower system, an intake system, and / or any other electronic, mechanical, and / or hardware systems.

[0097] The digital model can be stored in a storage device (such as a database) of the portable electronic device. In some examples, the digital model can be generated and stored in a remote or separate database (such as, Figure 3at the database 212), and the portable electronic device can access and / or retrieve the digital model from a remote or separate database. The digital model can be constructed or generated during the design, testing, manufacturing, installation, and / or operation phases of the physical object. The portable electronic device can be configured to align or spatially register the digital model of the physical object with the corresponding physical object in the real-world environment. For example, the portable electronic device can use a transformation or transfer function to align the digital model defined in the coordinate system of the object with the physical object in the real-world environment defined in the physical coordinate system. For example, the portable electronic device can transform or convert the coordinates of the physical object in the real-world environment into position coordinates in the coordinate system of the object.

[0098] At block 802, method 800 involves determining one or more positions of the orientation of the portable electronic device relative to one or more items of the object. Once the digital model of the object is aligned with the physical object in the real-world environment, the portable electronic device can receive and / or capture physical or sensor data associated with the physical object in the real-world environment. For example, the portable electronic device can receive sensor data (e.g., GPS data, image data, etc.) about the physical object in the real-world environment. In Figure 3 the example shown, the portable electronic device can capture image data about the fuselage of an aircraft.

[0099] The portable electronic device can determine the position of the portable electronic device in the real-world environment and / or the position relative to the physical object based on the sensor data. The portable electronic device can be configured to map the sensor data of the physical object in a physical coordinate system or reference frame. For example, the portable electronic device can generate mapping or position data representing a map based on spatial or physical coordinates of the physical object in the real-world environment. The portable electronic device can map the physical environment to establish a relationship between the position of the portable electronic device and the position of the physical object in the real-world environment, such that after mapping the physical object in the real-world environment, specific position coordinates within the physical coordinate system can be assigned to the physical object. The physical coordinate system can be based on the position of the portable electronic device within the real-world environment.

[0100] The portable electronic device may also be configured to track the position and orientation of the portable electronic device relative to physical objects in a real-world environment. The portable electronic device may process sensor or physical data to determine the position and orientation of the portable electronic device relative to the real-world environment and / or relative to physical objects. As the portable electronic device moves within a real-world environment (e.g., within the fuselage of an aircraft), the portable electronic device may track physical objects in the real-world environment to determine the position and orientation of the portable electronic device in the physical environment relative to the physical objects. For example, the portable electronic device may track changes in the proximity and angle of the portable electronic device relative to a physical object or an item or marker of a physical object in the real-world environment. In some examples, the physical object may include a fiducial marker. Based on the perceived changes in the real-world environment surrounding the portable electronic device, the portable electronic device may calculate the movement (e.g., translation and / or rotation) of the portable electronic device and determine the current position and orientation of the portable electronic device relative to the physical object.

[0101] After the portable electronic device determines the orientation of the portable electronic device relative to the physical object, the portable electronic device may use a measuring device to measure the distance or range from the portable electronic device to an area of interest (e.g., an anomaly, defect, fault, component, part, and / or condition) of the physical object. The portable electronic device may measure the distance from the portable electronic device to the area of interest of the physical object at one or more times or positions.

[0102] At block 804, method 800 involves determining the orientation of an area of interest of an object relative to one or more positions of the portable electronic device. After the portable electronic device determines the range / distance to the area of interest of the physical object, the portable electronic device may determine the position of the area of interest of the physical object in the real-world environment defined in a physical coordinate system. The portable electronic device may transform or convert the coordinates of the position of the area of interest of the physical object in the real-world environment to coordinates in the coordinate system of the object using a transformation or transfer function. The coordinates in the coordinate system of the object may be three-dimensional coordinates defined along three mutually perpendicular axes within the coordinate system of the object.

[0103] At block 806, method 800 involves identifying electronic or digital information associated with the orientation of an object's region of interest. After determining the orientation of the region of interest of a physical object in the object's coordinate system, a portable electronic device can identify and retrieve electronic or digital information (e.g., digital models, virtual representations, schematics, maintenance reports, specifications, designs, installation diagrams, digital twins, etc.) associated with or related to the region of interest of the physical object. The portable electronic device 208 can retrieve the electronic or digital information from a remote database and / or the memory of the portable electronic device. In some examples, the portable electronic device 208 can retrieve a 3-D digital model of the physical object (e.g., a digital model of an aircraft) or a 3-D model corresponding to the region of interest of the physical object (e.g., a digital model of the structure, system, or component of an aircraft).

[0104] At block 808, method 800 involves sending the electronic or digital information or the orientation of the region of interest to a remote computing device. Once the electronic or digital information is retrieved from the memory or database, the portable electronic device can send the electronic or digital information to other communication or computing devices. In some examples, the portable electronic device can send the orientation of the region of interest of the physical object in the object's coordinate system to other communication devices. The communication device can use the orientation of the region of interest 202 to access the electronic or digital information from a remote database or local memory.

[0105] By using the portable electronic device of the present application, maintenance personnel and technicians can efficiently troubleshoot complex physical objects (such as vehicles, machines, or structures). In addition, the time required to troubleshoot anomalies of physical objects in the real-world environment can be significantly reduced. For the aviation industry, the systems disclosed herein can reduce the number of flights delayed or cancelled due to repairs and maintenance.

[0106] Although the disclosed systems have been generally described and illustrated in connection with aircraft, these systems can be used to locate defects or faults in any physical object (such as complex systems created in the automotive, marine, electronics, power generation, and computer industries). Thus, the above description of the use of the disclosed systems and methods in aircraft is for illustrative and example purposes and not for purposes of limitation, as the above systems and methods are equally applicable to many different industries.

[0107] In addition, the description of the different advantageous arrangements has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the examples of the forms disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. Further, different advantageous examples describe different advantages compared to other advantageous examples. The one example or more examples selected were selected and described in order to best explain the principles of the examples, the practical application, and to enable those of ordinary skill in the art to understand the various examples of the disclosure and the various modifications suitable for the particular intended use.

[0108] The embodiments described herein can be implemented in hardware, software, or a combination of hardware and software. For example, the embodiments can be implemented in a centralized manner in at least one computer system, or in a distributed manner in which different elements are distributed across interconnected computer systems. Any kind of computer system or other device suitable for implementing the methods described herein can be employed. Further, the embodiments described herein can be embedded in a computer program product that includes all the features of the implementation of the operations described herein and, when loaded into a computer system, can implement these operations.

[0109] The flowcharts and block diagrams described herein illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various illustrative embodiments. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified one or more logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, depending on the functions involved, the functions of two consecutive blocks shown may be executed substantially simultaneously, or the functions of the blocks may sometimes be executed in the reverse order.

[0110] In addition, in this specification, instances where one element is "coupled" to another element can include direct and indirect coupling. Direct coupling can be defined as one element being coupled to another element and having some kind of contact with that other element. Indirect coupling can be defined as a coupling between two elements that do not directly contact each other but have one or more additional elements between the coupled elements. Further, as used herein, fixing one element to another element can include direct fixing and indirect fixing. Further, as used herein, "adjacent" does not necessarily mean in contact. For example, one element can be adjacent to another element without contacting that element.

[0111] As used herein, a system, apparatus, structure, article, element, component, or hardware that is "configured to" perform a specified function is indeed capable of performing the specified function without any further modification, rather than simply having the potential to perform the specified function after further modification. In other words, a system, apparatus, structure, article, element, component, or hardware that is "configured to" perform a specified function has been specifically selected, created, implemented, used, programmed, and / or designed for the purpose of performing the specified function. As used herein, being "configured to" denotes a characteristic of the existence of a system, apparatus, structure, article, element, component, or hardware that enables the system, apparatus, structure, article, element, component, or hardware to perform the specified function without further modification. For the purposes of this disclosure, a system, apparatus, structure, article, element, component, or hardware described as being "configured to" perform a particular function may additionally or alternatively be described as "adapted to" and / or "operable to" perform that function.

[0112] As used herein, the terms "substantially" and "about" mean that the characteristic, parameter, or value need not be precisely achieved, but that deviations or variations (including, for example, tolerances, measurement errors, measurement accuracy limitations, and other factors known to those of skill in the art) may occur in amounts that do not exclude the effect the characteristic is intended to provide.

[0113] Unless otherwise indicated, the terms "first," "second," etc. are used herein only as labels and are not intended to impose an order, position, or hierarchical requirement on the items to which these terms refer. Further, a reference to, for example, a "second" item does not require or preclude the existence of, for example, a "first" or lower-numbered item and / or, for example, a "third" or higher-numbered item.

[0114] Although the apparatus has been described with reference to certain examples, those skilled in the art will understand that various changes may be made and equivalents may be substituted without departing from the scope of the claims. Accordingly, it is intended that the apparatus not be limited to the particular examples disclosed, but that the disclosed apparatus include all embodiments falling within the scope of the appended claims.

[0115] Furthermore, the present invention also includes the following examples.

[0116] Example 1. A portable electronic device for obtaining information about an area of interest of an object, comprising:

[0117] A display;

[0118] An image capture device;

[0119] A measuring device configured to measure a distance or an angle to a spatial point; and

[0120] A processor configured to:

[0121] Determine one or more positions of the portable electronic device relative to one or more items of the object;

[0122] Determine the orientation of the region of interest of the object relative to the one or more positions of the portable electronic device;

[0123] Identify electronic or digital information associated with the orientation of the region of interest of the object; and

[0124] Send the electronic or digital information or the orientation of the region of interest to a remote computing device.

[0125] Example 2. The portable electronic device according to Example 1, wherein the orientation of the region of interest of the object includes three-dimensional coordinates.

[0126] Example 3. The portable electronic device according to Example 1, wherein the one or more items of the object are located in a fixed and known orientation, wherein the region of interest of the object corresponds to an anomaly, damage, crack, condition, defect, item, part, component, article, feature, point or portion of the object, and wherein the object includes a vehicle, aircraft, ship, building, spacecraft or submarine.

[0127] Example 4. The portable electronic device according to Example 1, wherein each of the one or more items includes a marker, article, feature or object, and wherein the one or more positions of the portable electronic device are defined in a physical coordinate system.

[0128] Example 5. The portable electronic device according to Example 1, wherein the processor is further configured to align or correlate the object with a digital model of the object.

[0129] Example 6. The portable electronic device according to Example 1, wherein the orientation of the region of interest of the object is defined in a physical coordinate system.

[0130] Example 7. The portable electronic device according to Example 6, wherein the orientation of the region of interest of the object corresponds to a three-dimensional orientation of the region of interest of the object in the coordinate system of the object.

[0131] Example 8. The portable electronic device according to Example 6, wherein the processor is further configured to use a transfer function to convert the orientation of the region of interest of the object into coordinates in the coordinate system of the object.

[0132] Example 9. The portable electronic device according to Example 6, wherein triangulation, trilateration or multilateration techniques are used to determine the orientation of the region of interest of the object in the physical coordinate system.

[0133] Example 10. The portable electronic device according to Example 1, wherein determining the one or more positions of the portable electronic device further comprises:

[0134] determining a first position of the orientation of the portable electronic device relative to one or more items of the object;

[0135] determining a second position of the orientation of the portable electronic device relative to one or more items of the object; and

[0136] determining a third position of the orientation of the portable electronic device relative to one or more items of the object.

[0137] Example 11. The portable electronic device according to Example 10, wherein the processor is further configured to:

[0138] determine the distance between the first position of the portable electronic device and the region of interest of the object;

[0139] determine the distance between the second position of the portable electronic device and the region of interest of the object; and

[0140] determine the distance between the third position of the portable electronic device and the region of interest of the object.

[0141] Example 12. The portable electronic device according to Example 1, wherein the measuring device includes a light-emitting device, a laser device or an optical device.

[0142] Example 13. The portable electronic device according to Example 1, wherein the display is configured to display a virtual or digital representation of the region of interest of the object based on the electronic or digital information.

[0143] Example 14. The portable electronic device according to Example 1, wherein the electronic or digital information includes digital pictures, technical documents, schematic diagrams, repair reports, system data, photo records, silhouette images or combinations thereof.

[0144] Example 15. The portable electronic device according to Example 1, wherein the processor is further configured to retrieve the electronic or digital information from a memory or a database.

[0145] Example 16. The portable electronic device according to Example 1, wherein the portable electronic device includes an augmented reality device having a head-mounted device configured to be worn by a user.

[0146] Example 17. A method for obtaining information about an area of interest of an object, the method comprising:

[0147] determining, by one or more processors, one or more positions of an orientation of the portable electronic device relative to one or more items of the object;

[0148] determining, by the one or more processors, an orientation of the area of interest of the object relative to the one or more positions of the portable electronic device;

[0149] identifying, by the one or more processors, electronic or digital information associated with the orientation of the area of interest of the object; and

[0150] sending the electronic or digital information or the orientation of the area of interest to a remote computing device.

[0151] Example 18. The method according to Example 17, wherein the orientation of the area of interest of the object is defined in a physical coordinate system, and the method further comprises:

[0152] converting, using a transfer function, the orientation of the area of interest of the object to coordinates in a coordinate system of the object.

[0153] Example 19. The method according to Example 18, wherein the object includes a vehicle, an aircraft, a ship, a building, a spacecraft, or a submarine, wherein the one or more items of the object are located at fixed and known orientations, and wherein triangulation, trilateration, or multilateration techniques are used to determine the orientation of the area of interest of the object in the physical coordinate system.

[0154] Example 20. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for obtaining information about an area of interest of an object, the operations comprising:

[0155] determining one or more positions of an orientation of the portable electronic device relative to one or more items of the object;

[0156] determining an orientation of the area of interest of the object relative to the one or more positions of the portable electronic device;

[0157] Identify electronic or digital information associated with the orientation of the region of interest of the object; and

[0158] Send the electronic or digital information or the orientation of the region of interest to a remote computing device.

Claims

1. A portable electronic device (208) for obtaining information about a region of interest (202, 602) of an object, comprising: Display (530); Image capture device; a measuring device configured to measure distances or angles to points in space; as well as A processor (526) configured to: determining one or more positions of the portable electronic device relative to the orientation of one or more items of the object; determining a location of the region of interest of the object relative to the one or more locations of the portable electronic device; identifying electronic or digital information associated with the location of the region of interest of the object; as well as The electronic or digital information or the location of the area of ​​interest is sent to a remote computing device. 2 . The portable electronic device of claim 1 , wherein the location of the region of interest of the object comprises three-dimensional coordinates.

3. A portable electronic device according to claim 1, wherein the one or more items of the object are positioned at a fixed and known position, wherein the area of ​​interest of the object corresponds to an anomaly, damage, crack, condition, defect, item, part, component, object, feature, point or portion of the object, and wherein the object includes a vehicle, an aircraft (100), a ship, a building, a spacecraft or a submarine.

4. The portable electronic device of claim 1, wherein each of the one or more items comprises a tag, an article, a feature, or an object, and wherein the one or more positions of the portable electronic device are defined in a physical coordinate system. 5 . The portable electronic device of claim 1 , wherein the processor is further configured to align or correlate the object with a digital model of the object. The portable electronic device of claim 1 , wherein the location of the region of interest of the object is defined in a physical coordinate system. 7 . The portable electronic device of claim 6 , wherein the position of the region of interest of the object corresponds to a three-dimensional position of the region of interest of the object in a coordinate system of the object. 8 . The portable electronic device of claim 6 , wherein the processor is further configured to convert the position of the region of interest of the object into coordinates in a coordinate system of the object using a transfer function.

9. The portable electronic device of claim 6, wherein the position of the region of interest of the object in the physical coordinate system is determined using triangulation, trilateration, or multilateration techniques.

10. The portable electronic device of claim 1, wherein determining the one or more locations of the portable electronic device further comprises: determining a first position of the portable electronic device relative to the orientation of the one or more items of the object; determining a second position of the portable electronic device relative to the orientation of the one or more items of the object; as well as A third position of the portable electronic device relative to the orientation of the one or more items of the object is determined.

Images