System, method, and apparatus for correlating location of real-world object to digital world

Abstract

To provide a portable electronic device for acquiring information on an area of interest of an object.SOLUTION: A portable electronic device 208 may include a display, an image capture device, a measurement device, and a processor. The processor may be configured to determine one or more positions of the portable electronic device 208 relative to the location of one or more items of an object and determine the location of an area of interest of the object relative to the one or more positions of the portable electronic device 208. The processor may also be configured to identify electronic or digital information associated with the location of the area of interest of the object and send the electronic or digital information or the location of the area of interest to a remote computing device.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001]

[0001] The present disclosure relates generally to data processing systems, and more particularly to systems and methods for determining the location of an area of interest of a physical object (e.g., a vehicle) within a real-world environment and obtaining electronic or digital information (e.g., a digital model, virtual representation, schematic diagram, etc.) related to the area of interest of the physical object. [Background technology]

[0002]

[0002] This background art discussion is provided for the purpose of broadly presenting the context for the present disclosure. Unless otherwise stated herein, no admission, express or implied, is made that the material described in this section is prior art to the present disclosure or the appended claims.

[0003] Many complex objects and structures, such as large transport aircraft, are composed of multiple elements and systems, and each system may include a significant number of components and / or parts. These complex objects are frequently inspected by on-site support personnel (e.g., maintenance personnel, inspectors, etc.) to maintain the object and repair defects (e.g., breakage, malfunctions, etc.) in the object. To analyze an area of the object, it would be desirable to determine the location and measurement information of the object's defects and obtain information related to the defects (e.g., schematics, drawings, maintenance records, etc.). However, obtaining information about defects in large, complex objects can be a daunting task. For example, support personnel may need to obtain and review numerous technical manuals and documents to find the appropriate information for troubleshooting or repairing the object's defects.

[0004] In some situations, on-site support personnel may communicate with on-site and / or off-site analysts or experts (e.g., service engineers, maintenance engineers, etc.) regarding defects in an object (e.g., a vehicle). Typically, on-site support personnel can use a camera to take photographs of the defect (e.g., damage) in the object and record the location of the defect from the on-site support personnel's location. For example, maintenance personnel can create a damage report by taking photographs of damage to a vehicle (e.g., an aircraft), recording the location of the damage, and sketching a diagram of the damaged area on the vehicle. FIG. 1 shows photographs for a lightning strike report showing the location of damage to a skin or known items and locations on an aircraft. The on-site support personnel can provide the damage photographs, damage report, and details of the damage location to on-site and off-site analysts. The on-site and / or off-site analyst (e.g., a structural engineer) can use the information to perform structural analysis, conduct engineering substantiation, complete a conformance inspection record, and / or submit records to the Federal Aviation Administration (FAA) for review.

[0005]

[0005] After the location information, photographs, and damage reports are received, on-site and / or off-site analysts may attempt to determine the location of the damage to the object by visually comparing the photographs with available documentation (e.g., drawings, technical manuals, etc.). However, determining the exact location of the damage may be difficult in areas where uniquely identifiable landmarks do not exist. Furthermore, on-site and off-site analysts can only obtain a subjective indication of the location of the damage to the object. As a result, damage assessment or analysis is likely to be performed with location errors. Additionally, human analysis of a large number of defects (e.g., damage) may be prone to error.

[0006]

[0006] Furthermore, photographs or images taken by support personnel's cameras and transmitted to on-site and / or off-site analysts may contain embedded metadata, such as GPS location data and camera setting information. The embedded metadata from the images can be used by on-site and / or off-site analysts to estimate the location of the camera relative to the object. However, without further context about the object's location (i.e., position and orientation) relative to the camera's frame of reference, the metadata may not be sufficient to determine the camera's location relative to the object. Furthermore, because the camera's location relative to the object may not be accurately determined, it may be difficult to locate defects on the object using the images.

[0007] It can also be difficult for on-site and / or off-site analysts to obtain accurate measurements of the location of defects in an object from a photograph or image. Often, an item of known dimensions (e.g., a tape measure) is inserted into the image to provide the analyst with a reference for size. However, even with a reference scale, it can be difficult to use the image to determine the exact location of the defect in the object's coordinate system. For at least these reasons, it would be advantageous to develop systems and methods that can determine the location of areas of interest (e.g., defects, damage, etc.) of a physical object within a real-world environment and obtain electronic or digital information related to the areas of interest of the physical object. Summary of the Invention

[0008] This application is directed to embodiments relating to systems, methods, and apparatus for determining the location or position of an area of interest of a physical object (e.g., a vehicle, an aircraft, a system, etc.) within a real-world environment. Embodiments can use the location of the area of interest to obtain electronic or digital information (e.g., a 3D digital model, a virtual representation, a schematic, an engineering drawing, etc.) related to the area of interest of the physical object. The area of interest may correspond to an anomaly, defect, damage, malfunction, component, part, object, item, and / or condition of the physical object.

[0009]

[0009] Embodiments may be used to inspect, troubleshoot, and / or repair physical objects. For example, embodiments may be used by support personnel to perform maintenance on physical objects. Embodiments may quickly and accurately determine the physical location or position of an area of interest relative to a physical object. For example, embodiments may determine the coordinates of an area of interest (e.g., an anomaly, defect, damage, etc.) of a physical object within the object's coordinate system (e.g., an object-centered coordinate system). In some examples, embodiments may calculate the physical location of an object of interest on or within a vehicle, such as an aircraft. As a result, the likelihood of human error in accurately determining the location of an area of interest of a physical object may be reduced.

[0010] Embodiments may convert the physical location of an area of interest of a physical object to a position within the object's coordinate system (e.g., a vehicle-centered coordinate system). Embodiments may obtain electronic or digital information about the area of interest of the physical object based on the location of the area of interest within the object's coordinate system. For example, embodiments may use the location of the area of interest of the physical object within the object's coordinate system to identify and retrieve electronic information and records related to the area of interest of the physical object (e.g., a vehicle, aircraft, etc.). The electronic information is organized or categorized using the object's coordinate system (e.g., a vehicle coordinate system) and may include CAD / CAM electronic records, 3D digital models, virtual representations, schematics, engineering drawings, technical documentation, maintenance reports, trouble reports, service bulletins, system reports, structural analysis, flight logs, and other information related to the area of interest of the physical object.

[0011] Embodiments may be configured to search electronic or digital records and reports (e.g., damage report records, service requests, etc.) in one or more databases for information related to an area of interest for a physical object. For example, embodiments may search electronic reports and records for problems related to the location of damage to a particular type of vehicle or aircraft. Embodiments may identify and display relevant electronic or digital records to support or maintenance personnel. Embodiments may also store or record any current damage reports for a physical object as a new record or report in a database. For example, embodiments may create and store a service request or problem report in a database (e.g., an airline database) based on the physical location of the damage, including a description and type of damage, the type of object or vehicle, a photograph or sketch of the damage, etc.

[0012]

[0012] Embodiments may transmit retrieved electronic and digital reports and records related to an area of interest of a physical object to devices of onsite and offsite analysts or specialists. For example, embodiments may transmit graphical and virtual representations of an area of interest of a physical object (e.g., a virtual representation of an aircraft cockpit) to onsite and / or offsite analysts. The electronic and digital information may assist the onsite and / or offsite analysts in analyzing the area of interest of the physical object. For example, embodiments may enable the onsite and offsite analysts to view electronic or digital information related to the area of interest of the physical object. As a result, the onsite and offsite analysts may not need to visit and inspect the physical object and may remotely determine repair of any defects (e.g., anomalies, damage, conditions, etc.) associated with the area of interest.

[0013] By enabling efficient and reliable determination of areas of interest of physical objects (e.g., vehicles, systems, etc.) within a real-world environment, embodiments can improve the process for troubleshooting and repairing physical objects. For example, embodiments may reduce the time required to identify and repair defects (e.g., defects, damage, cracks, conditions, failures, etc.) in a physical object and reduce the time required to obtain electronic and digital information related to the defect. The embodiments may also automate and speed the process for repairing physical objects (e.g., vehicles), improve the accuracy of locating defects, reduce errors in determining defect locations, and create new records useful for future repairs. Furthermore, embodiments may advantageously increase the reliability, safety, maintainability, and availability of physical objects (e.g., vehicles), resulting in improved performance and operational capabilities of the physical object. Furthermore, in the aviation industry, embodiments may reduce the number of flights delayed or canceled due to maintenance or repair issues.

[0014] In one aspect, a portable electronic device for obtaining information about an area of interest of an object is disclosed. The portable electronic device may include a display, an imaging device, a measuring device, and a processor. The processor may be configured to determine one or more positions of the portable electronic device relative to the location of one or more items of the object and to determine a location of the area of interest of the object relative to the one or more positions of the portable electronic device. The processor may also be configured to identify electronic or digital information associated with the location of the area of interest of the object and to transmit the electronic or digital information or the location of the area of interest to a remote computing device.

[0015] In another aspect, a method for obtaining information about an area of interest of an object is disclosed. The method may include determining one or more positions of a portable electronic device relative to locations of one or more items of the object, and determining a location of the area of interest of the object relative to the one or more positions of the portable electronic device. The method may also include identifying electronic or digital information associated with the location of the area of interest of the object, and transmitting the electronic or digital information or the location of the area of interest to a remote computing device.

[0016] In yet another aspect, a non-transitory computer-readable medium having stored thereon instructions is disclosed. The instructions, when executed by one or more processors, cause the one or more processors to perform a plurality of steps for obtaining information about an area of interest of an object. The steps may include determining one or more positions of a portable electronic device relative to locations of one or more items of the object, and determining a location of the area of interest of the object relative to the one or more positions of the portable electronic device. The steps may also include identifying electronic or digital information associated with the location of the area of interest of the object, and transmitting the electronic or digital information or the location of the area of interest to a remote computing device.

[0017]

[0017] The foregoing summary is for illustrative purposes only and is not intended to be in any way limiting. In addition to the exemplary aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the detailed description that follows.

[0018]

[0018] A more complete understanding of the embodiments of the present disclosure may be obtained by reference to the detailed description and claims in conjunction with the following drawings, in which like reference numbers indicate similar elements throughout the drawings. The numbers are provided to aid in understanding the disclosure and are not intended to limit the breadth, scope, scale, or applicability of the disclosure. The drawings are not necessarily drawn to scale. [Brief explanation of the drawings]

[0019] [Figure 1]

[0019] Photographs for an aircraft damage report detailing the location of the damage relative to other parts of the aircraft are shown. [Figure 2]

[0020] 1 is an illustration of an exemplary aircraft. [Figure 3]

[0021] 1 is a schematic diagram of a system for obtaining information related to an area of interest of a physical object. [Figure 4A]

[0022] FIG. 4 is a front view of a portable electronic device of the system of FIG. 3 showing an image and a digital representation of a portion of an aircraft cockpit. [Figure 4B]

[0023] FIG. 4 is a rear view of the portable electronic device of FIG. 3. [Figure 5]

[0024] FIG. 4 is a schematic diagram of components of the portable electronic device of FIG. 3. [Figure 6]

[0025] FIG. 1 is a schematic diagram of a technique for determining the location of an area of interest of a physical object. [Figure 7A]

[0026] 7A-7E illustrate another technique for determining the location of an area of interest of a physical object. [Figure 7B] 7A-7E illustrate another technique for determining the location of an area of interest of a physical object. [Figure 7C] 7A-7E illustrate another technique for determining the location of an area of interest of a physical object. [Figure 7D] 7A-7E illustrate another technique for determining the location of an area of interest of a physical object. [Figure 7E] 7A-7E illustrate another technique for determining the location of an area of interest of a physical object. [Figure 8]

[0027] FIG. 1 is an example flow diagram illustrating a method for obtaining information related to an area of interest of a physical object. DETAILED DESCRIPTION OF THE INVENTION

[0020]

[0028] The drawings and the following description illustrate several specific, exemplary embodiments. Those skilled in the art will recognize that, even if not explicitly set forth herein, they may devise various configurations that embody the principles described herein and are within the scope of the claims that follow. Furthermore, any examples described herein are intended to aid in the understanding of the principles of the disclosure and are not intended to be limiting. Consequently, the disclosure is not limited to the specific embodiments or examples described below, but is limited by the claims and their equivalents.

[0021]

[0029] Certain embodiments are described herein with reference to the drawings. In the description, common features are given common reference numbers throughout the drawings. In some drawings, multiple instances of a particular type of feature are used. Although these features are physically and / or logically different, the same reference number is used for each, and different instances are distinguished by the addition of a letter to the reference number.

[0022]

[0030] As used herein, various terms are used for the purpose of describing particular embodiments only and are not intended to be limiting. For example, the singular forms "a / an" and "the" are intended to include the plural (unless the context clearly dictates otherwise). Furthermore, the terms "comprise," "comprises," and "comprising" may be used interchangeably with "include," "includes," or "including." Furthermore, the term "wherein" is used interchangeably with the term "where." As used herein, "exemplary" refers to an example, an implementation, and / or an aspect and should not be construed as limiting or as indicating a preferred or preferred embodiment. As used herein, ordinal terms (e.g., "first," "second," "third," etc.) modifying elements such as structures, components, operations, etc., do not in themselves indicate a priority or order of one element over another, but merely distinguish one element from another element having the same name (apart from its use as an ordinal term). As used herein, the term "set" refers to a grouping of one or more elements, and the term "plurality" refers to a plurality of elements.

[0023]

[0031] This application is directed to systems, methods, and apparatus for determining the location or position of an area of interest of a physical object (e.g., a vehicle, aircraft, system, etc.) within a real-world environment. Embodiments can use the location of the area of interest to obtain electronic or digital information (e.g., a 3D digital model, a virtual representation, a schematic, an engineering drawing, etc.) regarding the area of interest of the physical object. The area of interest may correspond to an anomaly, defect, damage, malfunction, component, part, object, item, and / or condition of the physical object.

[0024]

[0032] Embodiments may be used to inspect, troubleshoot, and / or repair physical objects. For example, embodiments may be used by support personnel to perform maintenance on physical objects. Embodiments may quickly and accurately determine the physical location or position of an area of interest relative to a physical object. For example, embodiments may determine the coordinates of an area of interest (e.g., an anomaly, defect, damage, etc.) of a physical object in the object's coordinate system (e.g., an object-centered coordinate system). In some examples, embodiments may calculate the physical location of an object of interest on or within a vehicle, such as an aircraft. As a result, the likelihood of human error in accurately determining the location of an area of interest of a physical object may be reduced.

[0025]

[0033] Embodiments may convert the physical location of an area of interest of a physical object to a position within the object's coordinate system (e.g., a vehicle-centered coordinate system). Embodiments may obtain electronic or digital information about the area of interest of a physical object based on the location of the area of interest within the object's coordinate system. For example, embodiments may use the location of the area of interest of the physical object within the object's coordinate system to identify and search electronic information and records related to the area of interest of the physical object (e.g., a vehicle, aircraft, etc.). The electronic information is organized or categorized using the object's coordinate system (e.g., a vehicle coordinate system) and may include CAD / CAM electronic records, 3D digital models, virtual representations, schematics, engineering drawings, technical documentation, maintenance reports, trouble reports, service bulletins, system reports, structural analysis, flight logs, and other information related to the area of interest of the physical object.

[0026]

[0034] Embodiments may be configured to search electronic or digital records and reports (e.g., damage report records, service requests, etc.) in one or more databases for information about an area of interest for a physical object. For example, embodiments may search electronic reports and records for problems related to the location of damage to a particular type of vehicle or aircraft. Embodiments may identify and display relevant electronic or digital records to support or maintenance personnel. Embodiments may also store or record any current damage reports for a physical object as new records or reports in a database. For example, embodiments may create and store a service request or problem report in a database (e.g., an airline database) based on the physical location of the damage, including a description and type of damage, the type of object or vehicle, a photograph or sketch of the damage, etc.

[0027]

[0035] Embodiments may transmit retrieved electronic and digital reports and records related to an area of interest of a physical object to devices of onsite and offsite analysts or specialists. For example, embodiments may transmit graphical and virtual representations of an area of interest of a physical object (e.g., a virtual representation of an aircraft cockpit) to onsite and / or offsite analysts. The electronic and digital information may assist the onsite and / or offsite analysts in analyzing the area of interest of the physical object. For example, embodiments may enable the onsite and offsite analysts to view electronic or digital information related to the area of interest of the physical object. As a result, the onsite and offsite analysts may not need to visit and inspect the physical object and may remotely determine repair of any defects (e.g., anomalies, damage, conditions, etc.) associated with the area of interest.

[0028]

[0036] By enabling efficient and reliable determination of areas of interest of physical objects (e.g., vehicles, systems, etc.) within a real-world environment, embodiments can improve the process for troubleshooting and repairing physical objects. For example, embodiments may reduce the time required to identify and repair defects (e.g., defects, damage, cracks, conditions, failures, etc.) in a physical object and reduce the time required to obtain electronic and digital information about the defect. The embodiments may also automate and speed the process for repairing physical objects (e.g., vehicles), improve the accuracy of locating defects, reduce errors in determining defect locations, and create new records useful for future repairs. Furthermore, embodiments may advantageously increase the reliability, safety, maintainability, and availability of physical objects (e.g., vehicles), resulting in improved performance and operational capabilities of the physical object. Furthermore, in the aviation industry, embodiments may reduce the number of flights delayed or canceled due to maintenance or repair issues.

[0029]

[0037] As shown in FIG. 2 , an exemplary embodiment of an aircraft 100 is illustrated. In that case, multiple embodiments of systems and methods for acquiring electronic or digital information (e.g., 3D digital models, schematics, drawings, etc.) related to an area of interest of a physical object, such as the aircraft 100, may be implemented. As shown in FIG. 2 , the aircraft 100 includes a fuselage 102 having a left side 104, a right side 106, a nose tip 108, and a tail section 110. A first main wing 112 is coupled to the left side 104 of the fuselage 102. A second main wing 114 is coupled to the right side 106 of the fuselage 102. In the illustrated example, the aircraft 100 includes a door 116 located on the left side 104 of the fuselage 102. Passengers and / or crew may enter (e.g., board) and / or exit (e.g., disembark) the aircraft 100 through the door 116. The aircraft 100 of FIG. 2 is merely an example, and thus, the embodiments disclosed herein may be used with other aircraft or vehicles without departing from the scope of the present disclosure.

[0030]

[0038] The locations or positions of structures and systems of aircraft 100 may be specified relative to local coordinates of aircraft 100. The coordinate system of aircraft 100 is shown in Figure 2, where the x-axis indicates the longitudinal direction (e.g., from the front to the back of aircraft 100), the y-axis indicates the lateral direction (e.g., left and right of the center of aircraft 100), and the z-coordinate indicates the vertical direction (e.g., from the bottom to the top of aircraft 100). The x, y, and z-coordinates relative to aircraft 100 may also be referred to as stations (or fuselage stations), butt lines, and water lines, respectively.

[0031]

[0039] 3 illustrates a system 200 for obtaining electronic or digital information about an area of interest 202 of a physical object 204 in a real-world environment 206, according to an illustrative embodiment. As shown in FIG. 3, system 200 includes a portable electronic device 208, a communication network 210, a database 212, and communication devices 214 and 216. As shown, portable electronic device 208 is in communication with communication network 210. Communication network 210 of system 200 can be used to provide communication links between various devices and computers connected together in system 200. Communication network 210 may include connections, such as wired, wireless communication links, or fiber optic cables.

[0032]

[0040] The portable electronic device 208 of the system 200 may be configured to display a live view of the physical object 204 (or a portion thereof) within the real-world environment 206, thereby allowing a user to view a representation of the physical object 204 in real-world environment 206. The portable electronic device 208 may also be configured to determine the position or location of the area of interest 202 of the physical object 204 within the real-world environment 206, such as the location of an anomaly (e.g., a defect, a fault, a condition, damage, etc.) in the aircraft 100 of FIG. 2. Additionally, the portable electronic device 208 may be configured to obtain electronic or digital information (e.g., a 3D digital model, a virtual representation, technical documentation, schematic diagrams, maintenance reports, system data, photographic records, silhouette images, etc.) related to the area of interest 202 of the physical object 204.

[0033]

[0041] Physics objects 204 of system 200 may include vehicles, buildings, structures, systems, subsystems, power plants, ships, spacecraft, vehicle surfaces or skins, components, parts, and / or any other suitable physical objects or items in real-world environment 206. Real-world environment 206 may be any type of environment in the physical world, such as a workspace. In the illustrated example, real-world environment 206 may be in physics object 204, such as the fuselage of aircraft 100 in FIG. 2. In other examples, real-world environment 206 may be, without limitation, a maintenance environment, a manufacturing environment, a production environment, a design environment, an installation environment, and / or any other suitable environment.

[0034]

[0042] 3, communication devices 214 and 216 of system 200 may be connected to communication network 210. Communication devices 214 and 216 may be wireless or computing devices operated by various personnel, such as, for example, maintenance personnel, mechanics, technicians, analysts, engineers, etc. Communication devices 214 and 216 may be located within physical object 204 (e.g., inside an aircraft) or may be located at a facility remote from physical object 204 (e.g., a maintenance facility).

[0035]

[0043] A database 212 of the system 200 may be connected to the communications network 210. The database 212 may store information related to the physical object 204. For example, the database 212 may include electronic or digital information about the architecture and structure of the physical object 204 (e.g., 3D digital models, virtual representations, schematics, specifications, designs, photographic records, silhouette images, etc.). The electronic or digital information may also include maintenance information about the physical object 204 (e.g., maintenance activities and messages, component installation and removal, etc.). Furthermore, the electronic or digital information may include information about the systems, subsystems, components, parts, etc. of the physical object 204. The electronic or digital information stored in the database 212 may be accessed and / or searched based on the object's coordinate system and / or its relationship to the position or location of the physical object 204 (e.g., coordinates). The electronic or digital information stored in the database 212 may also include information in a graph database format. A graph database stores data as nodes (entities) and the relationships between them. These relationships are represented as edges. Edges can have a variety of properties.

[0036]

[0044] The portable electronic device 208 of the system 200 may be configured to capture physical or sensor data about the physical object 204 within the real-world environment 206 and determine geocentric location data (e.g., GPS location information) and object-centered (e.g., aircraft-centered) relative position data of the portable electronic device 208. Based on the sensor data and / or location data, the portable electronic device 208 may be configured to determine the location of the portable electronic device 208 within the real-world environment and relative to the physical object 204. The portable electronic device 208 may also be configured to measure distance or range from the portable electronic device 208 to an area of interest 202 (e.g., an anomaly, defect, fault, component, part, damage, condition, etc.) of the physical object 204. In some examples, the portable electronic device 208 may use a measurement device to project a laser or light beam 218 onto the physical object 204 to illuminate the area of interest 202 of the physical object 204.

[0037]

[0045] Once the portable electronic device 208 determines the distance to the area of interest 202 of the physical object 204, the portable electronic device 208 can determine a location or position of the area of interest 202 of the physical object 204 within a physical coordinate system. For example, the portable electronic device 208 can determine the coordinates of the area of interest 202 within the real-world environment or the physical coordinate system of the portable electronic device 208. The portable electronic device 208 can transform the location of the area of interest 202 within the physical coordinate system to a position within the coordinate system of the physical object (e.g., the aircraft 100 of FIG. 2 ), as described further below. For example, the portable electronic device 208 can translate or transform the location of the area of interest 202 of the physical object 204 within the physical coordinate system to a corresponding position within the coordinate system of the physical object (e.g., a virtual coordinate system of a digital model of the physical object 204).

[0038]

[0046] After the location of the area of interest 202 of the physical object 204 is determined within the object's coordinate system, the portable electronic device 208 can identify and retrieve electronic or digital information (e.g., digital models, virtual representations, schematics, maintenance reports, specifications, designs, installation drawings, digital twins, etc.) associated with or related to the area of interest 202 of the physical object 204. The portable electronic device 208 can retrieve the electronic or digital information from the database 212 of the system 200 or from the memory of the portable electronic device 208. To retrieve the electronic or digital information, the portable electronic device 208 can generate a query based on the location of the area of interest 202 of the physical object 204 within the object's coordinate system. The portable electronic device 208 can use the query to retrieve electronic and digital information from the database 212 and / or the memory of the portable electronic device 208. For example, the portable electronic device 208 can retrieve electronic and digital information that matches or relates to the area of interest 202 of the physical object 204. In some embodiments, the portable electronic device 208 may retrieve a 3D digital model of a physical object (e.g., a digital model of an aircraft) or a 3D model corresponding to an area of interest of the physical object (e.g., a digital model of a structure, system, or component of the aircraft).

[0039]

[0047] The portable electronic device 208 may display electronic or digital information (e.g., virtual representations, schematic diagrams, specifications, etc.) retrieved from memory or database 212 and / or transmit electronic or digital information to other communication devices, such as communication devices 214 and 216. For example, the portable electronic device 208 may display a virtual representation of the area of interest 202 of the physical object 204 and transmit the virtual representation to communication devices 214 and 216. The communication devices 214 and 216 may be configured to display electronic or digital information received from the portable electronic device 208. In some examples, the portable electronic device 208 may transmit the location of the area of interest 202 of the physical object 204 in the object's coordinate system to the communication devices 214 and 216. The communication devices 214 and 216 may use the location of the area of interest 202 to access electronic or digital information from the database 212 or local memory. For example, on-site and / or off-site analysts may use communication devices 214 and 216 to access or retrieve electronic or digital information from database 212 based on the position or location of area of interest 202 of physical object 204 within the object's coordinate system.

[0040]

[0048] As shown in FIGS. 4A and 4B , the portable electronic device 208 of the system 200 may be a compact device that can be held or carried by a user or maintenance personnel. The portable electronic device 208 may be a smartphone, a tablet computer, a laptop computer, or any other suitable device. As shown in FIG. 4A , the portable electronic device 208 may display an image of an area of interest of a physical object along with a graphic or virtual representation of the area of interest. The portable electronic device 208 may correlate and match the low-fidelity digital image (based on a model) with the high-fidelity digital image (based on a photographic database). For example, once the portable electronic device 208 determines the location of the area of interest of an object (in both the digital and physical worlds), the portable electronic device 208 may match the low-fidelity digital image (based on a model) with the high-fidelity digital image (based on a photographic database). As one example, a user or mechanic evaluating the wiring to ensure it is in the correct location can view the real or physical world (e.g., the interior of an aircraft) as well as digital photos from a database (e.g., the digital world) that show where the wiring should be. In some examples, the portable electronic device 208 may include a wearable device (e.g., a pair of augmented reality glasses). For example, the wearable device may provide a live view of the real-world environment by overlaying a representation of the real-world environment onto a transparent or translucent display that functions similarly to eyeglass lenses, allowing the user to view the real-world environment through the display.

[0041]

[0049] 5, a schematic diagram of the components of a portable electronic device 208 is shown. The portable electronic device 208 may include a communications unit 520, a sensor system 522, a storage device 524 (e.g., memory or database), a processing unit 526, a user interface 528, and a display device 530. In other embodiments, the portable electronic device 208 may include additional components, hardware, or functionality. A bus 532 may couple the communications unit 520, the sensor system 522, the storage device 524, the processing unit 526, the user interface 528, and the display device 530 together and enable communication therebetween. Although only one bus is depicted, the portable electronic device 208 may include multiple buses or other types of communication paths between any of its elements or components.

[0042]

[0050] The communication unit 520 of the portable electronic device 208 may be configured to connect to a communication network (e.g., communication network 210 of FIG. 3 ). The communication unit 520 can receive data / communications from and transmit data / communications to other devices, such as telecommunications and / or computing devices (e.g., communication devices 214 and 216 of FIG. 3 ) in the communication network. The communication unit 520 may enable the portable electronic device 208 to communicate with other devices via a wireless channel or a wired communication link. For example, the communication unit 520 may enable the portable electronic device 208 to wirelessly transmit the position or location of an area of interest of a physical object (e.g., physical object 204 of FIG. 3 ) to other devices. The communication unit 520 may also enable the portable electronic device 208 to wirelessly transmit electronic or digital information (e.g., virtual representations, maintenance reports, drawings, component information, etc.) retrieved from the storage device 524 to other devices.

[0043]

[0051] The communication unit 520 may include a wireless connection, a wired connection, a cable connection, an optical fiber connection, etc., and may communicate over a wide area network (WAN), a local area network (LAN), a cellular network, a peer-to-peer communication network, or any other suitable network. The communication unit 520 may also be operable to interact with a communication network using any type of communication protocol, such as, for example, Wi-Fi (e.g., 802.xx protocols), radio frequency (RF) protocols (e.g., 900 MHz, 1.4 GHz, and 5.6 GHz), Bluetooth, cellular communication protocols (e.g., 2G, 3G, 4G, 5G, etc.), or any other communication protocol.

[0044]

[0052] The sensor system 522 of the portable electronic device 208 may be configured to capture and collect physical or sensor data (e.g., image data, range, distance, location information, etc.) about one or more physical objects in the real-world environment. The sensor system 522 may include various types of sensors, such as a GPS sensor, an inertial measurement unit (IMU) or sensor, a measurement sensor, an image sensor (e.g., an imaging device for capturing images of physical objects in the real-world environment), or any other suitable sensor. The sensor system 522 may transmit the sensor data to the processing unit 526. As described further below, the sensor data may be processed by the processing unit 526 to determine the position and orientation of the portable electronic device 208 within the real-world environment and / or relative to physical objects in the real-world environment.

[0045]

[0053] The GPS sensor of the sensor system 522 may be configured to provide information (e.g., location coordinates) regarding the position or location of the portable electronic device 208 within a real-world environment. The IMU of the sensor system 522 may detect changes in the position and orientation of the portable electronic device 208 based on inertial acceleration. For example, the IMU may detect the pitch and yaw of the portable electronic device 208 while the portable electronic device 208 is stationary or moving. The IMU may include one or more accelerometers that generate acceleration sensor data. The one or more accelerometers can be used to measure static acceleration, such as the tilt of the portable electronic device 208 relative to gravity, as well as dynamic acceleration resulting from movement of the portable electronic device 208. The IMU may also include one or more gyroscopes configured to generate sensor data indicative of the current location or orientation of the portable electronic device 208.

[0046]

[0054] The imaging device or sensor of the sensor system 522 may be configured to capture image data of the real-world environment within its field of view. The imaging device may add geographic location data to metadata fields of the captured image data. The captured image data may be used to determine the location or position of the portable electronic device 208 relative to physical objects in the real-world environment. The captured image data may also be displayed on the display device 530 of the portable electronic device 208. In some embodiments, the imaging device may be a camera that includes three-dimensional capabilities. As shown in FIG. 4B , the imaging device may be located on the rear or back of the portable electronic device 208.

[0047]

[0055] The measurement sensors of the sensor system 522 may be configured to measure range and / or distance from the portable electronic device 208 to an area of interest of a physical object in the real-world environment. For example, the portable electronic device 208 may measure the distance from the portable electronic device 208 to an anomaly (e.g., damage, defect, etc.) of a physical object (e.g., an aircraft) in the real-world environment. In some examples, the measurement devices may be configured to project a laser or light beam onto the physical object to illuminate the area of interest of the physical object. The measurement sensors may include light-emitting devices, laser devices, optical devices, or any other suitable measurement sensors.

[0048]

[0056] 5, the storage device 524 of the portable electronic device 208 may store physical or sensory data captured by sensors of the sensor system 522. The storage device 524 may also store information related to one or more physical objects. For example, the storage device 524 may include electronic or digital information about the physical objects (e.g., digital models, virtual representations, schematics, specifications, designs, installation drawings, system information, etc.). Additionally, the storage device 524 may store digital models and / or virtual representations of physical objects. Digital models may include representations of structures, assemblies, systems, and subsystems of the physical objects. In some examples, the storage device 524 may store a digital model representing an aircraft fuselage. The storage device 524 may also store mapping or location data representing spatial or physical coordinate-based maps of physical objects in a real-world environment.

[0049]

[0057] The storage device 524 of the portable electronic device 208 may also store a plurality of program instructions to be executed or performed by the processing unit 526 of the portable electronic device 208. The storage device 524 may include physical, non-transitory, computer-readable memory that temporarily or permanently stores data for use by the processing unit 526. The memory may include one or more volatile and / or non-volatile memory devices, such as random access memory (RAM), static random access memory (SRAM), dynamic RAM (DRAM), read-only memory (ROM), flash memory, or any other suitable medium or memory that may be used to store desired information (e.g., system information, virtual models, mapping information, etc.).

[0050]

[0058] The processing unit 526 of the portable electronic device 208 may communicate with various components of the portable electronic device 208. The processing unit 526 may include one or more processors. For example, the processing unit 526 may include one or more central processing units (CPUs), one or more graphical processing units (GPUs), one or more digital signal processors (DSPs), one or more peripheral interface controllers (PICs), or another type of microprocessor.

[0051]

[0059] Processing unit 526 may be configured to identify and select digital models of physical objects in the real-world environment. Processing unit 526 may retrieve the digital models of the physical objects from storage device 524 or a remote database, such as database 212 of FIG. 3. In some embodiments, processing unit 526 may select and / or retrieve a digital model representing an aircraft fuselage. In other embodiments, processing unit 526 may retrieve a digital model representing a building, a vehicle, an industrial facility, a power plant, a ship, a spacecraft, a submarine, or any other suitable object based on the physical object.

[0052]

[0060] The digital model may represent a 3D model or representation of a physical object, including the systems and structures of the physical object. In some embodiments, the digital model is a computer-aided design (CAD) model, and a coordinate system may be used to locate spatial or virtual content of the digital model of the physical object. The digital model may be based on the design, testing, manufacturing, installation, and / or operation phases of the physical object.

[0053]

[0061] Once the processing unit 526 has selected and retrieved the digital model of the physical object, the user can use the portable electronic device 208 to identify the position or location of the area of interest of the physical object within the real-world environment. As the user uses the portable electronic device 208, the processing unit 526 may receive and collect physical or sensor data associated with the physical object within the real-world environment 206 from the sensor system 522. For example, the processing unit 526 may capture image data about the physical object within the real-world environment. Based on the sensor data, the processing unit 526 may be configured to determine the location of the portable electronic device 208 within the real-world environment and the location or position of the physical object within the real-world environment. For example, the processing unit 526 can use GPS data to determine the location of the portable electronic device 208 within the real-world environment (e.g., earth-centered or aero-centered (with respect to the aircraft)) and can use image data to determine the position and orientation of the portable electronic device 208 within the real-world environment (e.g., earth-centered or aero-centered (with respect to the aircraft)). Additionally, the processing unit 526 may be configured to determine a distance of the portable electronic device 208 to physical objects in the real-world environment and / or items or markers on the physical objects. Additionally, the processing unit 526 may be configured to determine a location of the portable electronic device 208 relative to the physical objects and items or markers on the physical objects.

[0054]

[0062] The processing unit 526 may also be configured to map sensor data of the real-world environment into a physical coordinate system or frame of reference. For example, the processing unit 526 may generate mapping or position data representing a spatial or physical coordinate-based map of the real-world environment. The processing unit 526 may map the real-world environment to establish a relationship between the position or location of the portable electronic device 208 and the positions of physical objects within the real-world environment 206. Mapping the real-world environment thereby assigns physical objects specific position coordinates within the physical coordinate system. The physical coordinate system may be based on the position of the portable electronic device 208 within the real-world environment 206. In some examples, the physical coordinate system may be three-dimensional and include three mutually perpendicular axes.

[0055]

[0063] The processing unit 526 may also be configured to track the position and orientation of the portable electronic device 208 within the real-world environment. For example, the processing unit 526 may process sensor or physical data received from the sensor system 522 to determine the position and orientation of the portable electronic device 208 relative to physical objects in the real-world environment. As the portable electronic device 208 moves through the real-world environment (e.g., the fuselage of an aircraft), the processing unit 526 may be configured to track physical objects in the real-world environment 206 to determine the position and orientation of the portable electronic device 208 relative to the physical objects and / or items or markers of the physical objects in the real-world environment. For example, the processing unit 526 may track changes in the proximity and angle of the portable electronic device 208 relative to the physical objects or items of the physical objects in the real-world environment. In some embodiments, the items may include fiducial markers. Based on the perceived changes in the real-world environment surrounding the portable electronic device 208, the processing unit 526 may calculate the movement (e.g., translation and / or rotation) of the portable electronic device 208 and determine the current position and orientation of the portable electronic device 208 relative to physical objects in the real-world environment 206.

[0056]

[0064] The processing unit 526 may be configured to assign or spatially align physical objects in the real-world environment 206 to digital models or representations of the physical objects. The processing unit 526 may use a transformation or transfer function to assign physical objects defined in a physical coordinate system to digital or virtual content of the digital model of the physical object defined in a local or object coordinate system. Once the physical objects in the real-world environment are assigned to the digital models of the objects, the portable electronic device 208 may be configured to determine the location of an area of interest of the physical objects within the object coordinate system.

[0057]

[0065] 6 , the portable electronic device 208 may use triangulation techniques to determine the position or location of the area of interest 602 of the physical object 604 in the real-world environment 606. In other embodiments, the portable electronic device 208 may use trilateration techniques, multilateration techniques, or any other suitable technique to determine the location of the area of interest 602 of the physical object 604 in the real-world environment. As shown in FIG. 6 , the portable electronic device 208 may determine a first position (P1) of the portable electronic device at a first time. From the first position (P1), the portable electronic device 208 may determine a distance or length from the first position (P1) to the area of interest 602 of the physical object 604. At a second time, the portable electronic device 208 may determine a second position (P2) of the portable electronic device 208 and a distance or length from the second position (P2) to the area of interest 602 of the physical object 604.

[0058]

[0066] After the portable electronic device 208 determines the locations of the first and second positions, the portable electronic device 208 may calculate the distance (D) between the first position (P1) and the second position (P2). Based on these calculations, the portable electronic device 208 can calculate the location of the area of interest 602 of the physical object 604 based on the following formula: tan(angle 1)=d / x (1) tan(angle 2)=d / (Dx) (2) d=x*tan(angle 1) (3) d=(Dx)*tan(angle2) (4) x*tan(angle 1)=(Dx)*tan(angle 2) (5) x*tan(angle 1)=D*tan(angle 2)-x*tan(angle 2) (6) x*tan(angle 1)+x*tan(angle 2)=D*tan(angle 2) (7) x*[tan(angle 1)+tan(angle 2)]=D*tan(angle 2) (8) x=D*tan(angle 2) / [tan(angle 1)+tan(angle 2)] (9)

[0059]

[0067] The portable electronic device 208 can use several other techniques to determine the position or location of the area of interest 602 of a physical object 604 in a real-world environment 606. As shown in FIGS. 7A-7E, the portable electronic device may determine the position or location of the electronic device using a trilateration technique of the portable electronic device relative to three (3) known locations of the object. Initially, the portable electronic device may be geolocated at three (3) positions relative to three (3) known and / or fixed features or items (P1, P2, and P3) of the object, as shown in FIGS. 7A-7C. For example, in an aircraft, the portable electronic device may determine the location of the portable electronic device within the aircraft based on three known fixed items within the aircraft, such as an RFID chip, a QR code, and a door hinge at known reference points. The portable electronic device may determine a first location or position (PA) of the portable electronic device relative to the three known items (P1, P2, and P3) at a first time, as shown in FIG. 7A. The portable electronic device may also determine a second location (PB) of the portable electronic device relative to the three known items (P1, P2, and P3) at a second time, as shown in Figure 7B. Additionally, the portable electronic device may determine a third location (PC) of the portable electronic device relative to the three known items (P1, P2, and P3) at a third time, as shown in Figure 7C.

[0060]

[0068] After the portable electronic device has determined three locations of the portable electronic device relative to the three known items, the portable electronic device may determine a position or location of the area of interest of the physical object in the real world based on the first position (PA), the second position (PB), and the third position (PC), as shown in Figure 7D. The portable electronic device may calculate the distance from the first position (PA) to the area of interest (PX), calculate the distance from the second position (PB) to the area of interest (PX), and calculate the distance from the third position (PC) to the area of interest (PX). Based on these calculations, the portable electronic device 208 can calculate the location of the area of interest (PX) (e.g., the area of interest of the object) in three dimensions, as shown in Figures 7D-7E, based at least on the following equations:

[0061]

[0069] (D z -A z ) 2 +(D y -A y ) 2 -|AD| 2 =0 (10)

[0062]

[0070] (D z -B z ) 2 +(D y -B y ) 2 -|BD| 2 =0 (11)

[0063]

[0071] (D z -C z ) 2 +(D y -C y ) 2 -|CD| 2 =0 (12)

[0064]

[0072] Dz=(((-Ax) 2 +(Bx) 2 -(A y ) 2 +(B y )2 -|BD| 2 +|AD| 2 ) / 2)-(D y (B y -A y )) / (B x -A x ) (13) Here, A(A z , A y ) is the location of PA, and B(B z , B y ) is the location of PB, and C(C z , C y ) is the PC location, D(D z , D y ) is the location of PX, and the distances |AD|, |BD|, and |CD| are known.

[0065]

[0073] 6, the portable electronic device 208 may utilize any suitable coordinate system for determining the position of the portable electronic device 208 within a real-world environment and / or the position of the area of interest 602 of the physical object 604. For example, the portable electronic device 208 may use a geodetic coordinate system, in which locations on the Earth are specified by longitude (e.g., degrees east or west of the prime meridian) and latitude (e.g., degrees north or south of the equator), and altitude is specified by height above mean sea level (MSL). This coordinate system provides spherical coordinates (e.g., approximating the shape of the Earth).

[0066]

[0074] The portable electronic device 208 may also use an East, North, Up (ENU) local coordinate system, in which location is specified in units east and north of the coordinate system's origin (e.g., located on the Earth by a geodetic coordinate system), and altitude is specified by height above mean sea level (MSL). Additionally, the portable electronic device 208 may use a North, East, Down (NED) local coordinate system, which is similar to the ENU system, where the x component is the same as the ENU East component, the y component is the same as the ENU Up component, and the z component is minus the ENU North component. The NED component system is similar to the Earth-Centered Earth-Fixed (ECEF) coordinate system. The relationship between the NED and ECEF coordinate systems is given by the following equation: P NED =R T (P ECEF -P Ref ) where P NED is the 3D position in the NED system, and P ECEF is the corresponding ECEF position, and P Ref is the reference ECEF position (origin of the local tangent plane); R is a rotation matrix with columns for the north, east, and down axes; P REF can be defined from the latitude phi and longitude lambda corresponding to TIFF2025114472000002.tif23170

[0067]

[0075] In some implementations, the portable electronic device 208 can use GPS location information to determine the location (e.g., earth-centered or object / aircraft-centered (relative to the object / aircraft)) of the area of interest 602 of a physical object 604 (e.g., an aircraft) within a real-world environment. For example, the portable electronic device 208 can determine the GPS location of the area of interest 602 of the physical object 604 (e.g., an aircraft) and the GPS location of a known fixed item or marker. The portable electronic device 208 can then calculate an object-centered location (e.g., aircraft-centered location) of the area of interest 602 relative to the known fixed item or marker of the object (e.g., the aircraft). The portable electronic device 208 can also use the GPS location information to determine the location (e.g., earth-centered or object-centered (relative to the object)) of the portable electronic device 208 within a real-world environment.

[0068]

[0076] 5 , once the processing unit 526 of the portable electronic device 208 determines the location of the area of interest of the physical object in the real-world environment defined in the physical coordinate system, the processing unit 526 may be configured to determine the location of the area of interest of the physical object in the physical object's coordinate system. The processing unit 526 may use a transformation or transfer function to transform or convert the coordinates of the location of the area of interest of the physical object in the real-world environment into coordinates of the physical object's coordinate system. In some examples, the coordinates of the physical object's coordinate system may be three-dimensional coordinates defined along three mutually perpendicular axes in the object's coordinate system.

[0069]

[0077] Once the coordinates of the area of interest of the physical object are determined within the object's coordinate system, the processing unit 526 can retrieve electronic or digital information about the area of interest of the physical object from the storage device 524 or a database. The processing unit 526 may be configured to search electronic or digital records and reports (e.g., damage report records, service requests, etc.) in one or more databases for information related to the area of interest of the physical object. For example, the processing unit 526 may search electronic reports and records for issues related to the location of damage to a particular type of vehicle or aircraft. The processing unit 526 may identify and display relevant electronic or digital records to support or maintenance personnel. For example, the processing unit 526 may be configured to display electronic or digital information (e.g., a graphic or virtual representation of the area of interest) on the display device 530 of the portable electronic device 208. Furthermore, the processing unit 526 may transmit the electronic or digital information to other communication or computing devices to enable remote personnel to analyze the area of interest of the physical object. Additionally, processing unit 526 may also store or record any current damage reports to the physical object as new records or reports in a database. For example, processing unit 526 may create and store a service request or problem report in a database (e.g., an airline database) based on the physical location of the damage, including a description and type of damage, the type of object or vehicle, a photo or sketch of the damage, etc.

[0070]

[0078] 5 , the user interface 528 of the portable electronic device 208 may enable a user or maintenance personnel to interact with the portable electronic device 208. The user interface 528 may include an interactive touchscreen. In other embodiments, the user interface 528 may include a keyboard, a mouse, a microphone, or any other suitable input / output device. The user interface 528 of the portable electronic device 208 may be configured to receive input and / or user selections to enable the portable electronic device 208 to select one or more digital models representing physical objects. For example, a user may input into the user interface 528 information related to a physical object (e.g., an aircraft) in a real-world environment, a system of the physical object (e.g., the aircraft's electrical wiring system), components of the system, and / or other information related to the physical object. After the information is received, the portable electronic device 208 may display graphical or virtual content related to an area of interest of the physical object on the user interface 528 or on the display device 530, as described further below.

[0071]

[0079] The display device 530 of the portable electronic device 208 may be configured to present visual, auditory, and / or tactile information to a user or maintenance personnel. The display device 530 may include a screen or any other suitable type of display. In other embodiments, the display device 530 may be integrated into a transparent or translucent visor of an optical see-through augmented reality (AR) imaging device and viewed by a user or technician wearing the AR imaging device.

[0072]

[0080] The display device 530 of the portable electronic device 208 may be configured to display a live view of a real-world environment, allowing a user to view representations of physical objects within the real-world environment in real time. In some examples, the display device 530 may display a representation of an aircraft fuselage (or a portion thereof). The display device 530 may also display augmented reality (AR) content, such as graphical or virtual content. The display device 530 of the portable electronic device 208 may also be configured to display a graphical or virtual representation of an area of interest of a physical object and / or an image of the area of interest, as shown in FIG. 4A .

[0073]

[0081] 8 illustrates a flow diagram of a method 800 for obtaining electronic or digital information (e.g., a 3D digital model, a virtual representation, a schematic, a drawing, etc.) related to an area of interest of a physical object (e.g., an aircraft) in a real-world environment, according to an example embodiment. The method may be performed or implemented, in whole or in part, by a portable electronic device or an AR device, such as the portable electronic device 208 of FIG. 3 .

[0074]

[0082] The portable electronic device may identify and capture data of physical objects in a real-world environment. The physical objects may be vehicles, aircraft, buildings, industrial facilities, power plants, ships, spacecraft, submarines, or any other physical objects. The portable electronic device may be configured to retrieve digital models of the physical objects from a memory or a database. The digital models may be computer-aided design (CAD) models. The digital models of the physical objects may include structures and systems of the physical objects. In some examples, the digital models may represent one or more systems of an aircraft. For example, the systems may include a hydraulic system, an air trim system, an environmental system, a flight management system, a navigation system, a communication system, a sensor system, a propulsion system, a flight control system, an electrical system, a gas pressure system, a guidance system, a radar system, an air conditioning system, a blower system, an air intake system, and / or any other electronic, mechanical, and / or hardware systems of the aircraft.

[0075]

[0083] The digital model may be stored in a storage device (e.g., a database) of the portable electronic device. In some embodiments, the digital model may be generated and stored in a remote or separate database (e.g., database 212 of FIG. 3 ), and the portable electronic device may access and / or retrieve the digital model from the remote or separate database. The digital model may be constructed or generated during the design, testing, manufacturing, installation, and / or operation phases of the physical object. The portable electronic device may be configured to align or spatially align the digital model of the physical object with a corresponding physical object in a real-world environment. For example, the portable electronic device may use a transformation or transfer function to align the digital model, defined as the object's coordinate system, with the physical object in the real-world environment, defined in a physical coordinate system. For example, the portable electronic device may transform or convert the coordinates of the physical object in the real-world environment to position coordinates in the object's coordinate system.

[0076]

[0084] At block 802, method 800 includes determining one or more positions of the portable electronic device relative to the location of one or more items of the object. Once the digital model of the object is aligned with the physical object in the real-world environment, the portable electronic device may receive and / or capture physical or sensor data associated with the physical object in the real-world environment. For example, the portable electronic device may receive sensor data (e.g., GPS data, image data, etc.) about the physical object in the real-world environment. In one example shown in FIG. 3, the portable electronic device may capture image data about the fuselage of an aircraft.

[0077]

[0085] The portable electronic device may determine a position of the portable electronic device in a real-world environment and / or relative to physical objects based on the sensor data. The portable electronic device may be configured to map the sensor data of the physical objects into a physical coordinate system or frame of reference. For example, the portable electronic device may generate mapping or position data representing a spatial or physical coordinate-based map of the physical objects in the real-world environment. The portable electronic device may map the physical environment to establish a relationship between the position of the portable electronic device and the positions of the physical objects in the real-world environment. Thereby, upon mapping the physical objects into the real-world environment, the physical objects may be assigned specific position coordinates within the physical coordinate system. The physical coordinate system may be based on the position of the portable electronic device in the real-world environment.

[0078]

[0086] The portable electronic device may also be configured to track the position and orientation of the portable electronic device relative to physical objects in the real-world environment. The portable electronic device may process sensor or physical data to determine the position and orientation of the portable electronic device in the real-world environment and / or relative to the physical objects. As the portable electronic device moves through the real-world environment (e.g., the fuselage of an aircraft), the portable electronic device may track physical objects in the real-world environment to determine the position and orientation of the portable electronic device in the physical environment relative to the physical objects. For example, the portable electronic device may track changes in the proximity and angle of the portable electronic device relative to physical objects or items or markers of physical objects in the real-world environment. In some embodiments, the physical objects may include fiducial markers. Based on perceived changes in the real-world environment surrounding the portable electronic device, the portable electronic device may calculate a movement (e.g., translation and / or rotation) of the portable electronic device and determine the current position and orientation of the portable electronic device relative to the physical objects.

[0079]

[0087] After the portable electronic device determines its location relative to the physical object, the portable electronic device may use a measurement device to measure the distance or range from the portable electronic device to an area of interest (e.g., an anomaly, defect, fault, component, part, and / or condition) of the physical object. The portable electronic device may measure the distance from the portable electronic device to the area of interest of the physical object at one or more times or positions.

[0080]

[0088] At block 804, method 800 includes determining a location of the area of interest of the object relative to one or more positions of the portable electronic device. After the portable electronic device determines the range / distance to the area of interest of the physical object, the portable electronic device may determine a position of the area of interest of the physical object in the real-world environment defined in a physical coordinate system. The portable electronic device may use a transformation or transfer function to transform or convert coordinates of the position of the area of interest of the physical object in the real-world environment into coordinates of the object's coordinate system. The coordinates of the object's coordinate system may be three-dimensional coordinates defined along three mutually perpendicular axes in the object's coordinate system.

[0081]

[0089] At block 806, the method 800 includes identifying electronic or digital information associated with the location of the area of interest of the object. After the location of the area of interest of the physical object is determined within the object's coordinate system, the portable electronic device can identify and retrieve electronic or digital information (e.g., digital models, virtual representations, schematics, maintenance reports, specifications, designs, installation drawings, digital twins, etc.) associated with or related to the area of interest of the physical object. The portable electronic device 208 may retrieve the electronic or digital information from a remote database and / or memory of the portable electronic device. In some examples, the portable electronic device 208 may retrieve a 3D digital model of the physical object (e.g., a digital model of an aircraft) or a 3D model corresponding to the area of interest of the physical object (e.g., a digital model of a structure, system, or component of the aircraft).

[0082]

[0090] At block 808, the method 800 includes transmitting the electronic or digital information or the location of the area of interest to a remote computing device. Once the electronic or digital information is retrieved from memory or a database, the portable electronic device may transmit the electronic or digital information to another communication or computing device. In some embodiments, the portable electronic device may transmit the location of the area of interest of the physical object in the object's coordinate system to the other communication device. The communication device can use the location of the area of interest 202 to access the electronic or digital information from a database or local memory.

[0083]

[0091] By utilizing the portable electronic devices of the present application, maintenance personnel and technicians can efficiently troubleshoot complex physical objects, such as vehicles, machines, or structures. Furthermore, the time required to troubleshoot anomalies in physical objects in real-world environments can be substantially reduced. In the aviation industry, the systems disclosed herein can reduce the number of flights that are delayed or canceled due to repairs and maintenance.

[0084]

[0092] Although the disclosed system has been generally described and illustrated in connection with aircraft, the system may be used to locate defects or faults in any physical object, such as complex systems produced by the automotive, marine, electronics, power generation, and computer industries. Accordingly, the foregoing description of the use of the disclosed system and method in aircraft is for purposes of explanation and example, and not limitation, as the above-described system and method are equally applicable in many different industries.

[0085]

[0093] The description of the various preferred configurations has been presented for purposes of illustration and description and is not intended to be exhaustive or to limit the embodiments to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art. Moreover, various preferred embodiments demonstrate distinct advantages over other preferred embodiments. The selected embodiment or embodiments have been chosen and described in order to best explain the principles and practical application of those embodiments and to enable others skilled in the art to understand that the disclosure of the various embodiments, together with their various modifications, is well suited to the particular use contemplated.

[0086]

[0094] The embodiments described herein may be implemented as hardware, software, or a combination of hardware and software. For example, the embodiments may be implemented in a centralized manner in at least one computer system, or in a distributed manner where various elements are distributed across interconnected computer systems. Any type of computer system or other apparatus adapted to perform the methods described herein may be employed. The embodiments described herein may also be embedded in a computer program product, which contains all features enabling the implementation of the steps described herein and which, when loaded into a computer system, is capable of performing these operations.

[0087]

[0095] The flow diagrams and block diagrams described herein illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various exemplary embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing one or more specific logical functions. It should be further noted that in some alternative implementations, the functions noted in the blocks may occur in an order other than that noted in the figures. For example, depending on the functionality involved, the functions of two blocks shown in succession may be executed substantially concurrently, or the functions of those blocks may sometimes be executed in the reverse order.

[0088]

[0096] Furthermore, in this specification, an instance where one element is "coupled" to another element may include direct and indirect coupling. A direct coupling may be defined as one element being connected to another element and having some contact with the other element. An indirect coupling may be defined as a coupling between two elements that are not in direct contact with each other but have one or more additional elements between the coupled elements. Furthermore, in this specification, fixing one element to another element may include direct fixing and indirect fixing. In addition, as used herein, "adjacent" does not necessarily mean contact. For example, one element may be adjacent to another element without touching it.

[0089]

[0097] As used herein, a system, device, structure, article, element, component, or hardware that is "configured to" perform a specified function does not mean that it is, in fact, capable of performing the specified function without any modification and may merely perform the specified function after further modification. In other words, a system, device, structure, article, element, component, or hardware that is "configured to" perform a specified function is specifically selected, created, implemented, utilized, programmed, and / or designed for the purpose of performing the specified function. As used herein, the phrase "configured to" refers to the existing characteristics of a system, device, structure, article, element, component, or hardware that enable the system, device, structure, article, element, component, or hardware to perform a particular function without further modification. In this disclosure, a system, device, structure, article, element, component, or hardware described as being "configured to" perform a particular function may additionally or alternatively be described as being "adapted to" and / or "operative to" perform that function.

[0090]

[0098] As used herein, the terms "substantially," "similarity," and "about" mean that the stated characteristic, parameter, or value need not be exactly achieved, but that deviations or variations, including tolerances, measurement errors, measurement accuracy limits, and other factors known to those skilled in the art, may occur to the extent that they do not negate the effect that the characteristic is intended to achieve.

[0091]

[0099] Unless otherwise indicated, the terms "first," "second," etc. are used herein merely as designators and are not intended to impose any sequential, positional, or hierarchical requirements on the items they refer to. Furthermore, a reference to, e.g., a "second" item does not require or exclude the presence of, e.g., a "first" or lower numbered item and / or, e.g., a "third" or higher numbered item.

[0092]

[0100] While the device has been described with reference to specific embodiments, those skilled in the art will recognize that various modifications may be made and equivalents may be substituted without departing from the scope of the claims. Accordingly, the device is not limited to the specific embodiments disclosed, but rather the disclosed device is intended to include all embodiments that fall within the scope of the appended claims.

Claims

1. A portable electronic device (208) for obtaining information about an area of interest (202, 602) of an object, comprising: a display (530); Imaging devices, a measuring device configured to measure a distance or angle to a point in space; a processor (526) comprising: determining one or more positions of the portable electronic device relative to the location of one or more items of the object; determining a location of the area of interest of the object relative to the one or more positions of the portable electronic device; Identifying electronic or digital information associated with the location of the area of interest of the object; and transmitting the electronic or digital information or the location of the area of interest to a remote computing device.

2. The portable electronic device of claim 1 , wherein the location of the area of interest of the object comprises three-dimensional coordinates.

3. 2. The portable electronic device of claim 1, wherein the one or more items of the object are located at fixed, known locations, and the area of interest of the object corresponds to an anomaly, damage, crack, condition, defect, item, part, component, article, feature, point, or portion of the object, and the object comprises a vehicle, aircraft (100), ship, building, spacecraft, or submarine.

4. The portable electronic device of claim 1 , wherein each of the one or more items comprises a marker, an article, a feature, or an object, and the one or more positions of the portable electronic device are defined within a physical coordinate system.

5. The portable electronic device of claim 1 , wherein the processor is further configured to align or correlate the object to a digital model of the object.

6. The portable electronic device of claim 1 , wherein the location of the area of interest of the object is defined in a physical coordinate system.

7. The portable electronic device of claim 6 , wherein the location of the area of interest of the object corresponds to a three-dimensional location of the area of interest of the object within a coordinate system of the object.

8. The portable electronic device of claim 6 , wherein the processor is further configured to transform the location of the area of interest of the object into coordinates in a coordinate system of the object using a transfer function.

9. The portable electronic device of claim 6 , wherein the location of the area of interest of the object in the physical coordinate system is determined using triangulation, trilateration, or multilateration techniques.

10. Determining the one or more locations of the portable electronic device includes: determining a first position of the portable electronic device relative to the location of the one or more items of the object; determining a second position of the portable electronic device relative to the location of the one or more items of the object; and The portable electronic device of claim 1 , further comprising determining a third position of the portable electronic device relative to the location of the one or more items of the object.

11. The processor: determining a distance between the first position of the portable electronic device (208) and the area of interest of the object; determining a distance between the second position of the portable electronic device (208) and the area of interest of the object; and The portable electronic device of claim 10, further configured to: determine a distance between the third position of the portable electronic device (208) and the area of interest of the object.

12. The portable electronic device of claim 1 , wherein the measurement device comprises a light emitting device, a laser device, or an optical device.

13. The portable electronic device of claim 1 , wherein the display is configured to display a visual or digital representation of the area of interest of the object based on the electronic or digital information.

14. The portable electronic device of claim 1 , wherein the electronic or digital information comprises digital images, technical documentation, schematic diagrams, maintenance reports, system data, photographic records, silhouette images, or combinations thereof.

15. The portable electronic device of claim 1 , wherein the processor is further configured to retrieve the electronic or digital information from a memory or a database.

16. The portable electronic device of claim 1 , wherein the portable electronic device comprises an augmented reality device having a head-mounted device configured to be worn by a user.

17. A method (800) for obtaining information about an area of interest (202, 602) of an object, comprising: determining, by one or more processors, one or more positions of the portable electronic device relative to the location of one or more items of the object; determining, by the one or more processors, a location of the area of interest of the object relative to the one or more positions of the portable electronic device; identifying, by the one or more processors, electronic or digital information associated with the location of the area of interest of the object; and transmitting the electronic or digital information or the location of the area of interest to a remote computing device.

18. The location of the area of interest of the object is defined in a physical coordinate system, and the method comprises: The method of claim 17 , further comprising using a transfer function to transform the location of the area of interest of the object into coordinates in a coordinate system of the object.

19. 20. The method of claim 18, wherein the object comprises a vehicle, an aircraft (100), a watercraft, a building, a spacecraft, or a submarine, the one or more items of the object are located at fixed, known locations, and the location of the area of interest of the object within the physical coordinate system is determined using triangulation, trilateration, or multilateration techniques.

20. 1. A non-transitory computer-readable medium having stored thereon a plurality of instructions that, when executed by one or more processors, cause the one or more processors to perform a plurality of steps for obtaining information about an area of interest (202, 602) of an object, the plurality of steps comprising: determining one or more positions of a portable electronic device (208) relative to the location of one or more items of the object; determining a location of the area of interest of the object relative to the one or more positions of the portable electronic device; Identifying electronic or digital information associated with the location of the area of interest of the object; and a non-transitory computer readable medium, including transmitting the electronic or digital information or the location of the area of interest to a remote computing device.

Images