Augmented reality system for visualizing object's nonconforming data

Abstract

The present application discloses an augmented reality system for visualizing nonconforming data of an object. Methods, apparatus, and systems for visualizing nonconforming data of a physical object. An augmented reality application in a portable computing device plots points in a defined coordinate cube, wherein the points correspond to nonconforming locations on a physical object. The augmented reality application determines a subset of the plotted points, the subset of points corresponding to an area of ​​a physical object visible in an image of the area of ​​the physical object acquired by the portable computing device at the orientation of the portable computing device, wherein the subset of points excludes nonconforming locations that are obscured from view by physical object structures of the physical object in the image. The augmented reality application displays nonconforming data for a subset of points visible in the image in association with a subset of nonconforming locations of the physical object in the image displayed on a display system in the portable computing device.

Description

Technical Field

[0001] The present disclosure relates generally to manufacturing aircraft, and more particularly to methods, apparatus, and systems for visualizing nonconformance data of an aircraft during a manufacturing phase. Background Art

[0002] In manufacturing aircraft, many inspections are performed during different stages of manufacturing. Inspections can be performed by automated equipment, human operators, or some combination thereof. For example, a human operator can visually inspect an aircraft or an aircraft structure for nonconformities. An aircraft structure can be, for example, a fuselage section, an engine casing, a wing, or some other structure of an aircraft. A nonconformity can be, for example, a crack, a scratch, a delamination, an incorrect size, an incorrect color, or some other incorrect feature.

[0003] Further, the human operator may also touch the problematic area of ​​the aircraft structure to verify the physical condition of the aircraft structure. The human operator may also use tools (such as eddy current testers, ultrasonic machines or other tools) to inspect the aircraft structure.

[0004] When a location with a nonconformity is found on the aircraft structure, the location is physically marked with a physical marker, such as a strip of tape or a quick response (QR) code. In either case, a physical marker is placed on the aircraft structure at the nonconformity location.

[0005] Further, the human operator may also annotate the location description of the area that needs rework. The physical markings and annotations about nonconformities are recorded in a three-ring binder or notebook. For example, the location and identifier for the physical annotations may be recorded in a three-ring binder.

[0006] This information is also entered into the database so that a work order can be created to resolve the nonconformity. A human operator or another human operator can later use the work order to find the location of the physical tag for the nonconformity and perform the work required to resolve the nonconformity.

[0007] This type of inspection process is time consuming and inefficient. Therefore, it would be desirable to have methods and apparatus that take into account at least some of the problems described above as well as other possible problems. For example, it would be desirable to have methods and apparatus that overcome the technical problem of effectively identifying and resolving non-conformance of an aircraft. Summary of the invention

[0008] An embodiment of the present disclosure provides an augmented reality system for visualizing non-conforming data of a physical object, wherein the augmented reality system includes a portable computing device, a scanning application, a modeling application, a positioning application, and an augmented reality application. The portable computing device has a sensor system and a display system. The portable computing device is configured to generate scanning data describing points in the space of the physical object during the manufacturing stage. When executed, the scanning application uses the scanning data to generate a point cloud representation of the surface contour of an area of ​​the physical object. When executed, the modeling application uses the point cloud representation of the area of ​​the physical object in a defined coordinate cube to generate a model of the area of ​​the physical object. When executed, the positioning application locates the portable computing device based on a position target using the model of the area of ​​the physical object in the defined coordinate cube, and identifies the orientation of the portable computing device relative to the physical object. When executed, the augmented reality application maps points in the defined coordinate cube corresponding to non-conforming locations on the physical object; determines a subset of the points mapped in the defined coordinate cube, the subset of points being visible in an image of the area of ​​the physical object captured by the portable computing device, wherein the subset excludes mapped points determined to be outside the structure of the physical object when viewed at the orientation of the portable computing device; and displays the non-conforming data for the subset of the points visible in the image in association with the subset of the non-conforming locations of the physical object in the image displayed on the display system in the portable computing device.

[0009] Another illustrative embodiment provides an augmented reality system for visualizing nonconforming data of a physical object, the augmented reality system comprising a portable computing device and an augmented reality application. The portable computing device is configured to capture an image of a physical object. When executed by the portable computing device, the augmented reality application plots points in a defined coordinate cube, wherein the points correspond to nonconforming locations on the physical object; determines a subset of the points plotted in the defined coordinate cube, the subset of the points corresponding to areas of the physical object visible in an image captured by the portable computing device at the orientation of the portable computing device, wherein the subset of the points excludes nonconforming locations that are obscured from view by structures of the physical object in the image; and displays the nonconforming data of the subset of the points visible in the image in association with the subset of the nonconforming locations of the physical object in the image displayed on the display system in the portable computing device.

[0010] Yet another illustrative embodiment provides a method for visualizing nonconforming data for a physical object. An augmented reality application in a portable computing device plots points in a defined coordinate cube, the points corresponding to nonconforming locations on the physical object. The augmented reality application in the portable computing device determines a subset of the points plotted in the defined coordinate cube, the subset of points corresponding to an area of ​​the physical object visible in an image of the area of ​​the physical object acquired by the portable computing device, wherein the subset excludes plotted points determined to be outside of the structure of the physical object when viewed at the orientation of the portable computing device, wherein the subset of points excludes nonconforming locations that are obscured from view by the structure of the physical object in the image. The augmented reality application in the portable computing device displays the nonconforming data for the subset of the points visible in the image in association with the subset of the nonconforming locations of the physical object in the image displayed on the display system in the portable computing device.

[0011] The features and functions can be achieved independently in various embodiments of the present disclosure or may be combined in yet other embodiments, further details of which can be seen with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The novel features believed characteristic of the illustrative embodiments are set forth in the appended claims. However, the illustrative embodiments together with the preferred modes of use, further objects and features thereof will be best understood by reference to the following detailed description of illustrative examples of the disclosure when read in conjunction with the accompanying drawings, wherein:

[0013] Figure 1 is a diagram of an aircraft manufacturing environment according to an illustrative embodiment;

[0014] Figure 2 is an illustration of a block diagram of a visualization environment according to an illustrative embodiment;

[0015] Figure 3 is an illustration of a model of an aircraft structure in a defined coordinate cube according to an illustrative embodiment;

[0016] Figure 4 is an illustration of an image of a cockpit enhanced to indicate an unqualified location in accordance with an illustrative embodiment;

[0017] Figure 5 is an illustration of an image of a cockpit enhanced to indicate an unqualified location in accordance with an illustrative embodiment;

[0018] Figure 6 is an illustration of a panoramic view of an aircraft according to an illustrative embodiment;

[0019] Figure 7 is an illustration of a flow chart of a process for visualizing nonconforming data for a physical object according to an illustrative embodiment;

[0020] Figure 8 is an illustration of a flow chart of a process for determining a subset of points according to an illustrative embodiment;

[0021] Fig. 9 is an illustration of a flow chart of a process for identifying a location of a portable computing device according to an illustrative embodiment;

[0022] Fig.10 is an illustration of a flow chart of a process for filtering non-conforming data according to an illustrative embodiment;

[0023] Fig.11 is an illustration of a block diagram of a data processing system according to an illustrative embodiment;

[0024] Fig.12 is an illustration of a block diagram of an aircraft manufacturing and maintenance method according to an illustrative embodiment;

[0025] Fig.13 is an illustration of a block diagram of an aircraft in which the illustrative embodiments may be implemented; and

[0026] Fig.14 is an illustration of a block diagram of a product management system according to an illustrative embodiment. DETAILED DESCRIPTION

[0027] The illustrative embodiments recognize and take into account one or more different considerations. For example, the illustrative embodiments recognize and take into account that currently used techniques for inspecting vehicles (such as aircraft) take more time and effort than would be desirable using physical and paper annotations. The illustrative embodiments also recognize and take into account that currently used techniques for inspecting aircraft employ a series of disjointed processing systems without accurate metrics and measurements for collecting information about nonconformities.

[0028] The illustrative embodiments recognize and take into account that a process can be employed to provide an augmented reality experience for a human operator performing an inspection. In one illustrative example, the process identifies an orientation of a portable inspection device within a vehicle. User input is received, wherein the user input comprises a set of postures / gestures made by a human operator relative to an item in a set of items in a field of view of the portable inspection device based on the orientation of the mobile inspection device. An annotation is created in association with an item in a set of items in the field of view of the portable inspection device at a location relative to the vehicle, wherein the annotation is assigned to the location. The annotation is displayed in association with the item in the field of view on a graphical user interface of a display system for the mobile inspection device.

[0029] When this type of process is used to identify nonconformities of an object (such as an aircraft) during an inspection, resolving these nonconformities can be more effectively performed with an augmented reality system. The illustrative embodiments recognize and take into account that a human operator can locate nonconformities without relying on physical markers. Therefore, more efficient inspections can be performed.

[0030] In one illustrative example, an augmented reality system is used to visualize nonconformance data of a physical object. The augmented reality system includes a portable computing device and an augmented reality application. An example of an augmented reality application software that can be used and configured for implementations according to the present disclosure is Vuforia 100 developed by Vuforia and PTC. TM The portable computing device is configured to capture an image of a physical object. The augmented reality application plots points in a coordinate cube, wherein the points correspond to non-conforming locations of the physical object.

[0031] The augmented reality application determines a subset of the points plotted in the coordinate cube, the subset of points corresponding to the area of ​​the physical object visible in an image acquired by the portable computing device at the orientation of the portable computing device. The subset of points excludes non-conforming locations that are occluded from view by structures of the physical object in the acquired image. The augmented reality application displays the non-conforming data for the subset of points visible in the image in association with the subset of non-conforming locations of the physical object in the image displayed on the display system in the portable computing device.

[0032] Referring now to the drawings and in particular to Figure 1 , depicts a diagram of an aircraft manufacturing environment in accordance with an illustrative embodiment. In this example, aircraft manufacturing environment 100 includes fuselage section 102. As depicted, human operator 104 walks through interior 106 to perform an inspection of fuselage section 102 at a current stage of manufacturing.

[0033] In this illustrative example, human operator 104 carries tablet computer 108, which is a type of portable computing device. Tablet computer 108 enables human operator 104 to visualize locations of nonconformities of fuselage section 102. As depicted, the locations of nonconformities are locations within interior 106 that are visible based on the orientation of tablet computer 108. In this illustrative example, the orientation includes the location of tablet computer 108 in three-dimensional space and the orientation of tablet computer 108. In other words, tablet computer 108 can be pointed in different directions by human operator 104 to display images of interior 106 of fuselage section 102 to human operator 104.

[0034] In this illustrative example, the image displayed on the tablet computer 108 is a live view of the interior 106 of the fuselage section 102. The nonconformance data can be displayed on the live view to provide an augmented reality display. The nonconformance data can be displayed by overlaying the live view of the fuselage section 102 using an augmented reality application running on the tablet computer 108. The live view is a view of the interior 106 of the fuselage section 102 when the tablet computer 108 is in or near the interior 106 of the fuselage section 102. In other words, an image of the interior 106 of the fuselage section 102 is generated and enhanced with the superimposed nonconformance data. When the tablet computer 108 changes orientation, the image changes to show the view from the current orientation of the tablet computer 108. The image for the live view can be displayed in real time as a video. In other words, the human operator 104 can see what is currently in the view of the tablet computer 108.

[0035] As depicted, tablet computer 108 includes a sensor system configured to generate scan data describing points in space of fuselage section 102. The sensor system can include at least one of a camera, a stereo camera, a laser scanner, or some other suitable type of device capable of generating scan data.

[0036] The scan data (or image data collected by the camera or sensor) can be sent to the server computer 110 using the wireless communication link 112. In this illustrative example, the tablet computer 108 and the server computer 110 form an augmented reality system.

[0037] As depicted, server computer 110 processes the scan data to generate a point cloud representation of the surface contours of a region of fuselage section 102 that substantially corresponds to the same fuselage section in the live view. Further, server computer 110 generates a model of the region of fuselage section 102 using the point cloud representation.

[0038] In this illustrative example, tablet computer 108 (and its position relative to fuselage section 102) can be located using a model of the area of ​​fuselage section 102. Further, the position (and orientation) of tablet computer 108 relative to fuselage section 102 can be identified by positioning techniques, where the orientation may be the direction in which tablet computer 108 is pointed. The position and orientation can be identified using currently available techniques for simultaneous localization and mapping (SLAM), which enables building or updating a map of aircraft manufacturing environment 100 while tracking the position of tablet computer 108 within the environment as tablet computer 108 is moved.

[0039] Positioning can be performed using an identifiable reference point for the fuselage section 102. The reference point can be a position target identified or discerned by the simultaneous positioning and mapping process. The position target can be an object or structure on the fuselage section 102, such as a reflective target disposed on the surface of the fuselage section 102 that is identifiable via camera images, laser scanning, or other suitable detection means. For example, the target location can be a doorway, a monument, a window, an anchor plate, or some other structure or feature on the fuselage section 102 that has a known position relative to the model of the fuselage section 102.

[0040] The location object enables tablet computer 108 to associate the position of the location object with a corresponding position in the model of fuselage section 102. This identification enables tablet computer 108 to determine the position of tablet computer 108 relative to the location object having a known position, and further locate itself relative to fuselage section 102 and the model of fuselage section 102.

[0041] In this illustrative example, server computer 110 is able to plot points within a coordinate cube (which may be retrieved, for example, from a nonconformance database) corresponding to nonconformance locations on fuselage section 102. Server computer 110 is also able to determine which points of the nonconformance locations are visible in an image of fuselage section 102 displayed on tablet computer 108. These determined visible points are displayed in the image to enhance the view seen by human operator 104. As depicted, other plotted points that are determined to be obscured from view in the image based on the orientation of tablet computer 108 relative to the point cloud representation and generated model associated with the region of fuselage section 102 are not displayed in the image displayed by tablet computer 108.

[0042] Thus, tablet computer 108 provides human operator 104 with an augmented reality view that enables human operator 104 to visualize where nonconformances exist in fuselage section 102 .

[0043] Further, the human operator 104 can also operate the tablet computer 108 to input, mark or record newly identified non-conforming locations. Figure 1 The illustration of aircraft manufacturing environment 100 in FIG. 1 is provided as an example of one implementation of an augmented reality system and is not intended to limit the manner in which other augmented reality systems can be implemented. For example, depending on the amount of processing resources available on tablet computer 108, one or more of the different operations described as being performed by server computer 110 can be performed by tablet computer 108. Further, tablet computer 108 can be used to visualize nonconformance data for other objects besides fuselage section 102. For example, other objects can include an entire aircraft, a wing, a car, a train, a building, or some other suitable structure for which visualization of nonconformance data is desired.

[0044] Reference now Figure 2 , depicts an illustration of a block diagram of a visualization environment according to an illustrative embodiment. Visualization environment 200 is an environment in which information of physical object 204 can be visualized. In this illustrative example, physical object 204 can have many nonconformities 206 at many nonconformity locations 208. As used herein, "many" when used with reference to an item means one or more items. For example, many nonconformities 206 are one or more nonconformities 206.

[0045] In this illustrative example, nonconformance is when a portion of physical object 204 deviates from a specification, standard, or some other provision. In this illustrative example, the specification, standard, or some other provision can be from a government agency, the Federal Aviation Administration, a manufacturer, a maintenance provider, an alliance, or some other suitable source. The standard includes provisions regarding at least one of performance, safety, aesthetics, or other factors.

[0046] As depicted, physical object 204 can take many different forms. For example, physical object 204 is selected from the group consisting of a mobile platform, a stationary platform, a land-based structure, a water-based structure, a space-based structure, an aircraft, a commercial aircraft, a rotorcraft, a surface vessel, a tank, a personnel carrier, a train, a spacecraft, a space station, a satellite, a submarine, a car, a power plant, a bridge, a dam, a house, a manufacturing facility, a building, an aircraft structure, a wing, a fuselage section, an engine casing, an engine, a composite panel, a wall, a skin panel, a chair, and other suitable objects.

[0047] As depicted, nonconformance data 210 of physical object 204 can be visualized using augmented reality system 212. In this illustrative example, augmented reality system 212 includes a number of different components. As depicted, augmented reality system 212 includes portable computing device 214, computer system 216, scanning application 218, modeling application 220, positioning application 222, and augmented reality application 224.

[0048] Computer system 216 is a physical hardware system and includes one or more data processing systems. When more than one data processing system is present in computer system 216, those data processing systems communicate with each other using a communication medium. The communication medium can be a network. The data processing system can be selected from at least one of a computer, a server computer, a tablet computer, or some other suitable data processing system.

[0049] As depicted, portable computing device 214 is a data processing system that includes display system 226 for use in visualizing nonconformance data 210. Portable computing device 214 has at least one of a size, weight, or shape that is configured to be carried by human operator 228. Portable computing device 214 is capable of being held or worn by human operator 228.

[0050] The portable computing device 214 can take many different forms. For example, the portable computing device 214 can be selected from the group consisting of a mobile phone, a tablet computer, a head-mounted device, smart glasses, and other suitable devices.

[0051] In this illustrative example, portable computing device 214 is configured to generate scan data 230 using sensor system 232 in portable computing device 214 .

[0052] As depicted, nonconforming data 210 is located in database 233. In this illustrative example, database 233 is in a remote location to augmented reality system 212. In other illustrative examples, database 233 can be located in computer system 216.

[0053] In the depicted example, scan data 230 describes point 234 of physical object 204 at manufacturing stage 236. Manufacturing stage 236 can be selected from the group consisting of: structural stage, wing-body bonding, final body bonding, system installation, interior installation, flight testing, and other suitable stages. In this illustrative example, image 238 of area 240 of physical object 204 is generated using sensor system 232 in portable computing device 214.

[0054] As depicted, image 238 of area 240 of physical object 204 is displayed on display system 226 in portable computing device 214. In this illustrative example, image 238 of area 240 is a live view of physical object 204 as seen using portable computing device 214. As portable computing device 214 moves, new images can be generated and displayed on display system 226 in portable computing device 214. The new images show the current view of physical object 204. Images of physical object 204 can be periodically generated in real time and displayed on display system 226 as a video.

[0055] In this illustrative example, sensor system 232 is a physical sensor system that includes many components. For example, sensor system 232 can include at least one of a 3D scanner, a camera, a global positioning system receiver, a structured light 3D scanner, a radar system, a laser scanner, a conoscopic holography system, a time-of-flight 3D scanner, or other suitable types of sensors.

[0056] As used herein, when used with a list of items, the phrase "at least one of" means that different combinations of one or more of the listed items can be used, and only one of each item in the list may be required. In other words, "at least one of" means that any combination of items and multiple items can be used from the list, but not all items in the list are required. Items can be specific objects, things, or categories.

[0057] For example, but not limited to, "at least one of item A, item B, and item C" can include item A, item A and item B, or item B. This example can also include item A, item B, and item C, or item B and item C. Of course, any combination of these items can exist. In some illustrative examples, "at least one of" can be, for example, but not limited to, two of item A; one of item B; and ten of item C; four of item B and seven of item C; or other suitable combinations.

[0058] The laser scanner can scan the physical object 204 to generate scan data 230. The camera can generate an image 238 of an area 240 of the physical object 204.

[0059] Portable computing device 214 sends scan data 230 to computer system 216. In this illustrative example, portable computing device 214 is in communication with computer system 216. Communication can be facilitated using a communication medium such as a network, a wireless communication link, a wired communication link, or using some other suitable medium.

[0060] Scanning application 218 in computer system 216 is software. When scanning application 218 is executed by computer system 216, scanning application 218 generates point cloud representation 242 of area 240 of physical object 204 using scan data 230. In this illustrative example, scanning application 218 places points 234 in scan data 230 that are detected in area 240 of physical object 204 in point cloud representation 242.

[0061] Point cloud representation 242 can be imported into a modeling application to create a three-dimensional model. In this illustrative example, modeling application 220 in computer system 216 is software that generates model 244 from point cloud representation 242. Model 244 is a three-dimensional model and can be, for example, a computer-aided design (CAD) model, a computer-aided manufacturing (CAM) model, or some other suitable model. In this example, modeling application 220 can generate model 244 of area 240 of physical object 204 using point cloud representation 242 of area 240 of physical object 204 in defined coordinate cube 246.

[0062] In this illustrative example, positioning application 222 in portable computing device 214 is software that locates portable computing device 214 based on model 244 generated from point cloud representation 242 of surface contours of area 240 of physical object 204 in defined coordinate cube 246 using position target 250, and identifies position 248 of portable computing device 214 relative to physical object 204. In this example, positioning application 222 can implement currently available techniques for simultaneous localization and mapping (SLAM) that enables building or updating a map of aircraft manufacturing environment 100 while tracking position 248 of portable computing device 214 within visualization environment 200. In this illustrative example, position 248 of portable computing device 214 is the location of portable computing device 214 in three-dimensional space and the orientation of portable computing device 214.

[0063] In one illustrative example, positioning application 222 locates position target 250 of physical object 204 in locating portable computing device 214 with model 244. Positioning application 222 uses target position 252 of position target 250 to locate portable computing device 214 with model 244 of area 240 of physical object 204 defined in coordinate cube 246 and identifies position 248 of portable computing device 214 relative to physical object 204.

[0064] In this illustrative example, location target 250 is a physical feature. Location target 250 can be, for example, a plate, a barcode, a radio frequency identifier (RFID) device, an anchor, a spatial anchor, an anchor plate, or some other suitable structure. Location target 250 is in a known location on physical object 204 and has a corresponding location in model 244 that represents physical object 204. The location is in three dimensions, and if the orientation of location target 250 is also known, it can be target position 252.

[0065] The position of the location target 250 can be associated with a corresponding position in the model 244 to identify the position 248 of the portable computing device 214 relative to the physical object 204. The location target 250 can be used as a reference by the positioning application 222 to display and / or overlay augmented reality information on the live view of the physical object 204 in the captured image 238 with a desired level of accuracy via the augmented reality application 224 in the portable computing device 214 and displayed on the display system 226 in the portable computing device 214. In this example, the position 248 of the portable computing device 214 can be more accurately located by the positioning application 222 with the use of the location target 250.

[0066] As depicted, augmented reality application 224 is located in portable computing device 214. In this example, augmented reality application 224 plots points 234 in defined coordinate cube 246 that correspond to nonconforming locations 208 on physical object 204. Augmented reality application 224 determines a subset 254 of points 234 plotted in defined coordinate cube 246 that are visible in image 238 of area 240 of physical object 204 captured by portable computing device 214 at position 248 of portable computing device 214.

[0067] In this illustrative example, in determining a subset 254 of points 234 drawn in a defined coordinate cube 246 that are visible in image 238, augmented reality application 224 determines which points 234 drawn within defined coordinate cube 246 for non-conforming locations 208 are located at a predetermined distance outside of a physical object structure in image 238 that are obscured from view from portable computing device 214 at location 248, such as locations determined to be outside of the physical object structure when viewed at the location of portable computing device 214 (based on a model / point cloud representation).

[0068] For example, where the determined distance between a given non-conforming location and the location of the portable computing device 214 is determined to be greater than the distance of the physical object structure from the portable computing device 214, such non-conforming location may be outside the physical object structure as viewed from the location of the portable computing device. In another example, where the portable computing device 214 is located within the interior space of the physical object structure and is oriented toward a portion of the physical object structure, and the given non-conforming location is determined to be outside the interior space based on a comparison between the non-conforming location and the model representation of the physical object structure, such non-conforming location is outside the physical object structure when viewed from the location of the portable computing device 214. These points that are outside the physical object structure by a predetermined distance are excluded from the subset 254 of points 234.

[0069] In one illustrative example, physical object 204 is an aircraft, and physical object structure is an aircraft structure. Augmented reality application 224 determines which points in remaining points 258 for non-conforming location 208 in defined coordinate cube 246 are to be occluded by determining that each non-conforming location is located outside a predetermined distance from the displayed aircraft structure that corresponds to the non-conforming location being occluded from view.

[0070] The augmented reality application 224 displays the nonconformance data 210 for the subset 254 of the points 234 visible in the image 238 in association with the subset 256 of the nonconformance locations 208 of the physical object 204 in the image 238 displayed on the display system 226 in the portable computing device 214. In this example, the nonconformance data 210 for the subset 254 of the points 234 is live nonconformance data. In other words, the nonconformance data reflects nonconformances that still exist in the physical object 204. In other words, these live nonconformances have not yet been resolved.

[0071] In this illustrative example, in displaying nonconforming data 210, augmented reality application 224 displays nonconforming data 210 for subset 254 of points 234 visible in image 238 in association with subset 256 of nonconforming locations 208 of physical object 204 in image 238 displayed on display system 226 in portable computing device 214, without displaying nonconforming data 210 for nonconforming locations 208 that are occluded from view in image 238 of area 240 of physical object 204. For example, if a nonconforming location is located on a surface of physical object 204 that is not visible in image 238, the nonconforming location is considered occluded and is not displayed.

[0072] As depicted, display of at least one of point 234 visible to portable computing device 214 at position 248, nonconformance data 210, or other information of nonconformance location 208 is performed using graphical indicator 257. A graphical indicator in graphical indicator 257 can include at least one of an icon, pictogram, ideogram, graphic, image, text, animation, boldface, chart, line, arrow, or other suitable graphic.

[0073] In this manner, augmentation of image 238 with nonconformance data 210 is performed relative to the portion of physical object 204 that can be seen by human operator 228 in image 238. As human operator 228 moves with portable computing device 214, orientation 248 of portable computing device 214 changes. This change in orientation 248 can change which surfaces are visible, which can affect the display of nonconformance data 210. In this manner, the augmented view of physical object 204 can change dynamically as orientation 248 of portable computing device 214 changes relative to physical object 204.

[0074] In another illustrative example, points 234 of all non-conforming locations 208 can be displayed, including visible and occluded non-conforming locations. This feature can be useful for panoramic view 260 of physical object 204. For example, augmented reality application 224 can determine whether portable computing device 214 is greater than a selected distance of panoramic view 260 and display points 234 of all non-conforming locations 208 in defined coordinate cube 246 to provide panoramic view 260 of all non-conforming locations 208.

[0075] Further, augmented reality application 224 can also filter nonconformance data 210 based on time information to display at least one of historical nonconformance data or active nonconformance data. Historical nonconformance data is nonconformance data of at least one of physical object 204 or other instances of physical object 204. Nonconformance historical data can include nonconformances that have been resolved or reworked. Filtering can be for a specific time period. In this illustrative example, active nonconformance data is nonconformance data 210 of physical object 204 that has not yet been resolved.

[0076] In another illustrative example, filtering can be performed for nonconformance data 210 of other physical objects of the type. Thus, a view of nonconformance 206 can be observed for multiple physical objects. These physical objects at the same time can be physical objects manufactured in the same facility or line. In another example, nonconformance 206 can be observed from multiple physical objects maintained at the same maintenance facility.

[0077] At least one of the scanning application 218, the modeling application 220, the positioning application 222, or the augmented reality application 224 can be implemented in software, hardware, firmware, or a combination thereof. When software is used, the operations performed by these components can be implemented in a program code configured to run on hardware (such as a processor unit). When firmware is used, the operations performed by these components can be implemented in a program code and data manner and stored in a permanent memory to run on a processor unit. When hardware is employed, the hardware may include circuits that operate to perform the operations in these components.

[0078] In the illustrative examples, the hardware can take the form of at least one of a circuit system, an integrated circuit, an application specific integrated circuit (ASIC), a programmable logic device, or some other suitable type of hardware configured to perform a number of operations. Using a programmable logic device, the device can be configured to perform the many operations. The device can be reconfigured at a later time, or can be permanently configured to perform the many operations. The programmable logic device includes, for example, a programmable logic array, a programmable array logic, a field programmable logic array, a field programmable gate array, and other suitable hardware devices. In addition, the process can be implemented in an organic component manner integrated with an inorganic component, and can be composed entirely of organic components excluding humans. For example, the process can be implemented as a circuit in an organic semiconductor.

[0079] In one illustrative example, there are one or more technical solutions to overcome the technical problem of effectively identifying and resolving nonconformities of an aircraft. Thus, one or more technical solutions can provide a technical effect of enhancing the live view of physical object 204 in image 238 to provide visualization of nonconformity location 208 in image 238 that is visible and unobstructed in the view for human operator 228.

[0080] Computer system 216 and portable computing device 214 can be configured to perform at least one of the steps, operations, or actions described in the different illustrative examples using software, hardware, firmware, or a combination thereof. Thus, at least one of computer system 216 or portable computing device 214 operates as a special-purpose computer system, wherein different applications in these components enable visualization of non-conforming data 210 in an augmented view of physical object 204. Specifically, augmented reality application 224 transforms computer system 216 or portable computing device 214 into a special-purpose computer system compared to currently available general-purpose computer systems without augmented reality application 224.

[0081] In this manner, tablet computer 108 enables human operator 104 to view non-conforming data for fuselage section 102 using the point cloud generation, simultaneous positioning and mapping process, and the database of non-conforming data.

[0082] Figure 2 The illustration of the visualization environment 200 in the embodiment is not intended to mean that the physics or architecture of the way in which the illustrative embodiments can be implemented is limited. Other components in addition to or in place of the illustrated components can be used. Some components may be unnecessary. Moreover, blocks are presented to illustrate some functional components. When implemented in the illustrative embodiments, one or more of these blocks can be combined, divided, or combined and divided into different blocks.

[0083] For example, another component, such as an interface, can be present in augmented reality system 212. Interface can be configured to receive nonconforming location 208 on physical object 204. In these illustrative examples, interface can provide communication with database 233 of nonconforming data 210.

[0084] Additionally, some components illustrated as being located in computer system 216 can be implemented in other components. For example, in some illustrative examples, scanning application 218 can be located in portable computing device 214 instead of computer system 216.

[0085] In another illustrative example, positioning application 222 can be located in computer system 216 as a distributed process instead of or in addition to portable computing device 214. Thus, computer system 216 can include at least one of scanning application 218, modeling application 220, positioning application 222, or augmented reality application 224. Augmented reality device 214 can include at least one of scanning application 218, modeling application 220, positioning application 222, or augmented reality application 224.

[0086] Reference now Figure 3 , depicts an illustration of a model of an aircraft structure within a defined coordinate cube in accordance with an illustrative embodiment. In this illustrative example, model 300 is shown within defined coordinate cube 302 .

[0087] In this example, model 300 represents a portion of an aircraft. In this example, model 300 includes nose 304 and portion 308 of fuselage 310. In this example, model 300 is an example of a model generated using a point cloud representation of a portion of an aircraft.

[0088] For example, model 300 can be generated by extracting geometry and other information from a computer-aided design model of an aircraft. The portion of the computer-aided design model extracted from model 300 can be based on a point cloud representation of the aircraft. In this illustrative example, model 300 can also be a computer-aided design (CAD) model.

[0089] Reference now Figure 4 , depicts an illustration of an image of a cockpit enhanced to indicate an unqualified location according to an illustrative embodiment. In the illustrative examples, the same reference numeral may be used in more than one figure. Such reuse of reference numerals in different figures represents the same element in the different figures.

[0090] In this illustrative example, image 400 illustrates cockpit 402 in aircraft 403. Image 400 is displayed by portable computing device 214. Figure 2 403, which is a live view of cockpit 402. In this illustrative example, cockpit 402 in aircraft 403 is shown in FIG. Figure 3 The model is modeled in Model 300.

[0091] In this illustrative example, nonconforming locations exist in cockpit 402. In this example, the nonconforming locations include nonconforming location 410, nonconforming location 412, nonconforming location 414, nonconforming location 416, nonconforming location 418, nonconforming location 420, and nonconforming location 422. The nonconforming locations can be identified from nonconforming data and model 300 including details of cockpit 402.

[0092] The live view of cockpit 402 in image 400 is enhanced with a graphical indicator to indicate the presence of a nonconformance. In this illustrative example, nonconformances visible in image 400 are enhanced with graphical indicators, while nonconformances obscured in image 400 are not identified using graphical indicators.

[0093] As depicted, nonconforming location 410, nonconforming location 412, nonconforming location 414, and nonconforming location 416 are visible in image 400. However, nonconforming location 418, nonconforming location 420, and nonconforming location 422 are obscured. In the depicted example, these nonconforming locations are obscured in image 400 by chair 424 in the view of cockpit 402.

[0094] In this illustrative example, identifying visible nonconforming locations in image 400 is by augmenting the live view of image 400 with a graphical indicator to identify the nonconforming locations. In this example, the graphical indicator is a starburst-like shape. As depicted, nonconforming location 410 is indicated using a starburst-like shape 430; nonconforming location 412 is indicated using a starburst-like shape 432; nonconforming location 414 is indicated using a starburst-like shape 434; and nonconforming location 416 is indicated using a starburst-like shape 436. For nonconforming location 418, nonconforming location 420, and nonconforming location 422, graphical indicators are not displayed on image 400 because these locations are obscured from the view of cockpit 402.

[0095] In this illustrative example, selection of a graphical indicator can result in display of additional information about the nonconformance. For example, selection of starburst-like shape 434 for nonconformance location 414 results in display of nonconformance data 440 in window 442. Nonconformance data 440 can take many different forms. For example, nonconformance data 440 can include at least one of a description of the nonconformance, instructions for resolving the nonconformance, a three-point location of the nonconformance, a work order, an image, a link to additional information, or other suitable information.

[0096] When the orientation of portable computing device 214 changes, the visible nonconformance locations and the obscured nonconformance locations can change. These changes can be identified by displaying graphical indicators for the visible nonconformance locations on the new image of cockpit 402.

[0097] Reference now Figure 5 , depicts an illustration of an image of a cockpit enhanced to indicate an unqualified location in accordance with an illustrative embodiment.

[0098] In this figure, image 500 is a live view of cockpit 402 in which portable computing device 214 has changed position. As depicted, unqualified position 410, unqualified position 412, unqualified position 414, unqualified position 416, unqualified position 418, unqualified position 420, and unqualified position 422 are in this view of cockpit 402 in image 500.

[0099] In this orientation, changes occur in visible nonconforming locations and obscured nonconforming locations. Nonconforming location 410, nonconforming location 412, nonconforming location 414, nonconforming location 420, and nonconforming location 422 are visible in this view of cockpit 402 in image 500. Nonconforming location 416 and nonconforming location 418 are obscured in this view of cockpit 402 in image 500.

[0100] Thus, the live view of cockpit 402 in image 500 is enhanced to indicate the visible nonconforming locations in image 500. In this example, nonconforming location 410, nonconforming location 412, and nonconforming location 414 are still visible and are indicated using starburst-like shape 430, starburst-like shape 432, and starburst-like shape 434, respectively. Nonconforming location 420 has starburst-like shape 510, and nonconforming location 422 has starburst-like shape 512.

[0101] Reference now Figure 6 , depicting an illustration of a panoramic view of an aircraft according to an illustrative embodiment. In image 600, a live external view of aircraft 403 is depicted. Image 600 is a Figure 2 Another example of image 238 generated by portable computing device 214 in FIG.

[0102] In this view, portable computing device 214 is oriented at a selected distance from aircraft 403 resulting in panoramic view 602 of aircraft 403. In this view, image 600 is enhanced with graphical indicators to show all unacceptable locations of aircraft 403 within image 600.

[0103] In this example, the graphical indicator indicates whether the nonconformance is visible on the outside in the live view in image 600. In this example, the graphical indicator in the form of a starburst-like shape indicates a visible nonconformance location. The graphical indicator in the form of a circle indicates an occluded nonconformance location.

[0104] In this example, panoramic view 602 includes starburst-like shape 610 , starburst-like shape 612 , starburst-like shape 614 , starburst-like shape 616 , starburst-like shape 618 , starburst-like shape 620 , starburst-like shape 622 , and starburst-like shape 624 to indicate unqualified locations visible in this view of aircraft 403 in image 600 .

[0105] As depicted, ineligible locations that are occluded in this view of aircraft 403 in image 600 include circle 630 , circle 632 , circle 634 , circle 636 , circle 638 , circle 640 , circle 642 , circle 644 , circle 646 , circle 648 , circle 650 , circle 652 , and circle 654 .

[0106] In this way, the human operator is able to identify which non-conforming locations are actually visible in the live view of aircraft 403 in image 600 .

[0107] Next turn Figure 7 , depicts a flow diagram of a process for visualizing nonconforming data for a physical object, according to an illustrative embodiment. Figure 7The processes in can be implemented in hardware, software, or both. When implemented in software, the processes can take the form of program code that is executed by one or more processor units in one or more hardware devices in one or more computer systems. For example, the processes can be implemented in Figure 1 Tablet computer 108 or Figure 2 The processes can be implemented in an augmented reality application located in one of these computing systems.

[0108] The process begins by mapping points corresponding to non-conforming locations on a physical object in a defined coordinate cube (operation 700). In operation 700, the defined coordinate cube defines a region of the physical object being modeled.

[0109] In the illustrative example, the defined coordinate cube may contain all or part of the physical objects. The selection of how many physical objects are modeled depends on the processing resources available for the amount of data generated for the models of the physical objects.

[0110] The process determines a subset of points mapped in a defined coordinate cube corresponding to an area of ​​a physical object that is visible in an image of the area of ​​the physical object captured by a portable computing device, wherein the subset excludes mapped points determined to be located outside of physical object structure when viewed at the orientation of the portable computing device (operation 702). The subset of points excludes non-conforming locations that are occluded from view by structures of the physical object in the image. One manner in which operation 702 can be implemented includes determining which remaining points of the non-conforming locations are to be occluded by determining that each non-conforming location is located outside of a predetermined distance from the physical object structure displayed in the image that corresponds to the non-conforming location being occluded from view.

[0111] The process displays the nonconforming data for the subset of points visible in the image in association with the subset of nonconforming locations of the physical object in the image displayed on the display system in the portable computing device (operation 704). The process terminates thereafter.

[0112] Next reference Figure 8 , depicts an illustration of a flowchart of a process for determining a subset of points in accordance with an illustrative embodiment. Figure 8 The process shown in the figure is Figure 7 An example of an implementation of operation 702 in which a subset of points of ineligible locations that are visible to the portable computing device in a current orientation of the portable computing device are identified.

[0113] The process begins by determining which points mapped within a defined coordinate cube for non-qualified locations are located at a predetermined distance outside of a physical object structure in the image that is obscured from the view of the portable computing device at the orientation of the portable computing device (operation 800). The process excludes those points that are the predetermined distance outside of the physical object structure (operation 802). The remaining points form a subset of points. The process terminates thereafter.

[0114] Next turn Fig. 9 , depicts an illustration of a flowchart of a process for identifying an orientation of a portable computing device in accordance with an illustrative embodiment. Fig. 9 The processes in can be implemented in hardware, software, or both. When implemented in software, the processes can take the form of program code that is executed by one or more processor units in one or more hardware devices in one or more computer systems. For example, the processes can be implemented in Figure 1 The tablet computer 108 in the Figure 2 A computer running on at least one of the portable computing device 214 or the computer system 216 Figure 2 These processes can be implemented in an augmented reality application located in one of these computing systems.

[0115] The process begins by locating a place target for a physical object (operation 900). The place target can be a specific physical object structure. For example, the place target can be a barcode, a plate, a portal, a doorway, or some other suitable structure in the physical object. The place target has a known position in the physical object and a corresponding position in the model of the physical object.

[0116] The process uses a target position of a location target to locate a portable computing device using a model of an area of ​​a physical object in a defined coordinate cube (operation 902 ).

[0117] The process identifies the orientation of the portable computing device relative to the physical object (operation 904). The process terminates thereafter.

[0118] refer to Fig.10 , depicts an illustration of a flowchart of a process for filtering non-conforming data in accordance with an illustrative embodiment. Fig.10 The processes in can be implemented in hardware, software, or both. When implemented in software, the processes can take the form of program code that is executed by one or more processor units in one or more hardware devices in one or more computer systems. For example, the processes can be implemented in Figure 1 In at least one of the tablet computer 108 or the server computer 110 or Figure 2The invention is implemented in at least one of the portable computing device 214 or the computer system 216. Fig.10 The process in is an example of one manner in which nonconforming data can be selected for display on a portable computing device.

[0119] The process begins with identifying a volume where unqualified data is expected (operation 1000). In operation 1000, the volume can be a defined coordinate cube. The process identifies a set of physical objects, including the physical object in which the portable computing device is located (operation 1002). In operation 1002, a set of physical objects can include physical objects of the same type other than the physical object in which the portable computing device is located. In this way, unqualified data can be retrieved for other physical objects of the same type. This type of retrieval allows for unqualified comparisons across different physical objects of the same type. For example, objects of the same type can be all physical objects on a specific line or manufacturing facility.

[0120] The process also identifies a time when conforming data is expected (operation 1004). The time can be the current time at which nonconforming data is for a physical object for which nonconformities currently exist. In another illustrative example, the time can be a time range that includes nonconformities that can have been resolved for the physical object.

[0121] The process then retrieves the non-conforming data using the time and the set of physical objects (operation 1006). The process terminates thereafter.

[0122] The flow chart and block diagram in the different embodiments described illustrate the framework, function and operation of some possible embodiments of the equipment and method in the illustrative embodiment.In this respect, each square block in the flow chart or block diagram can represent at least one of a part of a module, a section, a function or an operation or a step.For example, one or more in the square block can be implemented as a combination of program code, hardware or program code and hardware.When implementing in hardware, hardware can for example take the form of an integrated circuit that is manufactured or configured to perform one or more operations in the flow chart or block diagram.When being implemented as a combination of program code and hardware, implementation can take the form of firmware.Each square block in the flow chart or block diagram can be implemented using a combination of a dedicated hardware system or dedicated hardware that performs different operations and a program code that is run by dedicated hardware.

[0123] In some alternative implementations of the illustrative examples, one or more functions annotated in the blocks may not appear in the order annotated in the figure. For example, in some cases, two blocks shown in succession may be executed substantially simultaneously, or blocks may sometimes be executed in reverse order, depending on the functions involved. Moreover, blocks other than the blocks shown may be added to the flow chart or block diagram.

[0124] Now turn to Fig.11 , depicts a diagram of a block diagram of a data processing system according to an illustrative embodiment. Data processing system 1100 can be used to implement Figure 1 Tablet computer 108 and server computer 110 in the data processing system 1100 can also be used to implement Figure 2 1104 and computer system 216. In this illustrative example, data processing system 1100 includes communications framework 1102, which provides communications between processor unit 1104, memory 1106, persistent storage 1108, communications unit 1110, input / output (I / O) unit 1112, and display 1114. In this example, communications framework 1102 takes the form of a bus system.

[0125] The processor unit 1104 is used to execute instructions for software that can be loaded into the memory 1106. The processor unit 1104 includes one or more processors. For example, the processor unit 1104 can be selected from at least one of a multi-core processor, a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a network processor, or some other suitable type of processor.

[0126] Memory 1106 and persistent storage 1108 are examples of storage 1116. A storage device is any piece of hardware capable of storing information (e.g., but not limited to, data, program code in functional form, or at least one of other suitable information on a temporary basis, a permanent basis, or both a temporary basis and a permanent basis). In these illustrative examples, storage 1116 may also be referred to as a computer-readable storage device. In these examples, memory 1106 can be, for example, a random access memory, or any other suitable volatile or non-volatile storage device. Persistent storage 1108 can take various forms, depending on the specific implementation.

[0127] For example, persistent storage 1108 may include one or more components or devices. For example, persistent storage 1108 can be a hard drive, a solid state drive (SSD), a flash memory, a rewritable optical disk, a rewritable tape, or some combination thereof. The media used by persistent storage 1108 can also be removable. For example, a removable hard drive can be used for persistent storage 1108.

[0128] In these illustrative examples, communications unit 1110 provides for communications with other data processing systems or devices. In these illustrative examples, communications unit 1110 is a network interface card.

[0129] Input / output unit 1112 allows for input and output of data with other devices that can be connected to data processing system 1100. For example, input / output unit 1112 may provide a connection for user input through at least one of a keyboard, a mouse, or some other suitable input device. Further, input / output unit 1112 may send output to a printer. Display 1114 provides a mechanism for displaying information to a user.

[0130] Instructions for at least one of an operating system, applications, or programs can be located in storage devices 1116, which are in communication with processor unit 1104 through communications framework 1102. The processes of the different embodiments can be performed by processor unit 1104 using computer-implemented instructions that may be located in a memory, such as memory 1106.

[0131] These instructions are referred to as program code, computer usable program code, or computer readable program code, which can be read and executed by a processor in processor unit 1104. The program code in different embodiments can be implemented on different physical or computer readable storage media, such as memory 1106 or persistent storage 1108.

[0132] Program code 1118 is located in functional form on selectively removable computer readable media 1120 and can be loaded or transferred to data processing system 1100 for execution by processor unit 1104. In these illustrative examples, program code 1118 and computer readable media 1120 form computer program product 1122. In the illustrative example, computer readable media 1120 is computer readable storage media 1124.

[0133] In these illustrative examples, computer readable storage media 1124 is a physical or tangible storage device used to store program code 1118 rather than a medium that propagates or transmits program code 1118 .

[0134] Alternatively, program code 1118 can be transferred to data processing system 1100 using a computer readable signal medium. Computer readable signal media can be, for example, a propagated data signal containing program code 1118. For example, a computer readable signal medium can be at least one of an electromagnetic signal, an optical signal, or any other suitable type of signal. These signals can be transmitted via a connection, such as a wireless connection, a fiber optic cable, a coaxial cable, an electric wire, or any other suitable type of connection.

[0135] The different components illustrated for data processing system 1100 are not intended to provide architectural limitations for the manner in which different embodiments may be implemented. In some illustrative examples, one or more of the components may be included in or otherwise form a part of another component. For example, in some illustrative examples, memory 1106 or portions thereof may be included in processor unit 1104. Different illustrative embodiments can be implemented in a data processing system including components in addition to or in place of those illustrated for data processing system 1100. Fig.11 Other components shown in the diagram can be different from the illustrated examples. Different embodiments can be implemented using any hardware device or system capable of running program code 1118.

[0136] Illustrative embodiments of the present disclosure may be found in Fig.12 Aircraft manufacturing and service method 1200 as shown in Fig.13 1300 is described in the context of the aircraft 1300 shown in FIG. Fig.12 , depicts an illustration of a block diagram of an aircraft manufacturing and service method according to an illustrative embodiment. During pre-production, aircraft manufacturing and service method 1200 may include Fig.13 Specification and design 1202 of the aircraft 1300 and material procurement 1204.

[0137] During production, Fig.13 Component and subassembly manufacturing 1206 and system integration 1208 of the aircraft 1300 in FIG. Fig.13 The aircraft 1300 in FIG. 1 may be certified and delivered 1210 so that it can be put into service 1212. When put into service 1212 by a customer, Fig.13 Aircraft 1300 is scheduled for routine maintenance and servicing 1214 (which may also include modification, reconfiguration, refurbishment, and other maintenance and servicing).

[0138] Each of the processes of aircraft manufacturing and service method 1200 may be performed or carried out by a system integrator, a third party, an operator, or some of them. In these examples, the operator may be a customer. For purposes of this description, a system integrator may include, but is not limited to, any number of aircraft manufacturers and major system subcontractors; a third party may include, but is not limited to, any number of sellers, subcontractors, and suppliers; and an operator may be an airline, a leasing company, a military entity, a service organization, etc.

[0139] Now refer to Fig.13 , depicts a diagram of an aircraft in which an illustrative embodiment may be implemented. In this example, aircraft 1300 is Fig.1213. The aircraft manufacturing and service method 1200 in the present invention is described herein and may include an aircraft airframe 1302 and a plurality of systems 1304 and an interior 1306. Examples of systems 1304 include one or more of a propulsion system 1308, an electrical system 1310, a hydraulic system 1312, and an environmental system 1314. Any number of other systems may be included. Although an aviation example is shown, different illustrative embodiments may be applied to other industries, such as the automotive industry.

[0140] The apparatus and methods embodied herein may be used in Fig.12 During at least one of the stages of aircraft manufacturing and service method 1200 is employed.

[0141] In an illustrative example, Fig.12 The parts or subassemblies produced in parts and subassembly manufacturing 1206 may be produced in the same manner as when aircraft 1300 is Fig.12 As another example, one or more apparatus embodiments, method embodiments, or a combination thereof may be produced or manufactured in a similar manner to the components or subassemblies produced when the components are put into service 1212 in the manufacturing process. Fig.12 One or more apparatus embodiments, method embodiments, or a combination thereof may be used during component and subassembly manufacturing 1206 and system integration 1208 of the aircraft 1300 when the aircraft 1300 is placed in service 1212, during Fig.12 During maintenance and servicing 1214, or both. For example, Figure 2 Augmented reality system 212 in can be used during component and subassembly manufacturing 1206 and system integration 1208 to visualize nonconformities in parts and assemblies that may exist in aircraft 1300 during the manufacturing phase. Additionally, augmented reality system 212 can also be used to locate nonconformities during maintenance and service 1214.

[0142] Use of a number of the different illustrative embodiments may substantially expedite the assembly of aircraft 1300 , reduce the cost of aircraft 1300 , or both expedite the assembly of aircraft 1300 and reduce the cost of aircraft 1300 . Figure 2 The use of augmented reality system 212 in reduces the time required to find the nonconforming location by augmenting the live view of aircraft 1300 or a part or assembly of aircraft 1300. In addition, augmented reality system 212 can also augment the live view with nonconforming data in addition to the nonconforming location.

[0143] Now turn to Fig.14, an illustration of a block diagram of a product management system is shown according to an illustrative embodiment. The product management system 1400 is a physical hardware system. In this illustrative example, the product management system 1400 may include at least one of a manufacturing system 1402 or a maintenance system 1404.

[0144] Manufacturing system 1402 is configured to manufacture products, such as Fig.13 As shown, manufacturing system 1402 includes manufacturing equipment 1406. Manufacturing equipment 1406 includes at least one of fabrication equipment 1408 or assembly equipment 1410.

[0145] The fabrication device 1408 can be used to fabricate Fig.13 Fabrication equipment 1408 may include equipment for fabricating components of parts for aircraft 1300. For example, fabrication equipment 1408 may include machines and tools. The machines and tools may be at least one of a drill, a hydraulic press, a furnace, a mold, a composite tape laying machine, a vacuum system, a lathe, or other suitable types of equipment. Fabrication equipment 1408 may be used to fabricate at least one of a metal part, a composite part, a semiconductor, a circuit, a fastener, a rib, a skin panel, a spar, an antenna, or other suitable types of parts.

[0146] Assembly equipment 1410 is used to assemble parts to form Fig.13 Specifically, assembly equipment 1410 may be used to assemble components and parts to form Fig.13 The assembly equipment 1410 may also include machines and tools. The machines and tools may be at least one of a robotic arm, a crawler, a fastener installation system, a rail-based drilling system, or a robot. The assembly equipment 1410 may be used to assemble parts such as seats, horizontal stabilizers, wings, engines, engine casings, landing gear transmission systems, and components for the aircraft. Fig.13 Other parts of the aircraft 1300.

[0147] In this illustrative example, maintenance system 1404 includes maintenance equipment 1412. Maintenance equipment 1412 may include Fig.13 Maintenance equipment 1412 may include any equipment used to perform maintenance on aircraft 1300 in FIG. Fig.13 The tools are used to perform various operations on parts on the aircraft 1300 in FIG. These operations may include disassembling a part, refurbishing a part, inspecting a part, reworking a part, manufacturing a replacement part, or using a part for repair. Fig.13 At least one of other operations for performing maintenance on aircraft 1300 in the embodiment of the present invention. These operations may be for routine maintenance, inspection, update, refurbishment, or other types of maintenance operations.

[0148] In the illustrated example, maintenance equipment 1412 may include ultrasonic inspection equipment, x-ray imaging systems, vision systems, drilling tools, crawler traction devices, and other suitable devices. In some cases, maintenance equipment 1412 may include fabrication equipment 1408, assembly equipment 1410, or both to produce and assemble parts that may require maintenance.

[0149] The product management system 1400 also includes a control system 1414. The control system 1414 is a hardware system and may also include software or other types of components. The control system 1414 is configured to control the operation of at least one of the manufacturing system 1402 or the maintenance system 1404. Specifically, the control system 1414 may control the operation of at least one of the fabrication equipment 1408, the assembly equipment 1410, or the maintenance equipment 1412.

[0150] The hardware in control system 1414 may be implemented using hardware that may include computers, circuits, networks, and other types of devices. The control may take the form of directly controlling manufacturing equipment 1406. For example, robots, computer-controlled machines, and other equipment may be controlled by control system 1414. In other illustrative examples, control system 1414 may manage operations performed by human operators 1416 in manufacturing aircraft 1300 or performing maintenance on aircraft 1300. For example, control system 1414 may assign tasks, provide instructions, display models, or perform other operations to manage operations performed by human operators 1416. In these illustrative examples, Figure 2 The augmented reality system 212 in can be implemented in the control system 1414 to manage Fig.13 At least one of manufacturing or maintenance of aircraft 1300.

[0151] For example, augmented reality system 212 can be used by human operator 1416 to obtain nonconformances. For example, portable computing device 214 in augmented reality system 212 can augment a live view of a part, assembly, or product during the manufacturing phase.

[0152] Augmentation can be used to enable human operators 1416 to locate nonconforming locations in less time. Further, augmentation can also be used to control Fig.13 Provides information and instructions to human operator 1416 during at least one of the manufacture or maintenance of aircraft 1300 .

[0153] In various illustrative examples, human operator 1416 may operate or interact with at least one of manufacturing equipment 1406, maintenance equipment 1412, or control system 1414. This interaction may be performed to manufacture Fig.13Aircraft 1300 in.

[0154] Of course, the product management system 1400 can be configured to manage Fig.13 Although product management system 1400 has been described with respect to manufacturing in the aerospace industry, product management system 1400 can be configured to manage products for other industries. For example, product management system 1400 can be configured to manufacture products for the automotive industry as well as any other suitable industry.

[0155] Thus, the illustrative examples provide an augmented reality system that enables a human operator to identify the location of nonconformities on a physical object, such as an aircraft, during the manufacturing phase. The illustrative examples combine three-dimensional point cloud generation, simultaneous localization and mapping processes, and data about nonconformities to provide information to a human operator operating a portable computing device based on the relative orientation of the portable computing device to the physical object.

[0156] In this way, irregularities that are often difficult to find in a defined physical local coordinate system (such as those in an aircraft) can be detected using Figure 2 The augmented reality system 212 in FIG. 1 is used to visualize the nonconformance. In the illustrated example, the augmented reality uses 3D scanner technology to generate a point cloud model. The points of the model use currently available simultaneous localization and mapping (SLAM) processes to identify the position of the portable computing device relative to the reference point. A database of nonconformance data is accessed to provide nonconformance information of the nonconformance visible in the live view from the portable computing device.

[0157] For purposes of illustration and description, descriptions of different illustrative embodiments have been presented, which are not intended to be exhaustive or limited to embodiments of the disclosed forms. Different illustrative examples describe components that perform actions or operations. In the illustrative embodiments, components can be configured to perform the described actions or operations. For example, components can have a configuration or design for a structure that provides the components with the ability to perform the actions or operations described as being performed by the components in the illustrative examples.

[0158] Further, the present disclosure includes embodiments according to the following embodiments:

[0159] Embodiment 1. An augmented reality system (212) for visualizing nonconforming data (210) of a physical object (204), the augmented reality system (212) comprising:

[0160] a portable computing device (214) having a sensor system (232) and a display system (226), wherein the portable computing device (214) and the sensor system (232) are configured to generate scan data (230) describing points in space of the physical object (204) during a manufacturing phase (236);

[0161] a scanning application (218) that, when executed, generates a point cloud representation (242) of a surface contour of the physical object (204) using the scan data (230);

[0162] a modeling application (220) which, when executed, generates a model (244) of the region (240) of the physical object (204) using the point cloud representation (242) of the region (240) of the physical object (204) in a defined coordinate cube (246);

[0163] a positioning application (222) that, when executed, locates the portable computing device (214) based on a position target (250) using the model (244) of the area (240) of the physical object (204) in the defined coordinate cube (246) and identifies an orientation (248) of the portable computing device (214) relative to the physical object (204); and

[0164] An augmented reality application (224) which, when executed,:

[0165] plotting points in the defined coordinate cube (246) corresponding to nonconforming locations (208) on the physical object (204);

[0166] determining a subset (254) of the points (234) mapped in the defined coordinate cube (246), the subset (254) of the points (234) being visible in an image (238) of the region (240) of the physical object (204) acquired by the portable computing device (214), wherein the subset (254) excludes mapped points determined to be located outside of the structure of the physical object when viewed at the location of the portable computing device (214); and

[0167] The nonconforming data (210) of the subset (254) of the points (234) visible in the image (238) is displayed in association with a subset (256) of the nonconforming locations (208) of the physical object (204) in the image (238) displayed on the display system (226) in the portable computing device (214).

[0168] Embodiment 2. An augmented reality system (212) according to embodiment 1, wherein when the non-conforming data (210) of the subset (254) of the points (234) visible in the image (238) is displayed in association with the subset (256) of the non-conforming locations (208) of the physical object (204) in the image (238) displayed on the display system (226) in the portable computing device (214), the augmented reality application (224) is connected to the portable computing device (214) to display the non-conforming data (210) of the subset (254) of the points (234) visible in the image (238) in association with the subset (256) of the non-conforming locations (208) of the physical object (204) in the image (238) displayed on the display system (226) in the portable computing device (214). The subset (256) of the non-conforming locations (208) of the physical object (204) in the image (238) displayed on the display system (226) in 14) is associated with displaying the non-conforming data (210) of the subset (254) of the points (234) visible in the image (238), without displaying the non-conforming data (210) of the non-conforming locations (208) that are obscured from view in the image (238) of the area (240) of the physical object (204).

[0169] Embodiment 3. An augmented reality system (212) according to embodiment 1, wherein when determining a subset (254) of the points (234) mapped in the defined coordinate cube (246), the subset (254) of the points (234) is visible in an image (238) of the area (240) of the physical object (204) acquired by the portable computing device (214), wherein the subset (254) excludes points that are determined to be located in the area (240) when viewed at the orientation of the portable computing device (214). Mapped points outside of a physical object structure, the augmented reality application (224) determining which of the points (234) mapped for the non-conforming location (208) within the defined coordinate cube (246) are located a predetermined distance outside of the physical object structure in the image (238) that are obscured from view from the portable computing device (214) at the orientation (248), wherein those points outside of the physical object structure at the predetermined distance are excluded from the subset (254) of the points (234).

[0170] Embodiment 4. An augmented reality system (212) according to embodiment 1, wherein the physical object (204) is an aircraft structure, and wherein when determining a subset (254) of the points (234) plotted in the defined coordinate cube (246), the subset (254) of the points (234) is visible in an image (238) of the area (240) of the physical object (204) captured by the portable computing device (214) at the orientation of the portable computing device (214), the augmented reality application (224) determines that the distance of each non-conforming location from the displayed aircraft structure is outside a predetermined distance corresponding to the non-conforming location being obscured from view.

[0171] Embodiment 5. An augmented reality system (212) according to Embodiment 1, wherein when positioning the portable computing device (214) using the model (244) of the area (240) of the physical object (204) in the defined coordinate cube (246) and identifying the orientation (248) of the portable computing device (214) relative to the physical object (204), the positioning application (222) positions a position target (250) of the physical object (204), uses the target orientation (252) of the position target (250) to position the portable computing device (214) using the model (244) of the area (240) of the physical object (204) in the defined coordinate cube (246), and identifies the orientation (248) of the portable computing device (214) relative to the physical object (204).

[0172] Embodiment 6. An augmented reality system (212) according to embodiment 1, wherein when the augmented reality application (224) is executed, the augmented reality application (224) determines whether the portable computing device (214) is greater than a selected distance for a panoramic view, and displays the points (234) of all the non-conforming locations (208) in the defined coordinate cube (246) to provide a panoramic view of all the non-conforming locations (208).

[0173] Embodiment 7. An augmented reality system (212) according to embodiment 1, wherein when the augmented reality application (224) is executed, the augmented reality application (224) filters the non-conforming data (210) based on time information to display at least one of historical non-conforming data (210) or active non-conforming data (210).

[0174] Embodiment 8. The augmented reality system (212) of Embodiment 1, wherein the non-conforming data (210) of the subset (254) of the points (234) is live non-conforming data (210).

[0175] Embodiment 9. The augmented reality system (212) according to embodiment 1 further comprises:

[0176] An interface is configured to receive the unqualified location (208) on the physical object (204).

[0177] Embodiment 10. An augmented reality system (212) according to embodiment 1, wherein the sensor system (232) includes at least one of a 3D scanner, a camera, a global positioning system receiver, a structured light 3D scanner, a lidar system, a laser scanner, a conoscopic holographic system, or a time-of-flight 3D scanner.

[0178] Embodiment 11. The augmented reality system (212) of Embodiment 1, wherein the portable computing device (214) is selected from the group consisting of: a mobile phone, a tablet computer, a head mounted device, and smart glasses.

[0179] Embodiment 12. An augmented reality system (212) according to embodiment 1, wherein the physical object (204) is selected from the group consisting of: a mobile platform, a stationary platform, a land-based structure, a water-based structure, a space-based structure, an aircraft, a commercial aircraft, a rotorcraft, a surface ship, a tank, a personnel carrier, a train, a spacecraft, a space station, a satellite, a submarine, a car, a power plant, a bridge, a dam, a house, a manufacturing facility, a building, an aircraft structure, a wing, a fuselage section, an engine casing, an engine, a composite panel, a wall, a skin panel and a chair.

[0180] Embodiment 13. An augmented reality system (212) for visualizing nonconforming data (210) of a physical object (204), the augmented reality system (212) comprising:

[0181] a portable computing device (214) configured to capture an image (238) of the physical object (204);

[0182] An augmented reality application (224) that, when executed by the portable computing device (214), plots points (234) in a defined coordinate cube (246), wherein the points (234) correspond to non-conforming locations (208) on the physical object (204); determines a subset (254) of the points (234) plotted in the defined coordinate cube (246), the subset (254) of the points (234) corresponding to non-conforming locations (208) visible in the image (238) captured by the portable computing device (214) at the location (248) of the portable computing device (214); an area (240) of the physical object (204) of the image (238), wherein the subset (254) of the points (234) excludes unqualified locations (208) that are obscured from view by physical object structures of the physical object (204) in the image (238); and the unqualified data (210) of the subset (254) of the points (234) visible in the image (238) is displayed in association with the subset (256) of the unqualified locations (208) of the physical object (204) in the image (238) displayed on a display system (226) in the portable computing device (214).

[0183] Embodiment 14. The augmented reality system (212) of embodiment 13, wherein the portable computing device (214) includes a sensor system (232) configured to generate scan data (230) describing points (234) in space of the physical object (204) during a manufacturing stage (236), further comprising:

[0184] a scanning application (218) that, when executed, generates a point cloud representation (242) of a surface contour of a region (240) of the physical object (204) using the scan data (230);

[0185] a modeling application (220) which, when executed, generates a model (244) of the region (240) of the physical object (204) using the point cloud representation (242) of the region (240) of the physical object (204) in the defined coordinate cube (246);

[0186] A positioning application (222) which, when executed, locates the portable computing device (214) based on a position target (250) using the model (244) of the area (240) of the physical object (204) in the defined coordinate cube (246), and identifies an orientation (248) of the portable computing device (214) relative to the physical object (204).

[0187] Embodiment 15. The augmented reality system (212) according to embodiment 14 further comprises:

[0188] A computer system, wherein at least one of the scanning application (218), the modeling application (220), and the positioning application is located in the computer system.

[0189] Embodiment 16. The augmented reality system (212) of Embodiment 14, wherein at least one of the scanning application (218), the modeling application (220), and the positioning application (222) is located in the portable computing device (214).

[0190] Embodiment 17. An augmented reality system (212) according to embodiment 13, wherein when determining that a subset (254) of the points (234) plotted in the defined coordinate cube (246) corresponds to the area (240) of the physical object (204) visible in the image (238) captured by the portable computing device (214) at the orientation (248) of the portable computing device (214), the augmented reality system (212) Determining (224) which of the points (234) plotted within the defined coordinate cube (246) for the non-conforming location (208) are located a predetermined distance outside the physical object structure of the physical object (204) in the image (238) that is obscured from view from the portable computing device (214) at the orientation (248), wherein those points that are the predetermined distance outside the physical object structure are excluded from the subset (254) of the points (234).

[0191] Embodiment 18. A method for visualizing nonconforming data (210) of a physical object (204), the method comprising:

[0192] Drawing, by an augmented reality application (224) in a portable computing device (214), a point (234) in a defined coordinate cube (246), the point (234) corresponding to a non-conforming location (208) on the physical object (204);

[0193] determining, by the augmented reality application (224) in the portable computing device (214), a subset (254) of the points (234) mapped in the defined coordinate cube (246), the subset (254) of the points (234) corresponding to an area (240) of the physical object (204) visible in an image (238) of the area (240) of the physical object (204) acquired by the portable computing device (214), wherein the subset (254) excludes mapped points determined to be located outside of the physical object structure when viewed at the location of the portable computing device (214), wherein the subset (254) of the points (234) excludes ineligible locations (208) that are occluded from view by the physical object structure of the physical object (204) in the image (238); and

[0194] The nonconforming data (210) of the subset (254) of the points (234) visible in the image (238) is displayed, via the augmented reality application (224) in the portable computing device (214), in association with the subset (256) of the nonconforming locations (208) of the physical object (204) in the image (238) displayed on a display system (226) in the portable computing device (214).

[0195] Embodiment 19. The method according to embodiment 18, wherein the determining step comprises:

[0196] determining, by the augmented reality application (224) in the portable computing device (214), that the point (234) mapped within the defined coordinate cube (246) for the non-conforming location (208) is located a predetermined distance outside of the physical object structure in the image (238) that is obscured from view from the portable computing device (214) at the location (248); and

[0197] Those points that are a predetermined distance outside of the physical object structure are excluded by the augmented reality application (224) in the portable computing device (214), wherein the remaining points (258) form the subset (254) of the points (234).

[0198] Embodiment 20. The method of embodiment 18, wherein the determining step comprises:

[0199] Determining, by the augmented reality application (224) in the portable computing device (214), which of the remaining points (258) of the non-conforming locations (208) are to be occluded by determining that the distance of each non-conforming location from the physical object structure displayed in the image (238) is outside a predetermined distance corresponding to the non-conforming location being occluded from view.

[0200] Embodiment 21. The method according to embodiment 18, further comprising:

[0201] locating a position target (250) of the physical object (204);

[0202] using the target position (252) of the place target (250) to locate the portable computing device (214) with a model (244) of the area (240) of the physical object (204) in the defined coordinate cube (246) based on the place target (250); and

[0203] The position (248) of the portable computing device (214) relative to the physical object (204) is identified.

[0204] Embodiment 22. The method of embodiment 18, wherein the displaying step comprises:

[0205] The nonconforming data (210) for the subset (254) of the points (234) visible in the image (238) is displayed, via an augmented reality application (224) in a portable computing device (214), in association with the subset (256) of the nonconforming locations (208) of the physical object (204) in the image (238) displayed on the display system (226) in the portable computing device (214), without displaying the nonconforming data (210) for the nonconforming locations (208) that are obscured from view in the image (238) of the area (240) of the physical object (204).

[0206] Embodiment 23. The method according to embodiment 18 further comprises:

[0207] A determination is made as to whether the portable computing device (214) is greater than a selected distance for a panoramic view, and the points (234) of all of the non-conforming locations (208) are displayed in the defined coordinate cube (246) to provide a panoramic view of all of the non-conforming locations (208).

[0208] Embodiment 24. A method according to embodiment 18, wherein when the augmented reality application (224) is executed, the augmented reality application (224) filters the non-conforming data (210) based on time information to display at least one of historical non-conforming data (210) or active non-conforming data (210).

[0209] Embodiment 25. The method of embodiment 18, wherein the nonconforming data (210) of the subset (254) of the points (234) is live nonconforming data (210).

[0210] Embodiment 26. The method of Embodiment 18, wherein the portable computing device (214) is selected from the group consisting of: a mobile phone, a tablet computer, a head mounted device, and smart glasses.

[0211] Embodiment 27. A method according to embodiment 18, wherein the physical object (204) is selected from the group consisting of: mobile platforms, stationary platforms, land-based structures, water-based structures, space-based structures, aircraft, commercial aircraft, rotorcraft, surface ships, tanks, personnel carriers, trains, spacecraft, space stations, satellites, submarines, cars, power plants, bridges, dams, houses, manufacturing facilities, buildings, as well as aircraft structures, wings, fuselage sections, engine casings, engines, composite panels, walls, skin panels and chairs.

[0212] Many modifications and variations will be apparent to those of ordinary skill in the art. In addition, different illustrative embodiments may provide different features than other desired embodiments. The selected embodiment or embodiments are selected and described to better explain the principles of the embodiments, practical applications, and to enable those of ordinary skill in the art to understand the disclosure of the various embodiments, wherein various modifications suitable for specific uses are contemplated.

Claims

1. An augmented reality system (212) for visualizing nonconforming data (210) of a physical object (204), the augmented reality system (212) comprising: a portable computing device (214) having a sensor system (232) and a display system (226), wherein the portable computing device (214) and the sensor system (232) are configured to generate scan data (230) describing points in space of the physical object (204) during a manufacturing phase (236) of the physical object; a scanning application (218) that, when executed, generates a point cloud representation (242) of a surface contour of the physical object (204) using the scan data (230); a modeling application (220) which, when executed, generates a model (244) of the region (240) of the physical object (204) using the point cloud representation (242) of the region (240) of the physical object (204) in a defined coordinate cube (246); a positioning application (222) that, when executed, locates the portable computing device (214) using the model (244) of the area (240) of the physical object (204) in the defined coordinate cube (246) based on the position target (250) of the physical object and identifies an orientation (248) of the portable computing device (214) relative to the physical object (204); as well as An augmented reality application (224) which, when executed,: plotting points in the defined coordinate cube (246) corresponding to nonconforming locations (208) on the physical object (204); determining a subset (254) of the points (234) mapped in the defined coordinate cube (246), the subset (254) of the points (234) being visible in an image (238) of the region (240) of the physical object (204) acquired by the portable computing device (214), wherein the subset (254) excludes mapped points determined to be located outside of the structure of the physical object when viewed at the location of the portable computing device (214); as well as The nonconforming data (210) of the subset (254) of the points (234) visible in the image (238) is displayed in association with a subset (256) of the nonconforming locations (208) of the physical object (204) in the image (238) displayed on the display system (226) in the portable computing device (214).

2. The augmented reality system (212) of claim 1, wherein the augmented reality application (224) is associated with the portable computing device (214) when the non-conforming data (210) of the subset (254) of the points (234) visible in the image (238) is displayed in association with the subset (256) of the non-conforming locations (208) of the physical object (204) in the image (238) displayed on the display system (226) in the portable computing device (214). 4) in association with the subset (256) of the unqualified locations (208) of the physical object (204) in the image (238) displayed on the display system (226) in FIG. 4, the unqualified data (210) of the subset (254) of the points (234) visible in the image (238) is displayed, while the unqualified data (210) of the unqualified locations (208) that are obscured from view in the image (238) of the area (240) of the physical object (204) is not displayed.

3. The augmented reality system (212) of claim 1, wherein in determining the subset (254) of the points (234) mapped in the defined coordinate cube (246), the subset (254) of the points (234) is visible in an image (238) of the area (240) of the physical object (204) acquired by the portable computing device (214), wherein the subset (254) excludes mapped points that are determined to be located outside of the structure of the physical object when viewed at the orientation (248) of the portable computing device (214). The augmented reality application (224) determines which of the points (234) mapped for the non-conforming location (208) within the defined coordinate cube (246) are located a predetermined distance outside of the physical object structure in the image (238), wherein the points located at the predetermined distance outside of the physical object structure in the image are obscured from view from the portable computing device (214) at the orientation (248), wherein the points located at the predetermined distance outside of the physical object structure are excluded from the subset (254) of the points (234).

4. The augmented reality system (212) of claim 1, wherein the physical object (204) is an aircraft structure, and wherein in determining that the subset (254) of the points (234) plotted in the defined coordinate cube (246) are visible in an image (238) of the area (240) of the physical object (204) acquired by the portable computing device (214) at the orientation (248) of the portable computing device (214), the augmented reality application (224) determines that each non-conforming location is located at a distance from the displayed aircraft structure that is outside a predetermined distance corresponding to the non-conforming location being obscured from view.

5. The augmented reality system (212) of claim 1, wherein when positioning the portable computing device (214) with the model (244) of the area (240) of the physical object (204) in the defined coordinate cube (246) and identifying the orientation (248) of the portable computing device (214) relative to the physical object (204), the positioning application (222) positions a position target (250) of the physical object (204), uses the target orientation (252) of the position target (250) to position the portable computing device (214) with the model (244) of the area (240) of the physical object (204) in the defined coordinate cube (246), and identifies the orientation (248) of the portable computing device (214) relative to the physical object (204).

6. The augmented reality system (212) of claim 1, wherein when the augmented reality application (224) is executed, the augmented reality application (224) determines whether the distance between the portable computing device (214) and the physical object is greater than a selected distance for a panoramic view, and displays the points (234) of all of the non-conforming locations (208) in the defined coordinate cube (246) to provide a panoramic view of all of the non-conforming locations (208).

7. The augmented reality system (212) of claim 1, wherein when the augmented reality application (224) is executed, the augmented reality application (224) filters the non-conforming data (210) based on time information to display at least one of historical non-conforming data (210) or active non-conforming data (210).

8. The augmented reality system (212) of claim 1, wherein the non-conforming data (210) for the subset (254) of the points (234) is live non-conforming data (210).

9. The augmented reality system (212) of claim 1, further comprising: An interface is configured to receive the unqualified location (208) on the physical object (204).

10. The augmented reality system (212) of claim 1, wherein the sensor system (232) comprises at least one of a 3D scanner, a camera, a GPS receiver, a structured light 3D scanner, a LiDAR system, a laser scanner, a conoscopic holographic system, or a time-of-flight 3D scanner.

11. The augmented reality system (212) of claim 1, wherein the portable computing device (214) is selected from the group consisting of a mobile phone, a tablet computer, a head mounted device, and smart glasses, the portable computing device (214) comprising a display for displaying the nonconforming data of the physical object (204), the physical object (204) comprising one of a land-based structure, a water-based structure, a space-based structure, an aircraft, a commercial aircraft, a rotorcraft, a surface vessel, a tank, a personnel carrier, a train, a spacecraft, a space station, a satellite, a submarine, a car, a power plant, a bridge, a dam, a house, a manufacturing facility, a building, an aircraft structure, a wing, a fuselage section, an engine casing, an engine, a composite panel, a wall, and a skin panel.

12. A method for visualizing nonconforming data (210) of a physical object (204), the method comprising: Drawing, by an augmented reality application (224) in a portable computing device (214), a point (234) in a defined coordinate cube (246), the point (234) corresponding to a non-conforming location (208) on the physical object (204); determining, by the augmented reality application (224) in the portable computing device (214), a subset (254) of the points (234) mapped in the defined coordinate cube (246), the subset (254) of the points (234) corresponding to the area (240) of the physical object (204) visible in an image (238) of the area (240) of the physical object (204) acquired by the portable computing device (214), wherein the subset (254) excludes mapped points determined to be located outside of physical object structure when viewed at the location (248) of the portable computing device (214), wherein the subset (254) of the points (234) excludes ineligible locations (208) that are occluded from view by physical object structure of the physical object (204) in the image (238); and The nonconforming data (210) of the subset (254) of the points (234) visible in the image (238) is displayed, via the augmented reality application (224) in the portable computing device (214), in association with the subset (256) of the nonconforming locations (208) of the physical object (204) in the image (238) displayed on a display system (226) in the portable computing device (214).

13. The method of claim 12, wherein the determining step comprises: determining, by the augmented reality application (224) in the portable computing device (214), that the points (234) mapped for the non-conforming locations (208) within the defined coordinate cube (246) are located at a predetermined distance outside of the physical object structure in the image (238), wherein the points located at the predetermined distance outside of the physical object structure in the image are occluded from view from the portable computing device (214) at the location (248); and The augmented reality application (224) in the portable computing device (214) excludes those points that are a predetermined distance outside the physical object structure, wherein the remaining points (258) form the subset (254) of the points (234), wherein the method determines, by the augmented reality application (224) in the portable computing device (214), which of the remaining points (258) of the non-conforming locations (208) are to be occluded by determining that the distance of each non-conforming location from the physical object structure displayed in the image (238) is outside the predetermined distance corresponding to the non-conforming location being occluded from view.

14. The method according to any one of claims 12 to 13, wherein the displaying step comprises: The nonconforming data (210) for the subset (254) of the points (234) visible in the image (238) is displayed, via an augmented reality application (224) in a portable computing device (214), in association with the subset (256) of the nonconforming locations (208) of the physical object (204) in the image (238) displayed on the display system (226) in the portable computing device (214), without displaying the nonconforming data (210) for the nonconforming locations (208) that are obscured from view in the image (238) of the area (240) of the physical object (204).

15. The method of any one of claims 12 to 14, wherein the portable computing device (214) is selected from the group consisting of a mobile phone, a tablet computer, a head mounted device, and smart glasses, and the portable computing device (214) includes a display for displaying nonconformance data of the physical object (204), the physical object (204) comprising a land-based structure, a water-based structure, a space-based structure, an aircraft, a commercial aircraft, a rotorcraft, a surface vessel, a tank, a personnel carrier, a train, a spacecraft, a space station, a satellite, a submarine, an automobile, a power plant, a bridge, a dam, a house, a manufacturing facility, a building, and one of an aircraft structure, a wing, a fuselage section, an engine casing, an engine, a composite panel, a wall, a skin panel, and a chair.

Images