Augmented reality system for manufacturing composite parts

Abstract

The present application relates to an augmented reality system for manufacturing composite parts. A method, apparatus, and system for augmenting a real-time view (212) of a task location (244) is disclosed. A portable computing device (238) is positioned to an object (206). A visualization (242) of the task location (244) is displayed on the real-time view (212) of the object (206) for performing the task using a model (246) of the object (206) and a combined map (234) of the object (206). The combined map (234) is generated by scanning the object (206) with different viewpoints (230) of the object (206) by the portable computing device (214).

Description

Technical Field

[0001] The present disclosure relates generally to manufacturing composite parts, and in particular, to methods, apparatus, and systems for manufacturing composite parts using an augmented reality system. Background Art

[0002] Composite parts are manufactured by laying up plies of composite material on a tool. Prior to placement, the plies may be pre-impregnated with resin. These types of plies are known as prepregs. After the plies are laid up on the tool, the plies are cured to form the composite part.

[0003] Laying up the plies in the correct location, order and orientation specified for the composite part is important to achieving the desired performance level. Currently, when operators place the plies, an overhead laser tracker (OLT) is used to place the plies on the tool to display the guide for ply placement. This overhead laser tracker improves the accuracy of the placement of the plies on the tool.

[0004] An overhead laser tracker consists of a laser projector mounted above the layup area, a computer controller, and a set of retro-reflective alignment pins or reference marks that are positioned on the tool and used as reference points to establish the tool position in three-dimensional space. The laser projector displays an outline on the tool that identifies the location of the ply.

[0005] These types of systems are effective but require dedicated space. For example, the laser projector requires a fixed space above the tool for laying the plies. In addition, the overhead laser tracker needs to be located above the tool. In addition, the overhead laser tracker is expensive and also requires calibration and other maintenance.

[0006] Therefore, it would be desirable to have a method and apparatus that takes into account at least some of the above problems as well as other possible problems.For example, it would be desirable to have a method and apparatus that overcomes technical problems by providing a guide for placing a ply on a tool. Summary of the invention

[0007] Embodiments of the present disclosure provide a method for visualizing task information of a layup location on a tool. A portable computing device on an operator is used to scan the tool from different viewpoints of the tool to generate scan data. A point cloud is created by a computer system from the scan data generated by the portable computing device. The computer system uses the point cloud to create a composite map of the tool. A portable computing device among a plurality of portable computing devices is positioned to the tool using the composite map of the tool. Task information of the layup location on the tool is displayed on a real-time view seen by a display device in the portable computing device, the portable computing device having been positioned using the composite map of the tool and a ply model of the composite ply, wherein the displayed task information enhances the real-time view of the tool.

[0008] Another embodiment of the present disclosure provides a method for enhancing a real-time view of a task location. A portable computing device is positioned to an object. A visualization of the task location is displayed on the real-time view of the object for performing the task using a model of the object and a combined map of the object, wherein the combined map is generated by scanning the object with different viewpoints to the object by the portable computing device.

[0009] Yet another embodiment provides an augmented reality system for visualizing a pavement location on a tool. The augmented reality system includes a computer system that operates to receive scan data from a portable computing device on an operator to generate scan data from different viewpoints of the tool. The computer system operates to create multiple maps of the tool using the scan data generated by the portable computing device, and to combine the multiple maps to form a combined map of the tool. The computer system operates to identify task information for the pavement location in a ply model. The computer system operates to send the task information for the pavement location on the tool to a portable computing device among a plurality of portable computing devices. The portable computing device displays the task information for the pavement location on the tool on a real-time view seen by a display device in the portable computing device, which has been positioned using the combined map of the tool and the ply model of the composite ply.

[0010] Yet another embodiment provides an augmented reality system for augmenting a real-time view of a task location. The augmented reality system includes a portable computing device, wherein the portable computing device is positioned to an object and displays a visualization of the task location on a real-time view of the object for performing the task using a model of the object and a combined map of the object. The combined map is generated by scanning the object with different viewpoints to the object by the portable computing device.

[0011] The features and functions can be achieved independently in various embodiments of the present disclosure or may be combined in yet other embodiments in which further details can be seen with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The novel features which are believed to be characteristic of the illustrative embodiments are set forth in the appended claims. However, the illustrative embodiments together with the preferred mode of use, further objects and features thereof will be best understood by reference to the following detailed description of illustrative embodiments of the present disclosure when read in conjunction with the accompanying drawings, in which:

[0013] Figure 1 is a diagrammatic representation of a network of data processing systems in which illustrative embodiments may be implemented;

[0014] Figure 2 is an illustration of a block diagram of an augmented reality environment according to an illustrative embodiment;

[0015] Figure 3 is an illustration of a block diagram of an augmented reality system for visualizing task information for placing a composite ply at a layup location on a tool in accordance with an illustrative embodiment;

[0016] Figure 4 is an illustration of an operator performing operations at a tool according to an illustrative embodiment;

[0017] Figure 5 is an illustration of a flowchart of a process for enhancing a real-time view of task locations on a portable computing device in accordance with an illustrative embodiment;

[0018] Figure 6 is an illustration of a flow chart of a process for processing scan data according to an illustrative embodiment;

[0019] Figure 7 is an illustration of a flow chart of a process for creating a combined map according to an illustrative embodiment;

[0020] Figure 8 is an illustration of a flow chart of a process for creating a combined map according to an illustrative embodiment;

[0021] Fig. 9 is an illustration of a flow chart of a process for visualizing task information for a laydown location on a tool in accordance with an illustrative embodiment;

[0022] Fig.10 is an illustration of a flowchart of a process for displaying a visualization of task information in accordance with an illustrative embodiment;

[0023] Fig.11 is an illustration of a block diagram of a data processing system according to an illustrative embodiment;

[0024] Fig.12 is an illustration of a block diagram of a portable computing device according to an illustrative embodiment;

[0025] Fig.13 is an illustration of a block diagram of an aircraft manufacturing and service method according to an illustrative embodiment;

[0026] Fig.14 is an illustration of a block diagram of an aircraft in which the illustrative embodiments may be implemented; and

[0027] Fig.15 is an illustration of a block diagram of a product management system in accordance with an illustrative embodiment. DETAILED DESCRIPTION

[0028] The illustrative embodiments recognize and take into account one or more different considerations. For example, the illustrative embodiments recognize and take into account that a portable computing device will be used instead of an overhead laser tracker to display the task location. However, the illustrative embodiments recognize and take into account that the use of a portable computing device involves the portable computing device positioning itself to the tool. The illustrative embodiments recognize and take into account that identifying the location and orientation of the portable computing device relative to the tool, and corresponding the view of the tool to the model of the tool, presents unprecedented challenges to the overhead laser tracker. The illustrative embodiments recognize and take into account that generating a map from a portable computing device may not have the desired accuracy.

[0029] The illustrative embodiments recognize and take into account that technical solutions may involve scanning using tools from multiple portable computing devices at different viewpoints. Therefore, the illustrative embodiments provide methods, apparatus, and systems for visualizing information for performing tasks on an object. The illustrative embodiments provide methods, apparatus, and systems for visualizing task locations. Visualization of task locations includes displaying information for performing operations at the task locations in addition to identifying the task locations.

[0030] In one illustrative example, a method enhances a real-time view of a task location. A portable computing device is positioned to an object. A visualization of the task location is displayed on the real-time view of the object for performing the task using a model of the object and a combined map of the object, wherein the combined map is generated by scanning the object with different viewpoints of the object by the portable computing device.

[0031] In another illustrative example, a method provides visualization of task information for a pavement location on a tool. A portable computing device on an operator is used to scan the tool at different viewpoints to generate scan data. A point cloud is created by a computer system from the scan data generated by the portable computing device. The computer system creates a composite map of the tool using the point cloud. A portable computing device from a plurality of portable computing devices is positioned to the tool using the composite map of the tool. Task information for the pavement location on the tool is displayed on a real-time view seen through a display device in the portable computing device, the portable computing device having been positioned using the composite map of the tool and a ply model of a composite ply, wherein the displayed task information augments the real-time view of the tool.

[0032] Referring now to the drawings, and in particular to Figure 1, depicts a pictorial representation of a network of data processing systems in which the illustrative embodiments may be implemented. Network data processing system 100 is a network of computers in which the illustrative embodiments may be implemented. Network data processing system 100 contains network 102, which is the medium used to provide communication links between various devices and computers connected together within network data processing system 100. Network 102 may include connections such as wired, wireless communication links, or fiber optic cables.

[0033] In the depicted example, server computer 104 and server computer 106 are connected to network 102 along with storage unit 108. In addition, client device 110 is connected to network 102. As shown, client device 110 includes client computer 112 and client computer 114. Client device 110 can be, for example, a computer, a workstation, or a network computer. In the depicted example, server computer 104 provides information such as boot files, operating system images, and applications to client device 110. In addition, client device 110 can also include other types of client devices, such as tablet computer 116, mobile phone 118, smart glasses 120, and smart glasses 122. In this illustrative example, server computer 104, server computer 106, storage unit 108, and client device 110 are network devices connected to network 102, wherein network 102 is a communication medium for these network devices. Some or all of client devices 110 can form an Internet of Things (IoT), wherein these physical devices can be connected to network 102 and exchange information with each other through network 102.

[0034] In this example, client device 110 is a client to server computers 104 and 106. Network data processing system 100 may include additional server computers, client computers, and other devices not shown. Client device 110 connects to network 102 using at least one of a wired, fiber optic, or wireless connection.

[0035] Program code located in network data processing system 100 may be stored on a computer recordable storage medium and downloaded to the data processing system or other devices for use. For example, program code may be stored on a computer recordable storage medium on server computer 104 and downloaded to client device 110 via network 102 for use on client device 110.

[0036] In the depicted example, network data processing system 100 is the Internet, wherein network 102 represents a global collection of networks and gateways that communicate with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols. The core of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, which are composed of thousands of commercial, government, educational, and other computer systems that route data and messages. Of course, network data processing system 100 may also be implemented using multiple different types of networks. For example, network 102 may be composed of at least one of the Internet, an intranet, a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN). Figure 1 It is intended as an example, not as an architectural limitation for the different illustrative embodiments.

[0037] As used herein, "plurality" when used with reference to an item means one or more items. For example, "plurality of different types of networks" is one or more different types of networks.

[0038] In addition, the phrase "at least one" when used with a list of items means that different combinations of one or more of the listed items can be used, and only one of each item in the list may be required. In other words, "at least one" means that any combination of items and numbers of items can be used from the list, but not all items in the list are required. The item can be a specific object, thing, or category.

[0039] For example, but not limited to, "at least one of item A, item B, and item C" may include item A, item A and item B, or item B. This example may also include item A, item B, and item C, or item B and item C. Of course, there may be any combination of these items. In some illustrative examples, "at least one" may be, for example, but not limited to, two items A; one item B; and ten items C; four items B and seven items C; or other suitable combinations.

[0040] In this illustrative example, operator 124 operates smart glasses 120 and operator 126 operates smart glasses 122. In this example, operator 124 and operator 126 place layers of composite plies on a tool as part of a process of manufacturing a composite structure. The composite structure may be, for example, a composite part of an aircraft.

[0041] In this illustrative example, composite ply 130 has been laid up on tool 128. As shown, operator 124 and operator 126 are preparing to place another composite ply onto tool 128.

[0042] In this illustrative example, visualization of placement of new composite plies may be performed using smart glasses 120 operated by operator 124 and smart glasses 122 operated by operator 126 .

[0043] As shown, smart glasses 120 and smart glasses 122 operate to scan tool 128. In this illustrative example, the scanning may be performed by smart glasses 120 using a simultaneous localization and mapping (SLAM) process 132 and by smart glasses 122 using a simultaneous localization and mapping (SLAM) process 134. The scanning may include scanning tool 128 with a laser, lidar, infrared scanner, or other suitable device that generates data about the surface of tool 128.

[0044] In this illustrative example, smart glasses 120 and smart glasses 122 each have different viewpoints of tool 128. As a result, scanning tool 128 from these different viewpoints may result in a more accurate scan of tool 128.

[0045] For example, smart glasses 120 may not be able to obtain a view of some locations on tool 128. Smart glasses 122 can obtain a view of these locations that smart glasses 120 missed. In a similar manner, smart glasses 122 may not be able to obtain a view of some locations within the field of view of smart glasses 120.

[0046] Scan data 136 generated by smart glasses 120 from the scan of tool 128 and scan data 138 generated by smart glasses 122 from the scan of tool 128 are sent to visualizer 140 running on server computer 104. Scan data 136 and scan data 138 may be point clouds generated by the smart glasses. In other illustrative examples, scan data 136 and scan data 138 may be other information that may be used to generate point clouds or other information suitable for mapping tool 128.

[0047] In this illustrative example, visualizer 140 generates combined map 142 of tool 128 using scan data 136 and scan data 138 . Combined map 142 includes the placement of composite plies 130 .

[0048] At least one of a more complete tool map and an accurate tool map is generated by using scan data 136 and scan data 138 to generate combined map 142. For example, scan data 136 and scan data 138 may cover more locations on tool 128 (compared to using scan data 136 or scan data 138 only with respect to different viewpoints from which scan data 136 and scan data 138 were generated).

[0049] In this illustrative example, composite structure model 146 in model database 148 is accessed by visualizer 140 to obtain laying information 144 regarding the laying location on tool 128. In this illustrative example, visualizer 140 sends laying information 144 to smart glasses 120 and smart glasses 122.

[0050] In this illustrative example, smart glasses 120, 122 position themselves relative to tool 128 using a simultaneous localization and mapping (SLAM) process running on the smart glasses. Using the positions of smart glasses 120, 122, layup information 144 may be displayed to augment the real-time view seen through smart glasses 120, 122. For example, layup information 144 may be an outline identifying where on tool 128 a new composite ply should be laid up.

[0051] In these illustrative examples, as operator 124 and operator 126 move to perform operations to lay up composite plies of a composite part, smart glasses 120 and smart glasses 122 may continue to scan tool 128 and send scan data 136 and scan data 138. As the operators move relative to tool 128, the viewpoint of the smart glasses for tool 128 may change. As a result, the additional scan data sent from the smart glasses may provide additional information to improve the accuracy of combined map 142 for tool 128.

[0052] Furthermore, additional scans may also provide updates on changes to the layup of composite plies on tool 128. As composite plies are added or positioned, combined map 142 may be updated using the scan data from these operations.

[0053] In these illustrative examples, the generation of updates to combined map 142 is performed in real time. In other words, scan data may be continuously or periodically generated by smart glasses 120 and smart glasses 122 while operators 124 and 126 are performing operations to lay up composite plies on tool 128.

[0054] As a result, combined map 142 may be a dynamic map of tool 128 and any composite plies placed on tool 128. In this manner, the accuracy of combined map 142 may be improved and may include changes such as the placement of composite plies on tool 128.

[0055] Figure 1The description of this example in is not meant to limit the manner in which other illustrative examples may be implemented. For example, one or more operators with smart glasses may be present during the process of generating scan data for tool 128. In another illustrative example, different types of objects other than tool 128 may be present. For example, an operator may perform a rework operation to form a scarf and install a patch composed of composite material layers. With this type of operation, layup information 144 may include the placement of the patch. In addition, scarf information may also be presented to assist the operator in forming the scarf.

[0056] refer to Figure 2 , an illustration of a block diagram of an augmented reality environment is depicted in accordance with an illustrative embodiment. Figure 1 The different hardware components in network data processing system 100 are examples of components that may be used in augmented reality environment 200 .

[0057] As shown, augmented reality environment 200 is an environment in which task information 202 may be visualized by a plurality of operators 204 viewing objects 206. In this example, a plurality of operators 204 may perform operations 208 on objects 206. For example, objects 206 may be selected from the group consisting of a tool, a wall, a workpiece, a wing, a fuselage section, an engine, a building, an aircraft, a vehicle, or some other suitable type of object.

[0058] In this illustrative example, visualization of information 202 is performed using augmented reality system 210. As shown, augmented reality system 210 augments real-time view 212 of task location 244 of object 206. Real-time view 212 is a view of real-world environment 216 seen through portable computing device 214. In this illustrative example, real-time view 212 may be an image or video generated by a camera in the portable computing device and displayed in real time on a display device in the portable computing device. In other examples, an operator may directly see real-time view 212 through the portable computing device.

[0059] In this example, task information 202 can be displayed on real-time view 212 to augment real-time view 212 of real-world environment 216. Task information 202 can include at least one of a task location, an outline of a slice, text with instructions for performing an operation, an image, a graphic, a video, and other suitable types of information that can be overlaid on real-time view 212 of object 206.

[0060] As shown, the augmented reality system includes a computer system 218 and a portable computing device 214. The computer system 218 is a physical hardware system and includes one or more data processing systems. When there is more than one data processing system in the computer system 218, those data processing systems communicate with each other using a communication medium. The communication medium may be a network. The data processing system may be selected from at least one of a computer, a server computer, a tablet computer, and some other suitable data processing system.

[0061] Portable computing device 214 is a physical hardware device used by operator 204 to visualize task information 202 about object 206. Portable computing device 214 may be selected from at least one of a mobile phone, a tablet computer, smart glasses, a head mounted display, and some other suitable computing device. In this illustrative example, portable computing device 214 scans object 206 using at least one of a laser scanner, a structured light 3D scanner, an infrared light scanner, and some other suitable type of device that can create scan data 228.

[0062] In this illustrative example, augmented reality system enables one or more of operators 204 operating portable computing device 214 to visualize real-time view 212 of object 206 by augmenting real-time view 212 with task information 202 to perform task 224 on object 206 .

[0063] In the illustrative example, task 224 can have a variety of different forms. Task 224 can be selected from at least one of placing composite plies, applying damage, applying decals, performing a task location check, drilling holes, installing fasteners, connecting parts to assemblies, removing parts, surgical procedures, forming metal bonds, painting, taking measurements, and some other operation of task 224.

[0064] In this illustrative example, visualizer 226 in computer system 218 is in communication with portable computing device 214. These components communicate with each other using a wireless communication link in these illustrative examples.

[0065] In this illustrative example, scan data 228 is generated by scanning object 206 from different viewpoints 230 of object 206 via portable computing device 214 on operator 204. In this illustrative example, scan data 228 is generated and received in real time.

[0066] In the illustrative example, the viewpoint is a perspective view from a sensor in the portable computing device. For example, the sensor can be a camera or a scanner. The viewpoint can be a location including a location in three-dimensional space and an orientation of the portable computing device.

[0067] Visualizer 226 operates to receive scan data 228 from portable computing device 214. As shown, visualizer 226 creates point cloud 232 from scan data 228 generated by portable computing device 214. In this illustrative example, visualizer 206 may receive scan data 228 in real time. Visualizer 226 uses point cloud 232 to create combined map 234 of object 206.

[0068] In creating combined map 234, visualizer 226 creates a map from each point cloud in point cloud 232 to form multiple maps 236, and combines multiple maps 236 to form combined map 234. The combination of multiple maps 236 can be performed by identifying common features between point clouds in combining point clouds using those local feature correspondences. These common features can be at least one of features that appear on object 206, fiducial markers placed on object 206, or fiducial markers placed near object 206. The features on object 206 can include at least one of holes, fasteners, seams, elongated protrusions, corners, ends, and some other features on object 206. In another illustrative example, combined map 234 can be generated by combining point clouds 232 and then creating combined map 234.

[0069] In the illustrative example, combined map 234 and map 236 are three-dimensional maps. These maps represent the surface of object 206 as well as any other items or things that may be on or touching object 206.

[0070] In this illustrative example, portable computing device 238 in portable computing devices 214 is located to object 206. The location identifies the position of computing device 238 to object 206. The location of portable computing device 238 is the position and orientation of portable computing device 238 in three-dimensional space.

[0071] The positioning may be performed using a simultaneous localization and mapping process (SLAM) 240 running on portable computing device 238. Portable computing device 238 displays task information 202 of a task location 244 on real-time view 212 of object 206. In this illustrative example, visualization 242 of task location 244 may be performed using task information 202. For example, task information 202 may include coordinates of task location 244. Visualization 242 may be a graphical indicator that visually identifies task location 244 on real-time view 212 of object 206. Visualization 242 is displayed for performing task 224 on object 206. Additionally, visualization 242 may also include other information, such as guides 243, to aid in placing components such as parts, assemblies, composite plies, or other components of object 206.

[0072] For example, guidance 243 may take the form of composite ply outlines, composite ply patterns, hole outlines, or other suitable graphical information to guide operator 204 in performing operations 208 to perform task 224 on object 206 and task location 244 .

[0073] Additionally, guides 243 can be used as a visual indication of task location 244. For example, guides 243 can be an outline of a composite ply that is displayed at task location 244 where the composite ply should be placed. In other words, the composite ply can be placed to fit within the outline displayed on real-time view 212 of object 206.

[0074] A visualization is performed on real-time view 212 of object 206 to indicate the location of task location 244 using model 246 of object 206 and combined map 234 generated from a scan of object 206 performed by portable computing device 214. In this illustrative example, model 246 is a reference map that can be compared to combined map 234 generated from scan data 228. Model 246 and combined map 234 can be aligned with each other for use in generating at least one of coordinates, program code, instructions, and other information for displaying task information at a desired location on real-time view 212 of object 206. For example, if task information 202 includes the location of a composite ply, the two models can be used to display the outline of the composite ply from model 242 of object 206 at the correct location in the real-time view of object 206.

[0075] In the illustrative example, task 224 to be performed at task location 244 can have a variety of different forms. For example, task location 244 is for at least one of a composite ply, a component in an assembly, where the object is a component, a patch, a decal, or some other suitable line. Task 224 can be selected from at least one of placing a composite ply, applying a damage, applying a decal, performing a task location inspection, drilling a hole, installing a fastener, connecting a component to an assembly, removing a component, and some operations of task 224.

[0076] In an illustrative example, the visualizer 226 can be implemented in software, hardware, firmware, or a combination thereof. When software is used, the operations performed by the visualizer 226 can be implemented in program code configured to run on hardware such as a processor unit. When firmware is used, the operations performed by the visualizer 226 can be implemented in program code and data stored in a permanent memory to run on a processor unit. When hardware is employed, the hardware may include circuits for performing the operations in the visualizer 226.

[0077] In the illustrative examples, the hardware may take the form of at least one selected from a circuit system, an integrated circuit, an application specific integrated circuit (ASIC), a programmable logic device, and some other suitable type of hardware configured to perform multiple operations. Through a programmable logic device, the device can be configured to perform multiple operations. The device can be reconfigured at a later time, or can be permanently configured to perform multiple operations. Programmable logic devices include, for example, programmable logic arrays, programmable array logic, field programmable logic arrays, field programmable gate arrays, and other suitable hardware devices. In addition, the process can be implemented in an organic component integrated with an inorganic component, and can be composed entirely of organic components other than humans. For example, the process can be implemented as a circuit in an organic semiconductor.

[0078] Next reference Figure 3 , depicts an illustration of a block diagram of an augmented reality system for visualizing task information to place composite plies at layup locations on a tool, according to an illustrative embodiment. In the illustrative examples, the same reference numeral may be used in more than one figure. Reference numerals are repeated in different figures to represent the same elements in the different figures.

[0079] In this illustrative example, augmented reality system 210 is used in augmented reality environment 300 to visualize task information 302 of layup locations 304 on tool 306. In this particular example, tool 306 is a structure on which composite plies 308 may be laid up as part of a process for manufacturing composite part 310. Tool 306 may, for example, be selected from the group consisting of a mandrel, a mold, a composite tool, and other suitable types of tools.

[0080] In this example, visualizer 226 in computer system 218 operates to receive scan data 314 from portable computing device 214 on operator 204 at different viewpoints 312 of tool 306 to generate scan data 314. In addition, visualizer 226 creates combined map 316 of tool 306 using scan data 314 obtained from portable computing device 214 at different viewpoints 312 of tool 306.

[0081] In one illustrative example, visualizer 226 creates point cloud 318 of tool 306 from scan data 314. In creating point cloud 318 of tool 306, point cloud 318 may also include any composite plies, release films, or other components that have been placed on the tool. Each point cloud in point cloud 318 is generated from scan data 314 received by one of portable computing devices 214. Visualizer 226 creates multiple maps 320 of tool 306 using scan data 314 generated by portable computing devices 214, and combines multiple maps 320 to form combined map 316 of tool 306.

[0082] In this example, visualizer 226 identifies common reference points in plurality of maps 320 and combines plurality of maps 320 using the common reference points to form combined map 316. These reference points may be for at least one of the features of tool 306 or may be fiducial markers used with tool 306.

[0083] In this example, combined map 316 has increased accuracy from multiple maps 320 created from scan data 314 generated from different viewpoints 312. For example, if only a single map was used, the scan data for the single map may be missing data for portions of tool 306 from a scan performed from the viewpoint of a portable computing device. For example, scan data may be missing for portions of object 206 that are not visible to the portable computing device.

[0084] By combining scan data from different viewpoints, there is a more complete data set for generating a map of tool 306. As a result, there is improved accuracy.

[0085] In another illustrative example, point clouds 318 may be combined to form combined point cloud 322 . Visualizer 206 may then generate combined map 316 from combined point cloud 322 .

[0086] As shown, visualizer 226 identifies task information 302 for layup locations 304 in ply model 324 for composite plies 308 of composite part 310. Task information 302 may include information selected from at least one of a file identification, a ply sequence, a ply orientation, placement information, a curing temperature, a curing time, and other suitable information for manufacturing composite part 310 on tool 306.

[0087] Visualizer 226 sends task information 302 for pavement location 304 on tool 306 to portable computing device 238 in portable computing device 214. In this illustrative example, portable computing device 238 displays task information 302 for pavement location 304 on tool 306 on real-time view 326 of tool 306 as seen by display device 328 in portable computing device 238, which has been positioned using combined map 316 of tool 306 and ply model 324 of composite plies 308. The displayed task information enhances real-time view 326 of tool 306.

[0088] As shown, portable computing device 238 identifies layup locations 304 in ply model 324 of composite plies 308 of composite part 310. This determination may be made in a number of different ways. For example, portable computing device 238 may include a local copy of ply model 324. In another illustrative example, mobile computing device 238 may access ply model 324 by sending a request to visualizer 226.

[0089] As shown, portable computing device 214 uses ply model 324 and combined map 316 to determine locations on real-time view 326 of plurality of guides 330 for plurality of composite plies 308. Portable computing device 214 displays plurality of guides 330 for plurality of composite plies 308 at the locations on real-time view 326 of tool 306 as seen through display device 328 in portable computing device 238 that has been positioned to tool 306. As shown, plurality of guides 330 assist in placing plurality of composite plies 308 on tool 306. The guides may assist in at least one of laying up the composite plies in the correct location for manufacturing a composite component on tool 306 and reworking composite component 310. In this illustrative example, plurality of guides 330 may be at least one of an outline, a pattern, and some other suitable visual display that guides the placement of composite plies on tool 306.

[0090] In another illustrative example, number of guides 330 may include number of additional outlines or patterns of number of composite plies 308 at layup locations 304 on real-time view 326 of tool 306 as seen via display device 328 in portable computing device 238 that has been positioned.

[0091] A plurality of additional contours or patterns illustrate a plurality of previous placements of a plurality of composite plies 308 on tool 306. These previous placements may be placements previously made on the same tool or on the same tool for the same composite part. In this manner, a comparison of composite ply arrangements may be made.

[0092] Additionally, portable computing device 238 may display other task information in task information 302 in addition to guidance 330 at layup location 304 as part of an augmented reality display, where the other task information is displayed on real-time view 326 of tool 306. For example, portable computing device 238 may display at least one of plies, instructions, images, and videos for placing the plurality of composite plies.

[0093] In one illustrative example, there are one or more technical solutions that overcome the technical problem of providing guidance for placing plies on a tool. In an illustrative example, one or more technical solutions use a portable computing device instead of an overhead laser tracker to provide visualization of a task location. In an illustrative example, one or more technical solutions use scans of a tool from multiple portable computing devices at different viewpoints to improve the accuracy of the visualization of the task location. One or more technical solutions involve combining scan data received from portable computing devices to form a combined map of an object such as a tool, such that scan data from one portable computing device that is missing scan data for a portion of the tool can be supplemented by scan data from another portable computing device that includes the tool portion.

[0094] As a result, one or more technical solutions may provide a technical effect of providing increased accuracy to visualization of task locations on an object by using scan data from multiple portable computing devices as compared to currently used techniques.

[0095] Computer system 218 may be configured to perform at least one of the steps, operations, and actions described in the different illustrative examples using software, hardware, firmware, or a combination thereof. As a result, computer system 218 operates as a special-purpose computer system, wherein visualizer 226 in computer system 218 is capable of visualizing task information 302 on real-time view 212 of object 206 in a manner that provides an augmented reality display, wherein task information 202 is overlaid on real-time view 212 of object 206. Specifically, visualizer 226 converts computer system 216 into a special-purpose computer system, as compared to currently available general-purpose computer systems that do not have visualizer 226.

[0096] Figure 2 and Figure 3 The illustration of the augmented reality system 210 in FIG. 1 is not meant to imply physical or architectural limitations on the manner in which the illustrative embodiments may be implemented. Other components may be used in addition to or in place of the components shown. Some components may be unnecessary. In addition, boxes are present to illustrate some functional components. When implemented in the illustrative embodiments, one or more of these boxes may be combined, divided, or combined and divided into different boxes.

[0097] For example, simultaneous positioning and mapping process 240 may be executed on computer system 218 instead of portable computing device 238 .

[0098] Go to Figure 4, depicts an illustration of operators performing operations at a tool according to an illustrative embodiment. In this illustrative example, operator 400 and operator 402 perform operations to lay up composite plies on tool 404 as part of a process for manufacturing a composite part. In yet another illustrative example, one or more tasks may be performed in addition to task 224. For example, task 224 may be placing composite plies. A subsequent task may be inspecting the composite plies to determine if they were placed correctly.

[0099] As shown, operator 400 wears smart glasses 406, and operator 402 wears smart glasses 408. Smart glasses 406 and smart glasses 408 are Figure 2 and Figure 3 An example of an implementation of a portable computing device 214 is shown in block form in FIG.

[0100] As shown, smart glasses 406 and smart glasses 408 are at different viewpoints relative to tool 404. In this example, smart glasses 406 have viewpoint 410 and smart glasses 408 have viewpoint 412. These viewpoints are directed from different locations toward tool 404. In other words, viewpoint 410 and viewpoint 412 are different viewpoints from one another.

[0101] In this illustrative example, smart glasses 406 scan tool 404 through view 414. In this illustrative example, the scanning is performed using at least one of a laser scanner and an infrared scanner or some other suitable type of sensor. For example, smart glasses 408 may use a laser scanner, while smart glasses 406 may use an infrared scanner. As another example, smart glasses 406 and smart glasses 406 may each have both a laser scanner and an infrared scanner.

[0102] View 414 is a three-dimensional space that can be scanned by smart glasses 406. View 416 is a three-dimensional space that can be scanned by smart glasses 408. The views can also define a real-time view seen through the operation of the smart glasses. For example, view 414 can be a real-time view seen by operator 400 through smart glasses 406. View 416 can be a real-time view seen by operator 402 through smart glasses 408. In other illustrative examples, the view used for scanning and the view used for viewing the real-time view may be different, but from the same viewpoint. A view is a portion of augmented reality environment 300 that can be scanned or seen by a sensor in a portable computing device.

[0103] Scanning these views from different viewpoints results in two sets of scans with different scan data. The scan data can be used to generate a combined map that combines the data from both scans. This combined map is more accurate than using only one scan from the smart glasses.

[0104] For example, smart glasses 406 and smart glasses 408 may scan tool 404, with some locations visible only in view 414 of smart glasses 406 and some locations visible only to view 416 of smart glasses 408. By scanning, some locations are visible in the views of both smart glasses.

[0105] For example, in view 414 for smart glasses 406, location 420, location 422, location 424, location 426 may be scanned by smart glasses 406 and in the real-time view of smart glasses 406. However, these locations are not within view 416 of smart glasses 408.

[0106] As shown, in the view 416 of the smart glasses 408, the location 430 and the location 432 can be scanned and visible in the real-time view of the smart glasses 408. These locations are not within the view 414 of the smart glasses 406.

[0107] In this illustrative example, location 440 , location 442 , location 444 , location 446 , location 448 , location 450 , location 452 , location 454 , patient 456 , and location 458 are locations that may be scanned from view 414 of smart glasses 406 and view 416 of smart glasses 408 .

[0108] As a result, the scan data generated by each of the smart glasses includes common locations scanned by both smart glasses and locations that are reserved by only one of the smart glasses. By using the scan data from the two smart glasses from different viewpoints, a more accurate map of the tool 404 can be generated for laying up composite plies of a composite part.

[0109] In this illustrative example, smart glasses 406 and smart glasses 408 use processes running on these smart glasses to locate themselves. These processes may include, for example, currently available simultaneous positioning and mapping processes. In other words, smart glasses 406 and smart glasses 408 can identify their positions relative to tool 404. In this illustrative example, the positions of smart glasses 406 and smart glasses 408 are in three-dimensional space and include orientation.

[0110] Additionally, the smart glasses may identify corresponding positions relative to the ply model of the composite component.The displayed model may also include a model of the tool 404 and the layup of plies on the tool 404 that form the composite component.

[0111] In this illustrative example, pattern 460 is displayed on the real-time view seen by both smart glasses 406 and smart glasses 408 to provide a visualization of the placement of the composite layers on tool 404. Using the scan data generated by both smart glasses, criteria may be combined to more accurately index or locate pattern 460 on tool 404.

[0112] Furthermore, as operator 400 and operator 402 move, additional scans of tool 404 may be taken. These additional scans may be used with the already generated scans of tool 404 to improve the accuracy of the map of tool 404. Furthermore, the scans may also include any composite plies already placed on tool 404. In addition to providing guidance for additional plies to be placed on tool 404 on top of the composite plies already placed on tool 404, these scans may also verify the accuracy of the plies already placed on the tool.

[0113] Next go to Figure 5 , an illustration of a flowchart of a process for enhancing a real-time view of task locations on a portable computing device is depicted in accordance with an illustrative embodiment. Figure 5 The processes in may be implemented in hardware, software, or both. When implemented in software, the processes may take the form of program code that is executed by one or more processor units in one or more hardware devices in one or more computer systems. For example, the processes may be implemented in Figure 1 Smart glasses in 120, Figure 1 Smart glasses 122, including Figure 2 The portable computing device 214 of the portable computing device 238 in the augmented reality system 210, Figure 4 Smart glasses 406 or Figure 4 The method is implemented in at least one of the smart glasses 408.

[0114] The process begins by positioning a portable computing device to an object (operation 500). The process displays a visualization of a task location on a real-time view of the object for performing the task using a model of the object and a combined map of the object (operation 502). The combined map is generated by scanning the object with different viewpoints to the object by the portable computing device. The process terminates thereafter.

[0115] Operation 502 may be performed in a variety of different ways. For example, models of objects in a combined map of objects may be downloaded from a computer system to a portable computing device. The portable computing device may then use these models to display a visualization of the task location. For example, the visualization may be an outline of a ply in place on a tool. In another illustrative example, a computer system may generate the outline and identify placement on a real-time view of the portable computing device. The computer system may then send the outline to the portable computing device along with at least one of program code, instructions, or other information required to display the outline on a real-time view seen by the portable computing device.

[0116] refer to Figure 6 , an illustration of a flowchart of a process for processing scan data is depicted in accordance with an illustrative embodiment. Figure 6 The processes in may be implemented in hardware, software, or both. When implemented in software, the processes may take the form of program code that is executed by one or more processor units in one or more hardware devices in one or more computer systems. For example, the processes may be implemented in Figure 1 The visualizer 140 running on the server computer 104 in Figure 2 The system may be implemented in at least one of the visualizers 226 in the computer system 218 in the embodiment of the present invention.

[0117] The process begins by receiving scan data from a portable computing device (operation 600). The scan data is generated by scanning an object at different viewpoints of the object using a portable computing device on an operator.

[0118] The process creates a combined map of the object using the scan data (operation 602). The process sends a visualization of the task location to a portable computing device for performing the task at the task location (operation 604). The process terminates thereafter.

[0119] Next reference Figure 7 , an illustration of a flowchart of a process for creating a combined map is depicted in accordance with an illustrative embodiment. Figure 7 The processes in may be implemented in hardware, software, or both. When implemented in software, the processes may take the form of program code that is executed by one or more processor units in one or more hardware devices in one or more computer systems. For example, the processes may be implemented in Figure 1 The visualizer 140 running on the server computer 104 in Figure 2 The system may be implemented in at least one of the visualizers 226 in the computer system 218 in the embodiment of the present invention.

[0120] The process begins by creating a map from each of a plurality of point clouds to form a plurality of maps (operation 700). The process combines the plurality of maps to form a combined map of tools that identify common reference points in the plurality of maps (operation 702). The process terminates thereafter.

[0121] In operation 702, maps are combined using common reference points. These common reference points may be selected for features present in each of the plurality of maps. Combining maps improves accuracy in maps created from scan data generated from different viewpoints.

[0122] Next reference Figure 8 , an illustration of a flowchart of a process for creating a combined map is depicted in accordance with an illustrative embodiment. Figure 8 The processes in may be implemented in hardware, software, or both. When implemented in software, the processes may take the form of program code that is executed by one or more processor units in one or more hardware devices in one or more computer systems. For example, the processes may be implemented in Figure 1 The visualizer 140 running on the server computer 104 in Figure 2 The system may be implemented in at least one of the visualizers 226 in the computer system 218 in the embodiment of the present invention.

[0123] The process begins by identifying corresponding points in a point cloud for the same corresponding feature on an object (operation 800). In operation 800, the number of corresponding points for the same corresponding feature may be, for example, 3 or 4 points. The process merges the point clouds using the corresponding points to form a combined point cloud (operation 802). The process creates a combined map using the combined point cloud (operation 804). Thereafter the process terminates.

[0124] exist Fig. 9 In the figure, an illustration of a flowchart of a process for visualizing task information for laydown locations on a tool is depicted in accordance with an illustrative embodiment. Fig. 9 The processes in may be implemented in hardware, software, or both. When implemented in software, the processes may take the form of program code that is executed by one or more processor units in one or more hardware devices in one or more computer systems. For example, the processes may be implemented in Figure 3 The operation can be implemented in the augmented reality system 210 in FIG. Figure 3 306. The system 306 may be implemented in at least one of the computer system 218 and the portable computing device 214. Operations may be implemented to provide visualization of the task information 302 on the real-time view 326 of the tool 306. This type of display enhances the real-time view 326 to provide an augmented reality display to the operator.

[0125] The process begins by using a portable computing device on an operator to scan a tool from different viewpoints of the tool to generate scan data (operation 900). The process creates a point cloud from the scan data generated by the portable computing device (operation 902). The process creates a combined map of the tool using the point cloud (operation 904).

[0126] The process locates a portable computing device from the plurality of portable computing devices to the tool using the combined map of the tool (operation 906). The process displays task information of the layup location on the tool on a real-time view viewed through a display device in the portable computing device that has been located using the combined map of the tool and the ply model of the composite ply (operation 908). The process terminates thereafter.

[0127] Next reference Fig.10 , an illustration of a flowchart of a process for displaying a visualization of task information is depicted in accordance with an illustrative embodiment. Fig.10 The process shown in Figure 5 Operation 502 and Fig. 9 An example of one implementation of operation 908 in .

[0128] The process begins by displaying guidelines for a ply on a real-time view of the tool as seen through a portable computing device (operation 1000). The guidelines displayed in operation 1000 are used to place the next ply on the tool.

[0129] The process displays a plurality of additional guides on a real-time view of the tool as seen through the portable computing device 1 (operation 1002). The plurality of additional guides may be displayed in a manner that distinguishes the plurality of additional guides from guides for the plies. In this example, the plurality of additional guides may be for composite plies that have been placed in the tool. In this manner, the alignment of previous composite plies may be identified and visualized in an augmented reality display of the tool. The augmented reality display may be used to determine whether previous composite plies have shifted during layup of composite plies on the tool.

[0130] In addition, a plurality of additional guides may also provide a visualization of how the same ply was previously placed on the tool prior to proceeding with the operation to form the composite part. The placement of the previous composite ply may be obtained from a database containing the composite ply placement history. The composite ply placement history may be identified from scans performed during the placement of the composite ply. In this manner, identification of changes between the current placement and the previous placement may be visualized. These changes may occur from at least one of changes in ply size, slippage, changes in tooling, and other potential sources that may change the alignment of the same ply over time.

[0131] The flow charts and block diagrams in different depicted embodiments show the architecture, functions and operations of some possible implementations of the devices and methods in the illustrative embodiments. In this regard, each box in the flow chart or block diagram can represent at least one of a module, a segment, a function or a part of an operation or step. For example, one or more of the boxes can be implemented as a combination of program code, hardware or program code and hardware. When implemented in hardware, the hardware can, for example, take the form of an integrated circuit, which is manufactured or configured to perform one or more operations in the flow chart or block diagram. When implemented as a combination of program code and hardware, the implementation can take the form of firmware. Each box in the flow chart or block diagram can be implemented using a dedicated hardware system, which performs different operations or combinations of program codes run by dedicated hardware and dedicated hardware.

[0132] In some alternative implementations of the illustrative embodiments, one or more functions mentioned in the blocks may not occur in the order shown in the figure. For example, in some cases, two blocks shown in succession may be performed substantially simultaneously, or the blocks may sometimes be performed in reverse order, depending on the functions involved. In addition, other blocks may be added in addition to the blocks shown in the flowchart or block diagram.

[0133] Now go to Fig.11 , depicts an illustration of a block diagram of a data processing system according to an illustrative embodiment. Data processing system 1100 may be used to implement Figure 1 The data processing system 1100 can also be used to implement the server computer 104, the server computer 106, and the client device 110. Figure 2 and Figure 3 1104, memory 1106, persistent storage 1108, communications unit 1110, input / output (I / O) unit 1112, and display 1114. In this example, communications framework 1102 takes the form of a bus system.

[0134] The processor unit 1104 is used to execute instructions of the software that may be loaded into the memory 1106. The processor unit 1104 includes one or more processors. For example, the processor unit 1104 may be selected from at least one of a multi-core processor, a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a network processor, and some other suitable types of processors.

[0135] Memory 1106 and permanent storage 1108 are examples of storage 1116. A storage device is any hardware capable of storing information, such as, but not limited to, data, program code in functional form, or at least one of other suitable information of a temporary, permanent, or both nature. In these illustrative examples, storage 1116 may also be referred to as a computer-readable storage device. In these examples, memory 1106 may be, for example, a random access memory or any other suitable volatile or non-volatile storage device. Permanent storage 1108 may take various forms, depending on the particular implementation.

[0136] For example, permanent storage 1108 may include one or more components or devices. For example, permanent storage 1108 may be a hard drive, a solid-state drive (SSD), a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination thereof. The media used by permanent storage 1108 may also be removable. For example, a removable hard drive may be used for permanent storage 1108.

[0137] Communications unit 1110 , in these illustrative examples, provides for communications with other data processing systems or devices. Communications unit 1110 , in these illustrative examples, is a network interface card.

[0138] Input / output unit 1112 allows input and output of data with other devices that may be connected to data processing system 1100. For example, input / output unit 1112 may provide a connection for user input through at least one of a keyboard, a mouse, and some other suitable input device. Additionally, input / output unit 1112 may send output to a printer. Display 1114 provides a mechanism for displaying information to a user.

[0139] Instructions for at least one of the operating system, applications, and programs may be located in storage devices 1116, which are in communication with processor unit 1104 through communications framework 1102. The processes of the different embodiments may be performed by processor unit 1104 using computer-implemented instructions, which may be located in a memory, such as memory 1106.

[0140] These instructions are referred to as program code, computer usable program code, or computer readable program code, which can be read and executed by a processor in processor unit 1104. The program code in different embodiments may be embodied on different physical or computer readable storage media, such as memory 1106 or permanent storage device 1108.

[0141] Program code 1118 is located in functional form on a selectively removable computer readable medium 1120 and can be loaded or transferred to data processing system 1100 for execution by processor unit 1104. Program code 1118 and computer readable medium 1120 form a computer program product 1122 in these illustrative embodiments. In this illustrative example, computer readable medium 1120 is computer readable storage medium 1124.

[0142] In these illustrative examples, computer readable storage media 1124 is a physical or tangible storage device used to store program code 1118 rather than a medium that propagates or transmits program code 1118 .

[0143] Optionally, program code 1118 may be transferred to data processing system 1100 using a computer readable signal medium. Computer readable signal media may be, for example, a propagated data signal containing program code 1118. For example, a computer readable signal medium may be at least one of an electromagnetic signal, an optical signal, or any other suitable type of signal. These signals may be transmitted via a connection, such as a wireless connection, an optical cable, a coaxial cable, an electrical wire, or any other suitable type of connection.

[0144] The different components illustrated for data processing system 1100 are not meant to provide architectural limitations to the manner in which different embodiments may be implemented. In some illustrative examples, one or more of the components may be incorporated into or otherwise form part of another component. For example, in some illustrative examples, 1106 or portions thereof may be incorporated into processor unit 1104. The different illustrative embodiments may be implemented in a data processing system that includes components in addition to or in place of those illustrated for data processing system 1100. Fig.11 Other components shown in the can be varied from the illustrative examples shown. Various embodiments can be implemented using any hardware device or system capable of running program code 1118.

[0145] refer to Fig.12 , depicts an illustration of a block diagram of a portable computing device, according to an illustrative embodiment. Portable computing device 1200 is an example of one manner in which smart glasses 120, smart glasses 122, portable computing device 214, smart glasses 406, and smart glasses 408 may be implemented. In this illustrative example, portable computing device 1200 includes physical hardware components such as a processor unit 1202, a communication framework 1204, a memory 1206, a data storage device 1208, a communication unit 1210, a display 1212, and a sensor system 1214.

[0146] Communications framework 1204 allows different components in portable computing device 1200 to communicate with each other when connected to communications framework 1204. Communications framework 1204 is a bus system in this illustrative example.

[0147] Processor unit 1202 processes program code of software loaded into memory 1206. Processor unit 1202 includes one or more processors. For example, processor unit 1202 may be selected from at least one of a multi-core processor, a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a network processor, and some other suitable types of processors.

[0148] Memory 1206 is coupled to processor unit 1202 via communications framework 1204. As shown, memory 1206 may include at least one of a random access memory (RAM), a read only memory (ROM), a static random access memory (SRAM), a dynamic random access memory (DRAM), and other suitable types of memory devices or circuits.

[0149] As shown, data storage 1208 is connected to communication framework 1204 and can store data, program code, or other information. Instructions in the program code can be loaded from data storage 1208 into memory 1206 for processing by processor unit 1202. For example, the instructions in the program code can include a simultaneous localization and mapping (SLAM) process 1203 and an augmented reality application 1205 for displaying task information on a real-time view of an object.

[0150] Data storage device 1208 may include at least one of a hard drive, a flash drive, a solid state drive, an optical drive, and some other suitable type of data storage device or system. Data storage device 1208 may store scan data, a map of an object, a model of an object, or other suitable information for use in an augmented reality display of mission information overlaid on a real-time view of an object.

[0151] In this illustrative example, communications unit 1210 provides for communications with other data processing systems or devices. In these illustrative examples, communications unit 1210 includes at least one of a network interface card, a wireless communication device, a universal serial bus port, and other suitable devices.

[0152] Display 1212 is connected to communication framework 1204 and provides a mechanism for displaying information to a user. In this example, display 1212 can be a touch screen display, which enables receiving user input through the display.

[0153] In this illustrative example, sensor system 1214 is connected to communication framework 1204. As shown, sensor system 1214 may include hardware, software, or both. In this illustrative example, sensor system 1214 may include at least one of a laser scanner, a structured light 3D scanner, a camera, and an infrared light scanner.

[0154] The illustration of portable computing device 1200 is an example of one manner in which portable computing device 1200 may be implemented. This illustration is not meant to limit the manner in which portable computing device 1200 may be embodied in other illustrative examples. For example, portable computing device 1200 may also include an audio interface, wherein an audio output device generates sound.

[0155] You can Fig.13 Aircraft manufacturing and service method 1300 as shown in Fig.14 Illustrative embodiments of the present disclosure are described in the context of aircraft 1400 shown in FIG. Fig.13 , a description of an aircraft manufacturing and service method is depicted according to an illustrative embodiment. During pre-production, aircraft manufacturing and service method 1300 may include Fig.14 Specification and design 1302 of the aircraft 1400 and material procurement 1304 .

[0156] During production, Fig.14 Components and subassemblies of the aircraft 1400 are manufactured 1306 and system integrated 1308. Thereafter, Fig.14 The aircraft 1400 in may be certified and delivered 1310 for entry into service 1312. In entry into service 1312 by a customer, Fig.14 Aircraft 1400 is scheduled for routine maintenance and service 1314, which may include modification, reconfiguration, refurbishment, and other maintenance or service.

[0157] Each of the processes of aircraft manufacturing and service method 1300 can be performed or carried out by a system integrator, a third party, an operator, or some combination thereof. In these examples, the operator can be a customer. For the purposes of this description, a system integrator can include, but is not limited to, any number of aircraft manufacturers and major system subcontractors; a third party can include, but is not limited to, any number of vendors, subcontractors, and suppliers; an operator can be an airline, a leasing company, a military entity, a service organization, etc.

[0158] Reference now Fig.14 , depicts a diagram of an aircraft in which an illustrative embodiment may be implemented. In this example, aircraft 1400 is comprised of Fig.1314. Aircraft manufacturing and service method 1300 may be produced in the context of FIG. 14 and may include fuselage 1402 having multiple systems 1404 and interior cabin 1406. Examples of systems 1404 include one or more of propulsion system 1408, electrical system 1410, hydraulic system 1412, and environmental system 1414. Any number of other systems may be included. Although an aerospace example is shown, different illustrative embodiments may be applied to other industries, such as the automotive industry.

[0159] Can be Fig.13 Apparatuses and methods embodied herein are employed during at least one of the stages of aircraft manufacturing and service method 1300 .

[0160] In one illustrative example, Fig.13 The components or subassemblies produced in component and subassembly manufacturing 1306 may be manufactured in a manner similar to that when aircraft 1400 is manufactured in Fig.13 As another example, one or more apparatus embodiments, method embodiments, or combinations thereof may be utilized during a production phase, such as Fig.13 Components and subassemblies are manufactured 1306 and integrated 1308. When the aircraft 1400 is put into service 1312, Fig.13 During maintenance and service 1314, or both, one or more device implementations, method implementations, or a combination thereof may be utilized.

[0161] For example, augmented reality system 210 may be used to provide visualizations of task locations. These visualizations may include displaying task information to be performed at the task location. Operators may utilize augmented reality system 210 during at least one of component and subassembly manufacturing 1306, system integration 1308, certification and delivery 1310, and maintenance and service 1314.

[0162] Use of a number of different illustrative embodiments may significantly speed up the assembly of aircraft 1400, reduce the cost of aircraft 1400, or speed up the assembly of aircraft 1400 and reduce the cost of aircraft 1400. For example, use of augmented reality system 210 during various steps such as component and subassembly manufacturing 1306, system integration 1308, or maintenance and service 1314 may improve the accuracy with which operators perform operations.

[0163] Now go to Fig.15 , an illustration of a block diagram of a product management system is depicted in accordance with an illustrative embodiment. Product management system 1500 is a physical hardware system. In this illustrative example, product management system 1500 may include at least one of manufacturing system 1502 and maintenance system 1504.

[0164] Manufacturing system 1502 is configured to manufacture products, such as Fig.14 As shown, manufacturing system 1502 includes manufacturing equipment 1506. Manufacturing equipment 1506 includes at least one of fabrication equipment 1508 and assembly equipment 1510.

[0165] Fabrication equipment 1508 is a device that can be used to fabricate Fig.14 Fabrication equipment 1508 may include, for example, machines and tools. The machines and tools may be at least one of a drill, a hydraulic press, a furnace, a mold, a composite tape laying machine, a vacuum system, a lathe, and other suitable types of equipment. Fabrication equipment 1508 may be used to fabricate at least one of a metal part, a composite part, a semiconductor, a circuit, a fastener, a rib, a skin panel, a wing spar, an antenna, and other suitable types of parts.

[0166] Assembly equipment 1510 is used to assemble components to form Fig.14 Specifically, assembly equipment 1510 may be used to assemble components and parts to form Fig.14 The assembly equipment 1510 may also include machines and tools. These machines and tools may be at least one of a robotic arm, a crawler traction device, a faster installation system, a track-based drilling system, and a robot. The assembly equipment 1510 may be used to assemble components such as seats, horizontal tails, wings, engines, engine housings, landing gear systems, and Fig.14 Other components of aircraft 1400.

[0167] In this illustrative example, maintenance system 1504 includes maintenance equipment 1512. Maintenance equipment 1512 may include Fig.14 Maintenance equipment 1512 may include any equipment required to perform maintenance on aircraft 1400. Fig.14 The tools are used to perform various operations on parts on aircraft 1400 in the aircraft. These operations may include removing parts, refurbishing parts, inspecting parts, reworking parts, replacing parts during manufacturing, and Fig.14 At least one of other operations for performing maintenance on aircraft 1400. These operations may be for routine maintenance, inspection, upgrade, refurbishment, or other types of maintenance operations.

[0168] In the illustrative example, maintenance equipment 1512 may include ultrasonic inspection equipment, x-ray imaging systems, vision systems, drills, crawler traction devices, and other suitable devices. In some cases, maintenance equipment 1512 may include fabrication equipment 1508, assembly equipment 1510, or both to produce and assemble parts that may be needed for maintenance.

[0169] The product management system 1500 also includes a control system 1514. The control system 1514 is a hardware system and may also include software or other types of components. The control system 1514 is configured to control the operation of at least one of the manufacturing system 1502 and the maintenance system 1504. Specifically, the control system 1514 may control the operation of at least one of the production equipment 1508, the assembly equipment 1510, and the maintenance equipment 1512.

[0170] The hardware in control system 1514 may be implemented using devices that may include computers, circuits, networks, and other types of devices. Control may take the form of directly controlling manufacturing equipment 1506. For example, robots, computer-controlled machines, and other devices may be controlled by control system 1514. In other illustrative examples, control system 1514 may manage operations performed by operators 1516 in manufacturing or performing maintenance on aircraft 1400. For example, control system 1514 may assign tasks, provide instructions, display models, or perform other operations to manage operations performed by operators 1516.

[0171] In these illustrative examples, Figure 2 and Figure 3 The augmented reality system 210 in the embodiment may be implemented to be used with the control system 1514 to manage Fig.14 At least one of manufacturing and maintenance of aircraft 1400 in the augmented reality system 210. For example, augmented reality system 210 can be operated to provide instructions and guidance to operators 1516 to perform operations on the object. These operations can include manufacturing operations or maintenance operations of the object. For example, control system 1514 can assign tasks such as laying composite plies on a tool to one or more of operators 1516. Control system 1514 can send task information to enhance the real-time view of portable computing device 214 in augmented reality system 210 worn or carried by operator 1516.

[0172] In different illustrative examples, operator 1516 may operate or interact with at least one of manufacturing equipment 1506, maintenance equipment 1512, and control system 1514. The interaction may be performed to manufacture Fig.14 Aircraft 1400 in.

[0173] Of course, product management system 1500 can be configured to manage Fig.14 Although product management system 1500 has been described with respect to manufacturing in the aerospace industry, product management system 1500 may be configured to manage products in other industries. For example, product management system 1500 may be configured to manufacture products in the automotive industry as well as any other suitable industry.

[0174] Thus, illustrative embodiments provide methods, apparatus, and systems for visualizing a task location. Visualization of the task location includes displaying information for performing operations at the task location in addition to identifying the task location. In an illustrative example, there are one or more technical solutions that overcome the technical problem of providing guidance for placing a ply on a tool. In an illustrative example, one or more technical solutions use a portable computing device instead of an overhead laser tracker to provide visualization of the task location.

[0175] In an illustrative example, one or more technical solutions use scans of a tool from multiple portable computing devices at different viewpoints to improve the accuracy of visualization of a task location. Thus, one or more technical solutions in the illustrative example involve combining scan data received from portable computing devices to form a combined map of an object such as a tool, such that scan data from one portable computing device that is missing scan data of a portion of the tool may be supplemented by scan data from another portable computing device that includes the portion of the tool.

[0176] As a result, by using scan data from multiple portable computing devices, one or more technical solutions may provide a technical effect of providing a visualization of task locations on an object compared to currently used techniques.

[0177] The description of different illustrative embodiments has been given for the purpose of illustration and description, and is not intended to be exhaustive or limited to the embodiments of the disclosed forms. Different illustrative examples describe components that perform actions or operations. In the illustrative embodiments, the components can be configured to perform the described actions or operations. For example, the components can have a configuration or design for a structure that provides the components with the ability to perform the actions or operations described as being performed by the components in the illustrative examples.

[0178] Furthermore, the present disclosure includes embodiments according to the following items:

[0179] Item 1. A method for visualizing task information (302) of a laying location (304) on a tool (306), the method comprising:

[0180] Scanning the tool (306) from different viewpoints (312) of the tool (306) using a portable computing device (214) on the operator (204) to generate scan data (314);

[0181] creating, by a computer system (218), a point cloud (318) from scan data (314) generated by a portable computing device (214);

[0182] creating, by the computer system (218), a combined map (316) of the tool (306) using the point cloud (318);

[0183] locating a portable computing device (238) of the portable computing devices (214) to the tool (306) using the combined map (316) of the tool (306); and

[0184] Task information (302) of a laydown location (304) on a tool (306) is displayed by the one portable computing device (238) on a real-time view (326) seen through a display device (328) in the portable computing device (238), the portable computing device (238) having been positioned using a combined map (316) of the tool (306) and a ply model (324) of the composite ply (308), wherein the displayed task information (302) enhances the real-time view (326) of the tool (306).

[0185] Item 2. The method of Item 1 further comprises:

[0186] Composite plies are placed using guidance in task information (302) displayed on a real-time view (326) of a layup location (304) on a tool (306), wherein the guidance facilitates placement of a plurality of composite plies (308) on the tool (306).

[0187] Item 3. The method of any preceding item, wherein displaying the task information (302) of the laying location (304) on the tool (306) comprises:

[0188] identifying a layup location (304) in a ply model (324) of a composite ply (308);

[0189] Determining layup locations (304) on a real-time view (326) of a plurality of guides (330) for a plurality of composite plies (308) using the ply model (324) and the combined map (316); and

[0190] A plurality of guides (330) for a plurality of composite plies (308) at a layup location (304) displayed by a portable computing device (238) on a real-time view (326) of a tool (306) as viewed through a display device (328) in a positioned portable computing device (238), wherein the plurality of guides (330) facilitate placement of the plurality of composite plies (308) on the tool (306).

[0191] Item 4. The method of Item 3 further comprises:

[0192] A plurality of additional guides for the plurality of composite plies (308) at the layup location (304) are displayed by the portable computing device (238) on a real-time view (326) of the tool (306) as viewed through a display device (328) in the already positioned portable computing device (238), wherein the plurality of additional guides illustrate a plurality of previous placements of the plurality of composite plies (308) on the tool (306).

[0193] Item 5. The method of Item 3, wherein the task information (302) of displaying the laying position (304) on the tool (306) further comprises:

[0194] At least one of a ply number, instructions, an image, and a video for placing the plurality of composite plies (308) is displayed.

[0195] Item 6. The method of any preceding item, wherein creating a combined map (316) of tools (306) from the point cloud (318) comprises:

[0196] creating a map from each of the point clouds (318) to form a plurality of maps (320); and

[0197] The plurality of maps (320) are combined to form a combined map (316) of the tool (306).

[0198] Item 7. The method of Item 6, wherein combining the plurality of maps (320) to form a combined map (316) comprises:

[0199] identifying a common reference point in a plurality of maps (320); and

[0200] A plurality of maps (320) are combined using a common reference point, wherein the combined map (316) has improved accuracy from the plurality of maps (320) created from scan data (314) generated from different viewpoints (312).

[0201] Clause 8. The method of clause 1, wherein using the model to locate a portable computing device (238) in the portable computing devices (214) to the tool (306) comprises:

[0202] A portable computing device (238) in the portable computing devices (214) is located to a tool (306) using the combined map (316) and the simultaneous location and mapping process (240).

[0203] Item 9. The method of any preceding item, wherein the task information (302) of the layup location (304) is a guide for laying up composite plies for manufacturing a composite component (310) and / or reworking a composite component (310).

[0204] Item 10. The method of any preceding item, wherein the tool (306) is selected from the group consisting of a mandrel, a mold, and a composite tool (306).

[0205] Item 11. The method of any preceding item, wherein the portable computing device (214) is selected from at least one of smart glasses (120, 122, 406, 408), a mobile phone (118), a tablet computer (116), and a head-mounted display.

[0206] Item 12. The method of any preceding item, wherein the portable computing device (214) scans the tool (306) using at least one of a laser scanner, a structured light 3D scanner, and an infrared light scanner.

[0207] Item 13. A method for enhancing a real-time view (212) of a task location (244), the method comprising:

[0208] positioning the portable computing device (238) toward the subject (206); and

[0209] A visualization (242) of a task location (244) is displayed on a real-time view (212) of an object (206) for performing the task using a model of the object (206) and a combined map (234) of the object (206), wherein the combined map (234) is generated by scanning the object (206) with a portable computing device (214) from different viewpoints (230) of the object (206).

[0210] Item 14. The method of Item 13 further comprises:

[0211] Scan data (228) is received from a portable computing device (214) including a portable computing device (238), wherein the scan data (228) is generated by scanning an object (206) at different viewpoints (230) of the object (206) with the portable computing device (214) on an operator (204).

[0212] Item 15. The method of Item 14, wherein the scan data (228) is received in real time.

[0213] Item 16. The method of Item 14 further comprises:

[0214] creating, by a computer system (218), a point cloud (232) from scan data (228) generated by a portable computing device (214); and

[0215] A combined map (234) of the object (206) is created by a computer system (218) using the point cloud (232).

[0216] Item 17. The method of Item 16, wherein creating, by the computer system (218), a combined map (234) of the object (206) using the point cloud (232) comprises:

[0217] creating a map from each of the point clouds (232) to form a plurality of maps (236); and

[0218] The plurality of maps (236) are combined to form a combined map (234).

[0219] Item 18. The method of any of Items 13 to 17, wherein the task location (244) is for at least one of a composite ply, a component in an assembly, wherein the object (206) is an assembly, a patch, or a decal.

[0220] Item 19. The method of any of Items 13 to 18, wherein the task is selected from placing a composite ply, applying a destruction, applying a decal, performing an inspection of a task location (244), drilling a hole, installing a fastener, connecting a component to an assembly, or removing a component.

[0221] Item 20. The method of any one of Items 13 to 19, wherein the object (206) is selected from the group consisting of a tool (306), a wall, a workpiece, a wing, a fuselage section, an engine, a building, an aircraft, and a vehicle.

[0222] Item 21. An augmented reality system for visualizing a paving location (304) on a tool (306), the augmented reality system comprising:

[0223] The computer system (218) operates to

[0224] receiving scan data (314) based on generating the scan data (314) from a different viewpoint (312) of the tool (306) using a portable computing device (214) on an operator (204);

[0225] creating a plurality of maps (320) of the tool (306) using the scan data (314) generated by the portable computing device (214);

[0226] combining the plurality of maps (320) to form a combined map (316) of the tool (306);

[0227] Task information (302) identifying a laydown location (304) in a ply model (324); and

[0228] Task information (302) of a layup location (304) on a tool (306) is sent to a portable computing device (238) in the portable computing device (214), wherein the portable computing device (238) displays the task information (302) of the layup location (304) on the tool (306) on a real-time view (326) viewed through a display device (328) in the portable computing device (238), the portable computing device (238) having been positioned using a combined map (316) of the tool (306) and a ply model (324) of the composite ply (308).

[0229] Item 22. An augmented reality system of Item 21, wherein the augmented reality system also includes a portable computing device (238), and wherein the portable computing device (238) identifies a laying position (304) in a ply model (324) of a composite ply (308); determines a position on a real-time view (326) of multiple guides (330) of the multiple composite plies (308) using the ply model (324) and the combined map (316); and displays the multiple guides (330) of the multiple composite plies at the position on a real-time view (326) of a tool (306) seen through a display device (328) in the positioned portable computing device (238), wherein the multiple guides (330) are guides for placing the multiple composite plies (308) on the tool (306).

[0230] Item 23. The augmented reality system of Item 22, wherein the portable computing device (238) displays at least one of a ply number, instructions, an image, and a video for placing the plurality of composite plies (308).

[0231] Item 24. The augmented reality system of any one of items 21 to 23, wherein the scan data (314) includes a point cloud (318), and wherein, in creating the combined map (316), the computer system (218) operates to:

[0232] creating a map from each of the point clouds (318) to form a plurality of maps (320); and

[0233] The plurality of maps (320) are combined to form a combined map (316).

[0234] Item 25. The augmented reality system of any one of Items 21 to 24, wherein, in combining the plurality of maps (320) to form the combined map (316), the computer system (218) operates to

[0235] identifying a common reference point in a plurality of maps (320);

[0236] A plurality of maps (320) are combined using a common reference point, wherein the combined map (316) has improved accuracy from the plurality of maps (320) created from scan data (314) generated from different viewpoints (312).

[0237] Item 26. The augmented reality system of any of items 21 to 25, wherein the portable computing device (238) in the portable computing device (214) locates the tool (306) using a combined map (316) and a simultaneous location mapping process (240).

[0238] Item 27. The augmented reality system of any one of Items 21 to 26, wherein the task information (302) of the layup location (304) is a guide for laying up composite plies for manufacturing a composite component (310) and / or reworking a composite component (310) on a tool (306).

[0239] Item 28. The augmented reality system of any one of Items 21 to 27, wherein the tool (306) is selected from the group consisting of a mandrel, a mold, and a composite tool (306).

[0240] Item 29. An augmented reality system for augmenting a real-time view (212) of a task location (244), the augmented reality system comprising:

[0241] A portable computing device (238) is positioned to an object and displays a visualization (242) of a task location (244) on a real-time view (212) of the object for performing a task using a model of the object and a combined map (234) of the object, wherein the combined map (234) is generated by scanning the object with different viewpoints (230) to the object by the portable computing device (214).

[0242] Item 30. The augmented reality system of Item 29 further comprises:

[0243] A computer system (218) in communication with a portable computing device (238), wherein the computer system (218) operates to

[0244] Scan data (228) is received from a portable computing device (214) including a portable computing device (238), wherein the scan data (228) is generated by scanning an object with a portable computing device (214) on an operator (204) at different viewpoints (230) of the object.

[0245] creating a point cloud (232) from scan data (228) generated by a portable computing device (214); and

[0246] A combined map (234) of the object is created using the point cloud (232).

[0247] Item 31. The augmented reality system of Item 30, wherein the scanning data (228) is received in real time.

[0248] Item 32. The augmented reality system of Item 30, wherein, in creating a combined map (234) of an object using point clouds (232), the computer system (218) creates a map from each point cloud in the point clouds (232) to form multiple maps (236), and combines the multiple maps (236) to form the combined map (234).

[0249] Item 33. The augmented reality system of Item 29, wherein the task location (244) is for at least one of a composite ply, a component in an assembly, wherein the object (206) is an assembly, a patch, or a decal.

[0250] Item 34. An augmented reality system of any one of Items 29 to 33, wherein the task is selected from at least one of placing composite plies, applying damage, applying decals, performing inspection of the task location (244), drilling, installing fasteners, connecting components to an assembly, and removing components.

[0251] Item 35. The augmented reality system of any one of Items 29 to 34, wherein the object (206) is selected from the group consisting of a tool (306), a wall, a workpiece, a wing, a fuselage section, an engine, a building, an aircraft, and a vehicle.

[0252] Many modifications and variations will be apparent to those of ordinary skill in the art. In addition, different illustrative embodiments may provide different features than other desired embodiments. The selected embodiment or embodiments are chosen and described in order to best explain the principles of the embodiments, the practical application, and to enable others skilled in the art to understand the present disclosure of various embodiments with various modifications suitable for the intended specific use.

Claims

1. A method for visualizing task information (302) of a laying location (304) on a tool (306), the method comprising: Scanning the tool (306) from different viewpoints (312) of the tool (306) using a portable computing device (214) on the operator (204) to generate scan data (314); creating, by a computer system (218), a point cloud (318) from the scan data (314) generated by the portable computing device (214); creating, by the computer system (218), a combined map (316) of the tool (306) using the point cloud (318); locating one of the portable computing devices (238) to the tool (306) using a combined map (316) of the tool (306); and displaying, by the one portable computing device (238), the task information (302) of the laydown location (304) on the tool (306) on a real-time view (326) seen through a display device (328) in the one portable computing device (238), the one portable computing device (238) having been positioned using the combined map (316) of the tool (306) and a ply model (324) of the composite ply (308), wherein the displayed task information (302) enhances the real-time view (326) of the tool (306), Wherein, creating the combined map (316) of the tool (306) from the point cloud (318) comprises: creating a map from each of the point clouds (318) to form a plurality of maps (320); and combining the plurality of maps (320) to form the combined map (316) of the tool (306), Wherein, combining the plurality of maps (320) to form the combined map (316) comprises: identifying a common reference point in the plurality of maps (320); and The plurality of maps (320) are combined using the common reference point, wherein the combined map (316) has an improved accuracy from the plurality of maps (320) created from the scan data (314) generated from the different viewpoints (312).

2. The method according to claim 1, further comprising: Composite plies are placed using guidance in the task information (302) displayed on the real-time view (326) of the layup location (304) on the tool (306), wherein the guidance facilitates placement of a plurality of the composite plies (308) on the tool (306).

3. A method according to any one of the preceding claims, wherein: Displaying the task information (302) of the laying location (304) on the tool (306) includes: identifying the layup location (304) in the ply model (324) of the composite ply (308); determining layup locations (304) on the real-time view (326) of a plurality of guides (330) for a plurality of the composite plies (308) using the ply model (324) and the combined map (316); and The plurality of guides (330) for the plurality of composite plies (308) at the layup location (304) are displayed by the one portable computing device (238) on the real-time view (326) of the tool (306) as viewed through the display device (328) in the one portable computing device (238) that has been positioned, wherein the plurality of guides (330) facilitate placement of the plurality of composite plies (308) on the tool (306).

4. The method according to claim 3, further comprising: A plurality of additional guides for the plurality of composite plies (308) at the layup location (304) are displayed by the one portable computing device (238) on the real-time view (326) of the tool (306) as viewed through the display device (328) in the one portable computing device (238) that has been positioned, wherein the plurality of additional guides illustrate a plurality of previous placements of the plurality of composite plies (308) on the tool (306).

5. The method according to claim 3, wherein: The task information (302) of displaying the laying position (304) on the tool (306) further includes: At least one of a ply number, instructions, an image, and a video for placing a plurality of the composite plies (308) is displayed.

6. The method according to any one of claims 1 to 2, wherein: Using a model to locate the one of the portable computing devices (214) (238) to the tool (306) includes: The one of the portable computing devices (214) (238) is located to the tool (306) using the combined map (316) and a simultaneous localization and mapping process (240).

7. The method according to any one of claims 1 to 2, wherein: The task information (302) of the layup location (304) is a guide for laying up composite plies for manufacturing a composite component (310) and / or reworking the composite component (310), and wherein the tool (306) is selected from the group consisting of a mandrel, a mold, a composite tool (306).

8. The method according to any one of claims 1 to 2, wherein: The portable computing device (214) is selected from at least one of smart glasses (120, 122, 406, 408), a mobile phone (118), a tablet computer (116), and a head-mounted display, and wherein the portable computing device (214) scans the tool (306) using at least one of a laser scanner, a structured light three-dimensional scanner, and an infrared light scanner.

9. An augmented reality system for visualizing a laying location (304) on a tool (306), the augmented reality system comprising: The computer system (218) operates to receiving scan data (314) based on generating the scan data (314) using a portable computing device (214) on an operator (204) at a different viewpoint (312) of the tool (306); creating a plurality of maps (320) of the tool (306) using the scan data (314) generated by the portable computing device (214); combining the plurality of maps (320) to form a combined map (316) of the tool (306); identifying task information (302) of the laydown location (304) in a ply model (324); and sending the task information (302) of the layup location (304) on the tool (306) to one of the portable computing devices (214) (238), wherein the one portable computing device (238) displays the task information (302) of the layup location (304) on the tool (306) on a real-time view (326) viewed through a display device (328) in the one portable computing device (238), the one portable computing device (238) having been positioned using the combined map (316) of the tool (306) and the ply model (324) of the composite ply (308), wherein the scan data (314) includes a point cloud (318), and wherein, in creating the combined map (316), the computer system (218) operates to: creating a map from each of the point clouds (318) to form a plurality of maps (320); and combining the plurality of maps (320) to form the combined map (316), wherein, in combining the plurality of maps (320) to form the combined map (316), the computer system (218) operates to identifying a common reference point in the plurality of maps (320); The plurality of maps (320) are combined using the common reference point, wherein the combined map (316) has an improved accuracy from the plurality of maps (320) created from the scan data (314) generated from the different viewpoints (312).

10. The augmented reality system according to claim 9, wherein: The augmented reality system also includes the one portable computing device (238), and wherein the one portable computing device (238) identifies the laying location (304) in the ply model (324) of the composite ply (308); determines the location of a plurality of guides (330) of a plurality of composite plies (308) on the real-time view (326) using the ply model (324) and the combined map (316); and displays the plurality of guides (330) of the plurality of composite plies at the location on the real-time view (326) of the tool (306) seen through the display device (328) in the one portable computing device (238) that has been positioned, wherein the plurality of guides (330) are guides for placing the plurality of composite plies (308) on the tool (306), and wherein the one portable computing device (238) displays at least one of a ply number, instructions, an image, and a video for placing the plurality of composite plies (308).

11. The augmented reality system according to any one of claims 9 to 10, wherein: The one of the portable computing devices (214) (238) is located to the tool (306) using the combined map (316) and a simultaneous localization and mapping process (240).

Images