Augmented reality system using augmented models

Abstract

An augmented reality system using an augmented model is disclosed. Methods, apparatus, and systems for visualizing information are provided. An augmented reality system includes a computer system and a visualizer in the computer system. The computer system communicates with an unmanned vehicle using a communication link. The visualizer system receives an image of a physical object from the unmanned vehicle moving relative to the physical object, and receives scan data of an area of ​​the physical object from the unmanned vehicle. The visualizer creates an augmented model of the physical object using the image and the scan data. The area of ​​the physical object in the augmented model has more details than other areas of the physical object. The visualization apparatus sends information to a portable computing device, which can be displayed by the portable computing device on a live view of the physical object. The information is identified using the augmented model of the physical object.

Description

Technical Field

[0001] The present disclosure relates to the technical field of augmented reality, and more particularly, to an augmented reality system using an augmented model. Background Art

[0002] Augmented reality involves enhancing the interactive experience with a real-world environment with information from a computer system. The information is displayed on a live view of the real-world environment seen through a portable computing device. The information is displayed on the live view to provide the user with descriptions or indications of objects in the live view. The information is also referred to as augmented reality information. In other cases, the augmented reality information is displayed on the live view of the real-world environment in a seamlessly interwoven manner so that the information is perceived as part of the real-world environment seen through the portable computing device.

[0003] The simultaneous localization and mapping process uses anchor points to locate the portable computing device in the surrounding environment. An anchor point is a feature point that is a distinctive location of a physical object or the environment near the physical object. An anchor point is used to correspond the model of the physical object to the physical object in the real world seen in the live view of the physical object.

[0004] Sometimes, as the distance of the augmented reality device from the anchor point increases, the augmented reality device displays the augmented reality information over the live view of the physical object with reduced accuracy.

[0005] For example, a distance greater than 5 meters may cause the augmented reality device to display augmented reality information at locations on or near physical objects with an unsatisfactory level of accuracy.

[0006] As the size of physical objects increases, the number of anchor points required to display information with a desired accuracy to augment a live view of the physical object may be greater than is possible or feasible for some physical objects. For example, augmenting a live view of a large object such as an airplane, an office building, or a dam may be more difficult than desired. The accuracy may not be as precise as expected, and the processing resources used may be greater than expected. Summary of the invention

[0007] An embodiment of the present disclosure provides an augmented reality system, which includes an unmanned aerial vehicle group, a computer system, and a portable computing device. The unmanned vehicle group operates to move relative to a physical object, generates an image of the physical object, and generates scan data of spatial points describing an area of ​​the physical object. The computer system communicates with the unmanned vehicle group using a communication link, and operates to receive an image of the physical object from the unmanned vehicle group moving relative to the physical object; receives scan data for the area of ​​the physical object from multiple unmanned vehicles in the unmanned vehicle group moving relative to the physical object; and creates an enhanced model of the physical object using the image and the scan data, wherein the area of ​​the physical object in the enhanced model has more details than other areas of the physical object in the enhanced model. The portable computing device operates to use the enhanced model to position relative to the physical object and display information on a live view of the physical object seen through the portable computing device. The information is identified using the enhanced model of the physical object.

[0008] Another embodiment of the present disclosure provides an augmented reality system, which includes a computer system and a visualizer in the computer system. The computer system communicates with the unmanned vehicle group using a communication link during operation of the computer system and during operation of the unmanned vehicle group. The visualizer operates to receive an image of a physical object from an unmanned vehicle group moving relative to the physical object, and receives scan data of an area of ​​the physical object from multiple unmanned vehicles in the unmanned vehicle group moving relative to the physical object. The visualizer operates to create an enhanced model of the physical object using the image and the scan data. The area of ​​the physical object in the enhanced model has more details than other areas of the physical object in the enhanced model. The visualizer sends information to a portable computing device. The information can be displayed by the portable computing device on a live view of the physical object seen through the portable computing device. The information is identified using the enhanced model of the physical object.

[0009] Another embodiment of the present disclosure provides a method for visualizing information on a live view of a physical object. A computer system receives an image of a physical object from an unmanned vehicle group moving relative to the physical object. The computer system communicates with the unmanned vehicle group using a communication link. The computer system receives scan data of an area of ​​the physical object. The computer system creates an enhanced model of the physical object using the image and the scan data. The area in the enhanced model has more details than other areas of the physical object in the enhanced model. At least a portion of the enhanced model is sent by the computer system to a portable computing device, wherein the portable computing device is positioned relative to the physical object using at least a portion of the enhanced model. Information that can be displayed by the portable computing device is sent by the computer system to the portable computing device. The portable computing device displays the information on the live view of the physical object seen by the portable computing device, and identifies the information using the enhanced model of the physical object.

[0010] These features and functions can be achieved independently in various embodiments of the present disclosure or may be combined in yet other embodiments, further details of which can be seen with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The novel features believed to be characteristic of the exemplary embodiments are set forth in the appended claims. However, the illustrative embodiments and their preferred modes of use, other objects and features thereof will be best understood by reference to the following detailed description of illustrative embodiments of the present disclosure when read in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a pictorial schematic representation of a network of data processing systems in which illustrative embodiments may be implemented;

[0013] Figure 2 is a schematic block diagram of an augmented reality environment according to an illustrative embodiment;

[0014] Figure 3 is a schematic block diagram showing creation of an enhanced model according to an illustrative embodiment;

[0015] Figure 4 is a schematic block diagram showing selection of an area of ​​a physical object according to an illustrative embodiment;

[0016] Figure 5 is a graphical representation of a visualization environment according to an illustrative embodiment;

[0017] Figure 6 is a graphical representation of a graphical user interface according to an illustrative embodiment;

[0018] Figure 7 is a schematic flow chart of a process for visualizing information on a live view of a physical object according to an illustrative embodiment;

[0019] Figure 8 is a schematic flow chart of a process for controlling an unmanned vehicle to generate information for creating an augmented model according to an illustrative embodiment;

[0020] Fig. 9 is a schematic flow chart of a process for creating an augmented model of a physical object according to an illustrative embodiment;

[0021] Fig.10 is a schematic flow chart of a process for visualizing information on a live view of a physical object according to an illustrative embodiment;

[0022] Fig.11 is a schematic block diagram of a data processing system according to an illustrative embodiment;

[0023] Fig.12 is a schematic block diagram of a portable computing device according to an illustrative embodiment;

[0024] Fig.13 is a schematic block diagram of an unmanned vehicle apparatus according to an illustrative embodiment;

[0025] Fig.14 is a schematic block diagram of an aircraft manufacturing and service method according to an illustrative embodiment;

[0026] Fig.15 is a schematic block diagram of an aircraft in which illustrative embodiments may be implemented; and

[0027] Fig.16 is a schematic block diagram of a product management system in accordance with an illustrative embodiment. DETAILED DESCRIPTION

[0028] The illustrative embodiments recognize and take into account one or more different considerations. For example, the illustrative embodiments recognize and take into account that physical objects, such as airplanes, buildings, fields, or some other large physical objects, may make displaying information to enhance views of these physical objects more difficult than desired. The illustrative embodiments recognize and take into account that one solution involves the additional use of one or more computing devices in addition to the portable computing device operated by the operator to generate models of the physical objects. The illustrative embodiments recognize and take into account that these additional computing devices may provide additional information for use in at least one of creating the model of the physical object, positioning the portable computing device with respect to the physical object, or performing other operations.

[0029] For example, the illustrative embodiments recognize and take into account that portable computing devices (such as unmanned vehicles) can be used to generate data for creating models of physical objects. These unmanned vehicles can include at least one of unmanned aerial vehicles, unmanned ground vehicles, or unmanned water vehicles. In addition, the data collected by the portable computing device can be used to generate additional data, which is used in conjunction with the data generated by the unmanned vehicle to generate the model.

[0030] As used herein, when the phrase "at least one" is used with a list of items, it means that different combinations of one or more of the listed items may be used, and only one of each item in the list may be required. In other words, "at least one" means that any combination of items and numbers of items in the list may be used, and not all items in the list are required. The item may be a specific object, thing, or category.

[0031] For example, but not limited to, "at least one of item A, item B, or item C" may include item A, item A and item B, or item B. This example may also include item A, item B, and item C, or item B and item C. Of course, there may be any combination of these items. In some illustrative embodiments, "at least one" may be, for example, but not limited to, two items A; one item B; and ten items C; four items B and seven items C; or other suitable combinations.

[0032] Furthermore, the illustrative embodiments recognize and take into account that, as the size of a physical object increases, scanning the entire physical object to create a model of the physical object may be more resource intensive than desired. For example, the illustrative embodiments recognize and take into account that using a three-dimensional scanner to generate scan data for use in creating a point cloud of a physical object (such as an airplane or a building) may require more bandwidth, processing power, storage, or other computing resources than desired. These three-dimensional scanners may include laser scanners, lidar systems, infrared scanners, or some other type of scanning system.

[0033] The illustrative embodiments recognize and take into account that images can be used to create a model of a physical object. The illustrative embodiments recognize and take into account that the amount of detail in the model of the physical object may be less than the detail in the scan used to generate the point cloud. However, the illustrative embodiments recognize and take into account that a three-dimensional scanner may be used to generate scan data for one or more regions of interest where a higher level of detail is desired. Therefore, by using two types of data to generate a model of the physical object, at least one of the amount of bandwidth, processing resources, storage, or other computing resources may be reduced.

[0034] Thus, illustrative embodiments provide methods, apparatus, and systems for visualizing information on a live view of a physical object. In one illustrative embodiment, a computer system receives an image of a physical object from an unmanned vehicle moving relative to the physical object. The computer system communicates with the unmanned vehicle using a communication link. The computer system receives scan data for an area of ​​the physical object. The computer system creates an enhanced model of the physical object using the image and the scan data, wherein the area in the enhanced model has more detail than other areas of the physical object in the enhanced model. The computer system sends at least a portion of the enhanced model to a portable computing device, wherein the portable computing device is positioned relative to the physical object using at least a portion of the enhanced model. The computer system also sends information that can be displayed by the portable computing device. The portable computing device displays the information on a live view of the physical object seen by the portable computing device, wherein the information is identified using the enhanced model of the physical object.

[0035] Now refer to the drawings, and specifically refer to Figure 1, shows a graphical schematic representation of a network of data processing systems in which the illustrative embodiments may be implemented. Network data processing system 100 is a network of computers in which the illustrative embodiments may be implemented. Network data processing system 100 includes network 102, which is a medium for providing communication links between various devices and computers connected together within network data processing system 100. Network 102 may include connections, such as wired communication links or wireless communication links. The wireless communication links may be established through at least one of air, vacuum, or water.

[0036] The wired communication link may include at least one of an electrical wire or an optical cable. As shown, a wired communication link may be bulky or limit the distance the unmanned vehicle can travel. These types of communication links may enhance safety compared to wireless communication links. These wired communication links may also include intermittent connections that may occur when the unmanned aerial vehicle returns and contacts a refueling station or base station.

[0037] In another example, the intermittent connection can be an intermittent wireless connection affected by line of sight, distance or other factors. Using this type of connection, the first unmanned aerial vehicle may lose the wireless connection. In this case, another unmanned aerial vehicle can move or position itself relative to the first unmanned aerial vehicle to provide a bridge connection with the server computer 104.

[0038] In the example shown, server computer 104 and server computer 106 and storage unit 108 are connected to network 102. In addition, client device 110 is connected to network 102. As shown, client device 110 includes unmanned aerial vehicle 112, unmanned aerial vehicle 114 and unmanned aerial vehicle 116. As shown, unmanned aerial vehicle 112 and unmanned aerial vehicle 114 are fixed-wing aircraft. As shown, unmanned aerial vehicle 116 is a quadcopter. Client device 110 can be, for example, a computer, a workstation or a network computer. In the example shown, server computer 104 provides information to client device 110, such as boot files, operating system images, and applications. In addition, client device 110 can also include other types of client devices, such as mobile phone 118, tablet computer 120 and smart glasses 122. In this illustrative example, server computer 104, server computer 106, storage unit 108 and client device 110 are network devices connected to network 102, wherein network 102 is a communication medium for these network devices. Some or all of the client devices 110 may form an Internet of Things (IoT), where these physical devices may connect to the network 102 and exchange information with each other through the network 102 .

[0039] In this example, client device 110 is a client to server computer 104. Network data processing system 100 may include additional server computers, client computers, and other devices not shown. Client device 110 connects to network 102 using at least one of a wired connection or a wireless connection.

[0040] Program code located in network data processing system 100 may be stored on a computer recordable storage medium and downloaded to a data processing system or other device for use. For example, program code may be stored on a computer recordable storage medium on server computer 104 and downloaded to client device 110 via network 102 for use on client device 110. In some implementations, the processor retrieves the program code and executes the instructions to initiate, perform, or control some of the operations described herein.

[0041] In the example shown, the network data processing system 100 is the Internet, wherein the network 102 represents a total collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol suite to communicate with each other. The core of the Internet is a backbone of high-speed data communication lines between master nodes or hosts, which consists of tens of thousands of commercial, government, educational and other computer systems that route data and messages. Of course, the network data processing system 100 can also be implemented using a variety of different types of networks. For example, the network 102 can include at least one of the Internet, an intranet, a local area network (LAN), a metropolitan area network (MAN), or a wide area network (WAN). As another example, the network 102 can be a mesh network or a self-organizing point-to-point mobile edge network. Figure 1 It is intended as an example, and not as an architectural limitation to the different illustrative embodiments.

[0042] As used herein, when "a plurality of" is used in reference to an item, it refers to one or more of the items. For example, "a plurality of different types of networks" is one or more different types of networks.

[0043] In the illustrative example, human operator 124 uses smart glasses 122 to view a physical object in the form of aircraft 126. In this illustrative example, human operator 124 sees a live view of aircraft 126 using smart glasses 122. This live view may be overlaid with information 142 displayed on the live view seen using smart glasses 122 to form an augmented reality view of aircraft 126.

[0044] In this illustrative example, smart glasses 122 communicate with visualizer 128 located in server computer 104. Visualizer 128 provides information 142 from reference model 132, which is displayed overlaid with a live view of aircraft 126 seen through smart glasses 122. Smart glasses 122 use augmented model 130 to position themselves relative to aircraft 126.

[0045] In this illustrative example, augmented model 130 is a three-dimensional model or map of aircraft 126. In other examples, the model or map may be a different physical object. As shown, unmanned aerial vehicle 112 generates an image 134 of aircraft 126 while flying relative to aircraft 126. Unmanned aerial vehicle 112 sends image 134 to visualizer 128. Unmanned aerial vehicle 114 also generates image 136 of aircraft 126 and sends it to visualizer 128. Unmanned aerial vehicle 116 generates image 138 of aircraft 126 and sends it to visualizer 128.

[0046] Visualizer 128 uses these images from unmanned aerial vehicle to generate a model of aircraft 126. The model is a three-dimensional model of the surface of aircraft 126 and may be generated using currently available techniques for generating models from images. These techniques perform three-dimensional construction from multiple images to create a three-dimensional model of aircraft 126. In this way, aircraft 126 may be drawn using these images.

[0047] In the illustrative example, region 144 of aircraft 126 may not be visible in image 134, image 136, or image 138. As a result, the model of aircraft 126 will have holes or missing portions for region 144. In other cases, region 144 may be included in the images.

[0048] In this example, human operator 124 may require a more detailed model of area 144 of aircraft 126 than can be provided using images 134, 136, and 138. In other words, area 144 may be present in the model generated from the images, but detail of area 144 may be missing or may not be as much as desired. As a result, information 142 displayed about area 144 may not be as accurate as desired.

[0049] In this case, one or more unmanned aerial vehicles may be used to generate a more detailed model of the area. For example, unmanned aerial vehicle 112 may also generate scan data 140 and send scan data 140 to visualizer 128 .

[0050] In this illustrated example, scan data 140 is used to generate a point cloud for region 144 of aircraft 126. The point cloud may then be used to modify region 144 in a model of aircraft 126 to form enhanced model 130. Enhanced model 130 includes region 144 of aircraft 126 with greater detail than other regions of aircraft 126 in enhanced model 130.

[0051] In this illustrative example, unmanned aerial vehicle 112, unmanned aerial vehicle 114, and unmanned aerial vehicle 116 provide two types of data: images and scan data, which are used to generate augmented model 130. Augmented model 130 may be used to localize smart glasses 122. Localization may be performed using a simultaneous localization and mapping (SLAM) process.

[0052] By using an unmanned aerial vehicle, the time and effort spent operating smart glasses 122, generating images, scan data, or some combination thereof to create a model is reduced. Human operators 124 can focus on performing tasks or other operations with aircraft 126.

[0053] In the illustrative example, the reduction in time and effort spent by human operator 124 may be achieved by using at least one of unmanned aerial vehicle 112, unmanned aerial vehicle 114, or unmanned aerial vehicle 116. The unmanned aerial vehicles may provide images 134, images 136, images 138, and scan data 140 that are processed by visualizer 128 to generate augmented model 130.

[0054] These operations performed by the unmanned aerial vehicle may be performed before human operator 124 views aircraft 126. In other words, the unmanned aerial vehicle provides the required data in advance, and human operator 124 does not need to perform any operations to generate data for creating augmented model 130. The unmanned aerial vehicle may also generate images 134, images 136, images 138, and scan data 140 while human operator 124 views aircraft 126.

[0055] Additional images and scan data, either real-time or pre-acquired, may be used to provide at least one of increased accuracy of augmented model 130 or improved detail or granularity in areas other than area 144 .

[0056] For example, among other things, the use of these unmanned aerial vehicles can increase the amount of data generated for the aircraft 126 in a manner that increases the granularity of information in the augmented model 130. In this illustrative example, one of the unmanned aerial vehicles 112, 114, or 116 can capture images from various viewpoints to augment or augment the images used to create the augmented model 130. For example, the unmanned aerial vehicle 112 can generate anchor points from the images and scan data 140. As used herein, an anchor point represents a common feature on the aircraft 126 and the reference model 132 that is used to match the reference model 132 to the aircraft 126. The unmanned aerial vehicle 112 can move a sufficient distance away from the aircraft 126 to capture the common feature where the anchor point is located in a single reference frame. In this way, the number of anchor points required to match the model to the aircraft 126 can be reduced. In other words, the augmented model 130 can be better matched to the aircraft 126. The unmanned aerial vehicle can generate anchor points from the images and scan data 140.

[0057] In addition, by using the unmanned aerial vehicle, the accuracy of the augmented model 130 used to determine the location features on the aircraft 126 can be improved. As shown, the unmanned aerial vehicle generates images and scan data from different viewpoints or positions. Features on the aircraft 126 (such as windows on the aircraft 126) can be captured in at least one of the images 134, images 136, images 138, or scan data 140. When the information related to the window is displayed from the reference model 132, these different viewpoints enable the accuracy of identifying the location of the window to be improved.

[0058] For example, these images and scan data from different positions of the unmanned aerial vehicle can be used to perform triangulation. For example, the visualizer 128 can use computer vision triangulation. Computer vision triangulation is the process of determining a point in three-dimensional space given the projection of the point on two or more images of the point. In the illustrative example, when there are more images, there are different viewpoints, which improves the accuracy of a specific point on the aircraft 126. As another example, stereo photogrammetry can be used to determine a three-dimensional portrayal of a point on an object (such as the aircraft 126). This technique can be performed using images 134, images 136, and images 138 taken from different positions by the unmanned aerial vehicle.

[0059] A human operator 124 using smart glasses 122 may approach an aircraft 126 in such a manner that the human operator 124 using smart glasses 122 may see anchor points. The location of the human operator 124 may be determined using the augmented model 130 and the anchor points seen through the smart glasses 122. As the human operator 124 approaches the aircraft 126, the human operator 124 may see other locations on the aircraft 126 through the smart glasses 120 that do not include the anchor points in the augmented model 130 or the reference model 132. The accuracy of displaying information from the reference model 132 may be maintained through reference points that are identified from the time the anchor points are initially viewed by the human operator 124 using the smart glasses 122. These reference points may be features or elements on the aircraft 126.

[0060] This type of model creation reduces the time and effort required to create a model of a physical object. This type of model creation is particularly useful for large objects, such as airplanes 126, buildings, fields, city blocks, dams, or other types of physical objects, which may result in undesirable use of time and processor resources when creating a model for displaying information for an augmented reality view of the physical object.

[0061] In addition, by processing an image of the physical object and scanned data of one or more regions of the physical object, the amount of computing resources required to process the data is reduced. For example, the processor resources used to generate a point cloud from the scanned data and create a model of the physical object based on the point cloud are reduced because the point cloud is only for one region of the physical object instead of the entire physical object. As another example, because the size of the enhanced model is smaller than a model generated using the scanned data as a whole, storage resources can be reduced.

[0062] Reference now Figure 2 , shows a schematic block diagram of an augmented reality environment according to an illustrative embodiment. In this illustrative example, the visualization environment 200 includes a hardware (such as Figure 1 Components implemented by hardware) in the network data processing system 100 shown in FIG.

[0063] As shown, visualization environment 200 is an environment in which information 202 of physical objects 204 can be seen by human operator 206 using portable computing device 208. In this illustrative example, physical objects 204 are selected from the group consisting of: aircraft, buildings, bridges, dams, vehicles, fields, lakes, mountains, engines, fuselage sections, runways, and other types of objects. In this illustrative example, information 202 is selected from at least one of: mission information, assemblies, videos, nonconformance indications, work instructions, exploded views of assemblies, schematics, or other information about physical objects 204.

[0064] The portable computing device 208 can take a variety of different forms. For example, the portable computing device 208 can be selected from the group consisting of: smart glasses, mobile phones, tablet computers, augmented reality contact lenses, virtual retinal displays, head-mounted displays, and other types of devices suitable for providing an augmented reality view of the physical object 204.

[0065] As shown, human operator 206 may view information 202 displayed on live view 212 of physical object 204 in graphical user interface 210 to assist human operator 206 in performing task 214. In this illustrative example, task 214 is selected from the group consisting of: a design task, a manufacturing task, an inspection task, a maintenance task, a testing task, a task using physical object 204, and other suitable tasks in which live view 212 of physical object 204 is enhanced using information 202 to assist in performing task 214 with respect to physical object 204.

[0066] In this illustrative example, augmented reality system 216 includes a number of different components. As shown, augmented reality system 216 includes computer system 218, unmanned vehicle group 220, and portable computing device 208.

[0067] As used herein, when a “group” is used with reference to an item, it refers to one or more items. For example, “unmanned vehicle group 220 ” is one or more unmanned vehicles 220 .

[0068] Computer system 218 is a physical hardware system and includes one or more data processing systems. When there is more than one data processing system in computer system 218, these data processing systems communicate with each other using a communication medium. The communication medium may be a network. The data processing system may be selected from at least one of the following: a computer, a server computer, a tablet computer, or some other suitable data processing system.

[0069] In this illustrative example, unmanned vehicle group 220 may take a variety of different forms. For example, unmanned vehicle group 220 may be selected from at least one of the following: an unmanned aerial vehicle, an unmanned aerial vehicle, an unmanned ground vehicle, or an unmanned water vehicle. Unmanned vehicle group 220 operates to move relative to physical object 204.

[0070] The unmanned vehicle group 220 generates an image 222 of the physical object 204 and generates scan data 224 of spatial points describing an area 226 of the physical object 204. In this illustrative example, the unmanned vehicle group 220 can generate the scan data 224 by having a plurality of unmanned vehicles 220 in the unmanned vehicle group 220 scan the physical object 204. In other words, all or a subset of the unmanned vehicle group 220 can generate the scan data 224.

[0071] In this illustrative example, unmanned vehicle group 220 operates to generate images 222 and scan data 224 while human operator 206 views live view 212 of physical object 204 via portable computing device 208. For example, unmanned vehicle 220 may generate images 222 and scan data 224 before human operator 206 views live view 212 of physical object 204. In this example, unmanned vehicle 220 may continue to generate images 222 and scan data 224 while human operator 206 views physical object 204. In other examples, generation of images 222 and scan data 224 may occur while human operator 206 sees live view 212 of physical object 204 via portable computing device 208.

[0072] As shown, computer system 218 communicates with unmanned vehicle group 220 using communication link 228. Communication link 228 can be selected from at least one of a wired communication link or a wireless communication link. The wired communication link includes, for example, at least one of a wire, a wire cable, a coaxial cable, an optical fiber, or an optical cable. The wireless communication link can be selected from at least one of a radio frequency signal, an optical signal, electromagnetic radiation, a microwave, or other suitable medium.

[0073] In this illustrative example, visualizer 230 is located in computer system 218 . During operation, visualizer 230 in computer system 218 receives image 222 of physical object 204 from unmanned vehicle group 220 moving relative to physical object 204 .

[0074] Visualizer 230 receives scan data 204 for area 226 of physical object 204 from a plurality of unmanned vehicles 220 in group of unmanned vehicles 220 that move relative to physical object 204. Visualizer 230 creates augmented model 232 of physical object 204 using images 222 and scan data 224.

[0075] Region 226 of physical object 204 in enhanced model 232 has more details than other regions 234 of physical object 204 in enhanced model 232. For example, scan data 224 may provide a higher resolution of region 226 than image 222. As another example, scan data 224 may also provide enhanced regularity in region 226 of enhanced model 232 of physical object 204. A three-dimensional scanner, such as a laser scanner, actively directs radiation toward physical object 204. The response from the radiated energy may increase brightness, reduce shadows, and provide other features that appear with enhanced regularity compared to a passive sensor system, such as a camera. A three-dimensional scanner may also use other types of radiation for scanning, such as electrical radiation, optical radiation, infrared radiation, radiation from other parts of the spectrum, electromagnetic spectrum radiation, acoustic spectrum radiation, or other types of scanning radiation. As yet another example, a three-dimensional scanner may include a pressure sensing device that employs interferometric scanning.

[0076] Using different types of scans that can be performed using a three-dimensional scanner, details of characteristics (such as opacity, reflectivity, hardness, color, hue) or other characteristics can be determined more easily or more accurately than using images from a camera. These characteristics can be used to evaluate physical object 204. For example, the health, remaining life, suitability for use, or other characteristics of physical object 204 can be determined. In an illustrative example, different types of sensors can be used to determine different types of non-conformities. For example, light detected by a camera can be used to identify non-conformities in structures in physical object 204. These non-conformities can be identified by comparing the structures in enhanced model 232 generated by the images detected by the camera with reference model 202 (such as a computer-aided design model or an enhanced model previously generated by physical object 204). Ultrasonic sensors can be used to identify gaps in composite structures in physical object 204. As another example, magnetic sensors can be used to identify inconsistencies in metal structures in physical object 204.

[0077] The visualizer 230 can use these characteristics to classify attributes such as nonconformities. These nonconformities can include at least one of scratches, dents, cracks, missing paint, missing fasteners, or other types of nonconformities.

[0078] As shown, portable computing device 208 uses augmented model 232 to locate relative to physical object 204 and display information 202 on a live view of physical object 204 as seen through portable computing device 208 .

[0079] In this illustrative example, information 202 is identified using augmented model 232 of physical object 204. For example, information 202 may be located in reference model 236. Reference model 236 is a model of physical object 204. Using image registration, corresponding locations in augmented model 232 and reference model 236 may be associated to have the same coordinate system. Additionally, the positioning of portable computing device 208 relative to physical object 204 may use the same coordinate system.

[0080] Reference model 236 can take a variety of different forms. For example, reference model 236 can be a computer-aided design model of physical object 204. In another example, reference model 236 can be a model of physical object 204 created from image 222 and scan data 224 at a previous time. For example, reference model 236 can be a model of physical object 204 before being manufactured. As another example, reference model 236 can be a model of physical object 204 generated at a previous time. This type of reference model 236 can be used to compare with augmented model 232 to determine whether changes have occurred. These changes can be non-conformances, changes in construction, or other types of changes.

[0081] In this illustrative example, visualizer 230 in computer system 218 controls unmanned vehicle group 220 to move relative to physical object 204 and generates image 222 of physical object 204 and scan data 224 of spatial points describing area 226 of physical object 204 .

[0082] In other illustrative examples, unmanned vehicle group 220 may operate autonomously, without input from visualizer 230 in computer system 218. For example, each of unmanned vehicle group 220 may include program code that recognizes physical object 204 as an object for which image 222 and scan data 224 may be generated.

[0083] In one illustrative example, human operator 206 using portable computing device 208 may also contribute data to generate augmented model 232. For example, portable computing device 208 may generate at least one of additional images 238 or additional scan data 240. Portable computing device 208 communicates with computer system 218 using communication link 242. Communication link 242 may be selected from at least one of radio frequency signals, optical signals, electromagnetic radiation, microwaves, or other suitable media.

[0084] As shown, portable computing device 208 can send at least one of additional image 238 or additional scan data 240 to visualizer 230 in computer system 218. In creating augmented model 232 of physical object 204, visualizer 230 can use image 222, additional image 238, scan data 224, and additional scan data 240 to create augmented model 232 of physical object 204.

[0085] The visualizer 230 may be implemented in software, hardware, firmware, or a combination thereof. When software is used, the operations performed by the visualizer 230 may be implemented in program code configured to run on hardware (e.g., a processor unit). When firmware is used, the operations performed by the visualizer 230 may be implemented in program code and data and stored in a permanent memory to run on a processor unit. When hardware is used, the hardware may include circuits that operate to perform the operations in the visualizer 230.

[0086] In an illustrative example, the hardware can take the form of at least one selected from the following: a circuit system, an integrated circuit, an application specific integrated circuit (ASIC), a programmable logic device, or some other suitable type of hardware configured to perform multiple operations. Using a programmable logic device, the device can be configured to perform multiple operations. The device can be reconfigured later or can be permanently configured to perform the multiple operations. Programmable logic devices include, for example, programmable logic arrays, programmable array logic, field programmable logic arrays, field programmable gate arrays, and other suitable hardware devices. In addition, the process can be implemented in an organic component integrated with an inorganic component and can be composed entirely of organic components, which do not include humans. For example, the process can be implemented as a circuit in an organic semiconductor.

[0087] In one illustrative example, one or more technical solutions overcome the technical problems of displaying information to enhance a live view of a physical object while reducing the amount of processing resources used compared to currently used techniques for generating point clouds of the physical object.

[0088] Therefore, one or more technical solutions can provide the technical effect of using two types of data to reduce the amount of processing resources used to create a model of a physical object. In an illustrative example, using image and scan data reduces the amount of processing resources used compared to current techniques that only use point clouds.

[0089] For large physical objects, such as commercial aircraft, dams, and cruise ships, obtaining a 3D model of these types of physical objects becomes more difficult as the distance increases due to the range of currently used 3D scanners. In addition, as the distance increases, the stereo separation that occurs with the scan decreases and may not provide the desired separation. Therefore, using unmanned vehicles to generate both types of data, images and scan data, provides many technical effects. For example, the ability to zoom in to see more details of one or more areas. In addition, the use of unmanned vehicles allows these unmanned vehicles to move to locations that provide greater separation.

[0090] Computer system 218 may be configured to perform at least one of the steps, operations, or actions described in the different illustrative examples using software, hardware, firmware, or a combination thereof. Thus, computer system 218 operates as a special purpose computer system, wherein visualizer 230 in computer system 218 enables the use of two types of data to generate enhanced model 232, which uses fewer processing resources than current processes. Specifically, visualizer 230 converts computer system 218 into a special purpose computer system compared to currently available general purpose computer systems that do not have visualizer 230.

[0091] Next reference Figure 3 , shows a schematic block diagram of creating an enhanced model according to an illustrative embodiment. In illustrative examples, the same reference numeral may be used in more than one figure. Reference numerals are repeated in different figures to represent the same elements in different figures.

[0092] In this illustrative example, visualizer 230 creates augmented model 232 using images 222 received from group of unmanned vehicles 220 and scan data 224 received from number of unmanned vehicles 220 .

[0093] In creating augmented model 232 of physical object 204, visualizer 230 creates model 300 of physical object 204 using image 222. Visualizer 230 can use an object recognition process to recognize physical object 204 in image 222. Visualizer 230 can employ a three-dimensional reconstruction of physical object 204.

[0094] The recognition of physical object 204 from image 222 may be used to perform a three-dimensional reconstruction to create model 300, which is a three-dimensional model of physical object 204. Model 300 may be a computer-aided design (CAD) model, a computer-aided engineering (CAE) model, or some other suitable type of model.

[0095] In the illustrated example, images 222 are images of physical object 204 from different viewpoints. In other words, multiple unmanned vehicles 220 generate images 222 of physical object 204 from different positions. A position is the location of the unmanned vehicle in three-dimensional space and includes the orientation of the unmanned vehicle.

[0096] Scan data 224 is generated by number of unmanned vehicles 220 for area 226. Scan data 224 describes spatial points of area 226 of physical object 204. Area 226 may be missing in image 222, or may be an area where more detail than provided using image 222 is desired.

[0097] Visualizer 230 also creates plurality of point clouds 302 from scan data 224 generated by plurality of unmanned vehicles 220 in unmanned vehicle group 220. Plurality of point clouds 302 describes surface 304 of physical object 204 in area 226. Visualizer 230 modifies model 300 of physical object 204 using plurality of point clouds 302 to form augmented model 232.

[0098] For example, the plurality of point clouds 302 may be converted into a polygonal mesh model, a triangle mesh model, a non-uniform rational B-spline (NURBS) surface model, or a computer-aided model through a currently used process generally referred to as surface reconstruction.

[0099] Additionally, augmented model 232 may also be created using data received from portable computing device 208. For example, additional scan data 240 may be received from portable computing device 208 for area 226 and used to generate a point cloud in number of point clouds 302. As another example, additional image 238 may be received from portable computing device 208 and used to generate model 300.

[0100] Figure 4 2 is a schematic block diagram of a selection of an area of ​​a physical object depicted according to an illustrative embodiment. In this illustrative example, visualizer 230 in computer system 218 selects area 226 of physical object 204 and controls multiple unmanned vehicles 220 in unmanned vehicle group 220 to generate scan data 224 of area 226 of physical object 204.

[0101] Region 226 may be selected in a number of different ways. For example, visualizer 230 may select region 226 of physical object 204 based on gaze point 400 of human operator 206 using portable computing device 208. Gaze point 400 is where human operator 206 is looking. In this illustrative example, portable computing device 208 may measure gaze point 400 of human operator 206. The location of gaze point 400 on physical object 204 may be used to determine region 226 on physical object 204.

[0102] In another illustrative example, visualizer 230 may select region 226 of physical object 204 based on location 402 of task 214 performed by human operator 206 using portable computing device 208. In this example, region 226 surrounds location 402.

[0103] By identifying region 226 , visualizer 230 may control number of unmanned vehicles 220 to generate scan data 224 . Using scan data 224 , region 226 of physical object 204 in augmented model 232 may have more details than other regions 234 of physical object 204 in augmented model 232 .

[0104] Additionally, artificial intelligence system 404 can assist in identifying area 226. For example, artificial intelligence system 404 can predict where human operator 206 will look. In other words, artificial intelligence system 404 can predict gaze point 400 of human operator 206 and can direct multiple unmanned vehicles 220 to the predicted area on physical object 204 to generate scan data 224 and generate or update augmented model 232. This generation or updating of augmented model 232 can occur before human operator 206 changes gaze point 400 to the predicted area.

[0105] Additionally, portable computing device 208 may generate and send additional scan data 240 to visualizer 230 . In this illustrative example, additional scan data 240 may be used for gaze point 400 .

[0106] Figures 2 to 4 The schematic diagram of the visualization environment 200 shown in and the different components in the visualization environment 200 are not meant to imply that physical limitations or structural limitations are imposed on the mode that can realize the illustrative embodiments. Other components other than or in place of the components shown can be used. Some components may be unnecessary. In addition, these square blocks are shown to illustrate some functional components. When implemented in the illustrative embodiments, one or more of these square blocks can be combined, divided or combined and divided into different square blocks.

[0107] For example, in addition to or in lieu of portable computing device 208, one or more portable computing devices and one or more human operators using the one or more portable computing devices may be present in the visualization environment. As another example, although augmented model 232 has been described in the illustrative example as being used in augmented reality system 216, augmented model 232 may be used in other mixed reality systems, such as a virtual reality system. In a virtual reality system, augmented model 232 may be used for training to perform actions on physical object 204. For example, human operator 206 may view augmented model 232 of physical object 204 and train to perform task 214 on physical object 204. In another example, in addition to or in lieu of area 226, one or more areas may be scanned to create scan data for these one or more areas.

[0108] For example, scan data 224 may be used to update reference model 236 instead of creating augmented model 232. In this manner, reference model 236 may be updated to reflect changes to physical object 204 over time.

[0109] In this example, region 226 of physical object 204 may be scanned, and the corresponding region in reference model 236 may be updated. Updating reference model 236 may account for changes in physical object 204 over time. In addition, scan data 224 may be used to increase the detail in region 226 in reference model 236 to more accurately reflect changes in physical object 204 over time in reference model 236 at the same level of detail.

[0110] In one illustrative example, a structure in physical object 204 may be reshaped, deformed, or changed over time. If a structure that has changed is to be replaced, the current configuration with the changed structure is used to identify or equip the replacement structure. Reference model 236 that has not changed cannot be used to replace the structure. In this case, region 226 of physical object 204 where the structure is located may be scanned to generate scan data 224 for region 226 of physical object 204 in reference model 236 of physical object 204.

[0111] As another example, enhanced model 232 of image 222 and scan data 224 may be used to create reference model 236. This type of process is useful when reference model 236 does not previously exist.

[0112] In another illustrative example, the type of physical object is known, but more detailed identification of the reference model to be used may not be known. For example, for an aircraft, there may be several reference models for a particular type of aircraft, but the particular model of the aircraft or the identification number of the aircraft may not be known. Enhanced model 232 may be used to identify a particular reference model for use. In some cases, the confidence level in identifying the physical object may be less than 100%. In this case, a subset of the reference models may be identified for use.

[0113] Next go to Figure 5 , depicts a diagram of a visualization environment according to an illustrative embodiment. In this illustrative example, the visualization environment 500 is Figure 2 An example of an implementation of the visualization environment 200 in FIG.

[0114] In this illustrative example, visualization environment 500 includes fuselage section 502. As shown, human operator 504 is inspecting fuselage section 502 at a current stage of manufacture. As shown, human operator 504 is wearing smart glasses 506, which is a type of portable computing device. Additionally, unmanned aerial vehicles 508 and 510 are quadcopters in the depicted example.

[0115] As shown, unmanned aerial vehicle 508 and unmanned aerial vehicle 510 generate images of fuselage portion 502. In addition, unmanned aerial vehicle 508 and unmanned aerial vehicle 510 may also scan one or more areas of fuselage portion 502 to generate scan data. For example, unmanned aerial vehicle 508 may scan area 520 surrounding nonconformance 512 in the form of a crack in fuselage portion 502.

[0116] The images and scan data generated by unmanned aerial vehicle 508 and unmanned aerial vehicle 510 are sent to server computer 512 via wireless communication link 514 and wireless communication link 516 .

[0117] As shown, server computer 512 generates an enhanced model using images and scan data received from unmanned aerial vehicle 508 and unmanned aerial vehicle 510. As shown, the scan data of area 520 provides a higher visualization resolution of area 520 including nonconformance 512.

[0118] At least a portion of the augmented model is sent to the smart glasses 506 via the wireless communication link 518. The portion of the augmented model sent to the smart glasses 506 is information that can be rendered and displayed by the smart glasses 506 to augment the live view of the area 520 for the human operator 504.

[0119] refer to Figure 6, a diagram of a graphical user interface is shown according to an illustrative embodiment. In this illustrative example, a graphical user interface 600 is displayed on Figure 5 On smart glasses 506 worn by human operator 504 in .

[0120] In this illustrative example, graphical user interface 600 includes live view 602 of fuselage section 502 and information 604 from an augmented model used to augment live view 602. Information 604 may also be referred to as augmented reality information. In this illustrative example, information 604 includes graphical indication 610 that identifies nonconformance 512 in live view 602 of fuselage section 502. Graphic indication 610 draws the attention of human operator 504 to the nonconformance in live view 602 of fuselage section 502. In addition, graphical indication 612 highlights nonconformance 512. Using scan data in addition to an image, graphical indication 612 more accurately displays nonconformance 512 of live view 602. In addition, information 604 also includes work instructions 614 displayed on live view 602, work instructions 614 identifying an operation to be performed on nonconformance 512.

[0121] Next go to Figure 7 , a schematic flowchart of a process for visualizing information on a live view of a physical object is depicted in accordance with an illustrative embodiment. Figure 7 The processes in may be implemented in hardware, software, or both. When implemented in software, the processes may take the form of program code executed by one or more processor units in one or more hardware devices in one or more computer systems. For example, the processes may be implemented in Figure 2 The visualization device 230 in the computer system 218 is implemented in the visualization device 230 in the computer system 218.

[0122] The process begins by receiving an image of a physical object from a group of unmanned vehicles moving relative to the physical object (operation 700). In operation 700, the image is received via a communication link with the unmanned vehicle.

[0123] The process receives scan data for an area of ​​a physical object (operation 702). The scan data is received from a plurality of unmanned vehicles in a group of unmanned vehicles via a plurality of communication links.

[0124] The process creates an enhanced model of the physical object using the image and scan data (operation 704). The area in the enhanced model has more details than other areas of the physical object in the enhanced model. The more details can be increased resolution, which helps to improve the accuracy of placing information on the live view of the area of ​​the physical object.

[0125] The process sends at least a portion of the augmented model to the portable computing device (operation 706). The portable computing device is positioned relative to the physical object using at least a portion of the augmented model.

[0126] The process sends the information displayable by the portable computing device, wherein the portable computing device displays the information on a live view of the physical object as seen through the portable computing device (operation 708). The process terminates thereafter.

[0127] In operation 708, the information is identified using the augmented model of the physical object. For example, by identifying a location on the physical object seen in a live view of the physical object, the location can be identified in the augmented model, which can be used to identify a corresponding location in a reference model of the physical object containing the information. For example, feature extraction and semantic scene segmentation can be performed on an electro-optical (EO) image or an infrared (IR) image. Using image classification and object recognition, the acquired image can match the scanned information to the reference model.

[0128] In some cases, there are other location features or signatures that can define hard (non-moving) or soft (primarily fixed) waypoints. These location signatures can be physical locations such as those determined by a global positioning system and directions obtained. Alternatively, at least one of a two-dimensional barcode or a three-dimensional barcode can be used for automatic identification. Radio frequency identifiers, known signs, or known identifier plates on aircraft with predetermined configurations can also be used for automatic identification. Information for expected corresponding locations in the reference model can be obtained from the reference model.

[0129] Reference Figure 8 , a schematic flow chart of a process for controlling an unmanned vehicle to generate information for creating an augmented model is depicted in accordance with an illustrative embodiment. Figure 8 The processes in may be implemented in hardware, software, or both. When implemented in software, the processes may take the form of program code executed by one or more processor units in one or more hardware devices of one or more computer systems. For example, the processes may be implemented in Figure 2 The visualization device 230 in the computer system 218 is implemented in the visualization device 230 in the computer system 218.

[0130] The information includes at least one of an image or scan data. The image can be used to create a model of the physical object. The scan data can be used to enhance one or more areas of the model to have more details, thereby forming an enhanced model.

[0131] The process begins by controlling the unmanned vehicle group to move relative to the physical object (operation 800). The process controls the unmanned vehicle group to generate an image of the physical object and scan data describing spatial points for an area of ​​the physical object (operation 802). Generating the image can occur when the unmanned vehicle moves relative to the physical object. In addition, generating the image can occur when one or more unmanned vehicles are in a fixed position relative to the object. In other words, the image can be generated when the unmanned vehicle moves, when the unmanned vehicle is in a specific fixed position, or some combination thereof. The process terminates thereafter.

[0132] Reference Fig. 9 , a schematic flowchart of a process for creating an augmented model of a physical object is depicted in accordance with an illustrative embodiment. Fig. 9 The process is Figure 7 An example of an implementation of operation 704 in .

[0133] The process begins by creating a model of a physical object using an image (operation 900). The process creates a plurality of point clouds from scan data generated by a plurality of unmanned vehicles in a fleet of unmanned vehicles (operation 902).

[0134] The process modifies the model of the physical object using the plurality of point clouds to form an enhanced model (operation 904). The process terminates thereafter. In operation 904, the process may replace a portion of the region in the model with a model of the region generated using the plurality of point clouds.

[0135] Steering Fig.10 , a schematic flowchart of a process for visualizing information on a live view of a physical object is depicted in accordance with an illustrative embodiment. Fig.10 The processes in may be implemented in hardware, software, or both. When implemented in software, the processes may take the form of program code executed by one or more processor units in one or more hardware devices of one or more computer systems. For example, the processes may be implemented in Figure 2 The augmented reality system 216 is implemented in the portable computing device 208.

[0136] The process begins by positioning a portable computing device relative to a physical object using an augmented model of the physical object (operation 1000). The positioning in operation 1000 may be performed using a simultaneous localization and mapping (SLAM) process running on the portable computing device.

[0137] The process displays information on a live view of a physical object seen by a display device in a portable computing device, which uses an enhanced model of the physical object and a reference model of the physical object to locate (operation 1002). In operation 1002, a position on the live view of the physical object can be associated with a corresponding position on the enhanced model. Conversely, a position in the enhanced model can be associated with a reference model of the physical object. Information can be identified based on the position in the reference model. The information can be displayed on the live view of the physical object. These associations can be made using currently available image registration processes.

[0138] Operation 1002 can be implemented using currently available augmented reality applications, such as Vuforia and Vuforia developed by PTC. TM Augmented reality software. The process then terminates.

[0139] The flow chart and block diagram in the different embodiments depicted show the structure, function and operation of some possible implementations of the device and method in the illustrative embodiments. In this regard, each square block in the flow chart or block diagram can represent at least one of a part of a module, a part, a function or an operation or a step. For example, one or more square blocks can be implemented as a combination of program code, hardware or program code and hardware. When implemented in hardware, the hardware can, for example, take the form of an integrated circuit that is manufactured or configured to perform one or more operations in the flow chart or block diagram. When implemented as a combination of program code and hardware, the implementation can take the form of firmware. Each square block in the flow chart or block diagram can be implemented using a combination of a dedicated hardware system or dedicated hardware that performs different operations and a program code run by dedicated hardware.

[0140] In some alternative implementations of the illustrative embodiments, one or more functions noted in the blocks may not occur in the order noted in the drawings. For example, in some cases, two blocks shown in succession may be performed substantially simultaneously, or the blocks may sometimes be performed in reverse order, depending on the functions involved. Furthermore, other blocks may be added in addition to the blocks shown in the flowchart or block diagram.

[0141] Furthermore, these examples are about Figure 2 In other illustrative examples, these processes may be performed by including Figure 4 The artificial intelligence system in 404 or Figure 1 The method may be performed by at least one other component of the visualizer 128 .

[0142] Now go to Fig.11 , depicts a schematic block diagram of a data processing system according to an illustrative embodiment. The data processing system 1100 may be used to implement Figure 1 The data processing system 1100 can also be used to implement the server computer 104, the server computer 106 and the client device 110. Figure 2 1104, memory 1106, persistent storage 1108, communications unit 1110, input / output (I / O) unit 1112, and display 1114. In this example, communications framework 1102 takes the form of a bus system.

[0143] Processor unit 1104 is used to execute instructions of software that may be loaded into memory 1106. Processor unit 1104 includes one or more processors. For example, processor unit 1104 may be selected from at least one of a multi-core processor, a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a network processor, or some other suitable type of processor.

[0144] Memory 1106 and permanent storage 1108 are examples of storage devices 1116. A storage device is any hardware capable of storing information temporarily, permanently, or temporarily and permanently, such as, but not limited to, at least one of data, program code in the form of a function, or other suitable information. In these illustrative examples, storage devices 1116 may also be referred to as computer-readable storage devices. In these examples, memory 1106 may be, for example, a random access memory or any other suitable volatile or non-volatile storage device. Permanent storage 1108 may take various forms, depending on the particular implementation.

[0145] For example, permanent storage 1108 may include one or more components or devices. For example, permanent storage 1108 may be a hard drive, a solid-state drive (SSD), flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination thereof. The media used by permanent storage 1108 may also be removable. For example, a removable hard drive may be used for permanent storage 1108.

[0146] Communications unit 1110 , in these illustrative examples, provides for communications with other data processing systems or devices. Communications unit 1110 , in these illustrative examples, is a network interface card.

[0147] Input / output unit 1112 allows input and output of data with other devices that may be connected to data processing system 1100. For example, input / output unit 1112 may provide a connection for user input through at least one of a keyboard, a mouse, or some other suitable input device. Additionally, input / output unit 1112 may send output to a printer. Display 1114 provides a mechanism for displaying information to a user.

[0148] Instructions for at least one of an operating system, applications, or programs may be located in storage devices 1116, which are in communication with processor unit 1104 through communications framework 1102. The processes of the different embodiments may be performed by processor unit 1104 using computer-implemented instructions, which may be located in a memory, such as memory 1106.

[0149] These instructions are referred to as program code, computer usable program code, or computer readable program code that can be read and executed by a processor in processor unit 1104. The program code in different embodiments may be implemented on different physical or computer readable storage media, such as memory 1106 or permanent storage 1108.

[0150] Program code 1118 is located in functional form on computer readable media 1120, which may be selectively moved and loaded or transferred to data processing system 1100 for execution by processor unit 1104. In these illustrative examples, program code 1118 and computer readable media 1120 form computer program product 1122. In the illustrative example, computer readable media 1120 is computer readable storage media 1124.

[0151] In these illustrative examples, computer readable storage media 1124 is a physical or tangible storage device used to store program code 1118 rather than a medium that propagates or transmits program code 1118 .

[0152] Alternatively, program code 1118 may be transferred to data processing system 1100 using a computer readable signal medium. Computer readable signal media may be, for example, a propagated data signal containing program code 1118. For example, a computer readable signal medium may be at least one of an electromagnetic signal, an optical signal, or any other suitable type of signal. These signals may be transmitted over a connection such as a wired communication link or a wireless communication link. A wired communication link may include a connection using an optical cable, a coaxial cable, a wire, or any other suitable type of connection.

[0153] The different components shown for data processing system 1100 are not meant to provide architectural limitations to the manner in which different embodiments may be implemented. In some illustrative examples, one or more of these components may be incorporated into or otherwise form part of another component. For example, in some illustrative examples, memory 1106 or portions thereof may be incorporated into processor unit 1104. The different illustrative embodiments may be implemented in a data processing system including components in addition to or in place of those shown for data processing system 1100. Fig.11 Other components shown in the can be different from the illustrative examples shown. The different embodiments can be implemented using any hardware device or system capable of running program code 1118.

[0154] refer to Fig.12 , depicts a schematic block diagram of a portable computing device according to an illustrative embodiment. The portable computing device 1200 can implement Figure 2 12 is an example of one embodiment of a portable computing device 208 in FIG. In this illustrative example, the portable computing device 1200 includes physical hardware components such as a processor unit 1202, a communication framework 1204, a memory 1206, a data storage device 1208, a communication unit 1210, a display 1212, and a sensor system 1214.

[0155] Communications framework 1204 allows different components in portable computing device 1200 to communicate with each other when connected to communications framework 1204. In this illustrative example, communications framework 1204 is a bus system.

[0156] Processor unit 1202 processes program code for software loaded into memory 1206. In this illustrative example, the program code may include applications such as augmented reality application 1205 and simultaneous localization and mapping (SLAM) method 1207.

[0157] The augmented reality application 1205 may operate to display information over a live view of a physical object seen through the display 1212 in the portable computing device 1200 to provide an augmented reality view.

[0158] The simultaneous positioning and mapping method 1207 can be operated to create a map or model of a physical object. In addition, this process can also be operated to locate or identify the position of the portable computing device 1200 relative to the physical object. This process can be used to determine where to display information relative to a live view of the physical object.

[0159] Processor unit 1202 includes one or more processors. For example, processor unit 1202 may be selected from at least one of a multi-core processor, a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a network processor, or some other suitable type of processor.

[0160] Memory 1206 is coupled to processor unit 1202 via communications framework 1204. As shown, memory 1206 may include at least one of a random access memory (RAM), a read only memory (ROM), a static random access memory (SRAM), a dynamic random access memory (DRAM), or other suitable types of memory devices or circuits.

[0161] As shown, data storage 1208 is connected to communication framework 1204 and can store data, program code, or other information. Instructions in the program code can be loaded from data storage 1208 into memory 1206 for processing by processor unit 1202. Data storage 1208 may include at least one of a hard drive, a flash drive, a solid state drive, an optical drive, or some other suitable type of data storage device or system.

[0162] In this illustrative example, communications unit 1210 provides for communications with other data processing systems or devices. In these illustrative examples, communications unit 1210 includes at least one of a network interface card, a wireless communication device, a universal serial bus port, or other appropriate devices.

[0163] The display 1212 is connected to the communication framework 1204 and provides a mechanism for displaying information to the user. In this example, the display 1212 can be a touch screen display, which enables receiving user input through the display.

[0164] In this illustrative example, sensor system 1214 is connected to communication framework 1204. As shown, sensor system 1214 may include hardware, software, or both that control the operation of camera system 1216 and three-dimensional scanner 1218 in sensor system 1214. Camera system 1216 is physical hardware that includes one or more cameras capable of recording or capturing images. Camera system 1216 is one or more digital cameras and may include at least one of a stereo camera, a mirrorless camera, or some other type of imaging device. The camera may also be, for example, at least one of an electro-optical camera or an infrared camera. The image may be a separate image of an image of a video.

[0165] The three-dimensional scanner 1218 is hardware capable of scanning a physical object to generate scan data. The scan data describes points on the physical object. The scan data can be used to generate a model of an area of ​​the object that has more detail than other areas of the object created using images. The data can be used in conjunction with the simultaneous positioning and mapping method 1207 to map the object and position the portable computing device 1200 relative to the physical object. The three-dimensional scanner 1218 can take a variety of different forms. For example, the three-dimensional scanner 1218 can be selected from at least one of a laser scanner, a lidar system, an infrared scanner, or some other type of scanning system.

[0166] The illustrated portable computing device 1200 is an example of one manner in which the portable computing device 1200 may be implemented. This diagram is not meant to limit the manner in which the portable computing device 1200 may be implemented in other illustrative examples.

[0167] Reference Fig.13 , depicts a schematic block diagram of an unmanned vehicle device according to an illustrative embodiment. The unmanned vehicle 1300 can be implemented Figure 1 Unmanned aerial vehicles 112, Figure 1 Unmanned aerial vehicles in 114, Figure 1 Unmanned aerial vehicle 116 and Figure 2 An example of a method of unmanned vehicle 220 in FIG.

[0168] In this illustrative example, unmanned vehicle 1300 includes multiple components. As shown, unmanned vehicle 1300 includes frame 1302, propulsion system 1304, computer 1306, communication system 1308, and sensor system 1310.

[0169] Frame 1302 is a physical structure designed based on the type of motion used by unmanned vehicle 1300. For example, if unmanned vehicle 1300 is an unmanned aerial vehicle, unmanned vehicle 1300 may have an aerodynamic surface. If unmanned vehicle 1300 is an unmanned water vehicle, unmanned vehicle 1300 may be a hull for use in water. In this illustrative example, propulsion system 1304, computer 1306, communication system 1308, and sensor system 1310 are connected to frame 1302.

[0170] The propulsion system 1304 is a hardware system that enables the unmanned vehicle 1300 to move. For example, when the unmanned vehicle 1300 is an unmanned aerial vehicle, the propulsion system 1304 may include a jet engine, a rotor, or other propulsion components.

[0171] Computer 1306 is hardware that controls the operation of components in unmanned vehicle 1300. For example, computer 1306 may control the operation of propulsion system 1304, communication system 1308, and sensor system 1310.

[0172] The communication system 1308 is hardware that provides communication using a wired communication link or a wireless communication link. The communication link can be established with a remote computer on the ground or in other unmanned vehicles. The wireless communication link can use radio frequency signals or optical signals.

[0173] Sensor system 1310 includes hardware, software, or both. As shown, sensor system 1310 includes camera system 1312 and three-dimensional scanner 1314.

[0174] You can Fig.14 Aircraft manufacturing and service method 1400 is shown and Fig.15 Illustrative embodiments of the present disclosure are described in the context of aircraft 1500 as shown. Fig.14 , a schematic block diagram of an aircraft manufacturing and service method is depicted according to an illustrative embodiment. During pre-production, aircraft manufacturing and service method 1400 may include Fig.15 Specification and design 1402 of aircraft 1500 and procurement of materials 1404.

[0175] During production, Fig.15 Component and subassembly manufacturing 1406 and system integration 1408 of aircraft 1500. Thereafter, Fig.15 The aircraft 1500 in the embodiment may be certified and delivered 1410 for entry into service 1412. When used 1412 by a customer, Fig.15 Aircraft 1500 is scheduled for routine maintenance and service 1414, which may include modification, reconfiguration, refurbishment, and other maintenance or service.

[0176] Each process of the aircraft manufacturing and service method 1400 can be performed or executed by a system integrator, a third party, an operator, or some combination thereof. In these examples, the operator can be a customer. For the purpose of description, a system integrator can include, but is not limited to, any number of aircraft manufacturers and major system subcontractors; a third party can include, but is not limited to, any number of vendors, subcontractors, and suppliers; an operator can be an airline, a leasing company, a military entity, a service organization, etc.

[0177] Reference now Fig.15 , shows a schematic block diagram of an aircraft in which the illustrative embodiments may be implemented. In this example, the aircraft 1500 is composed of Fig.1414. Aircraft manufacturing and service method 1400 is produced in the context of an aircraft manufacturing and service method 1400, and may include airframe 1502 having multiple systems 1504 and interior 1506. Examples of systems 1504 include one or more of propulsion system 1508, electrical system 1510, hydraulic system 1512, and environmental system 1514. Any number of other systems may be included. Although an aerospace example is shown, different illustrative embodiments may be applied to other industries, such as the automotive industry.

[0178] The devices and methods described herein can be used in Fig.14 The invention is used during at least one of the stages of aircraft manufacturing and service method 1400.

[0179] In one illustrative example, Fig.14 The components and subassemblies produced in component and subassembly manufacturing 1406 may be manufactured in a manner similar to that of aircraft 1500. Fig.14 As another example, the components and subassemblies produced during the use 1412 can be made or manufactured in the same manner as in the production stage. Fig.14 One or more apparatus embodiments, method embodiments, or a combination thereof are used during component and subassembly manufacturing 1406 and system integration 1408 in the aircraft 1500. Fig.14 One or more apparatus embodiments, method embodiments, or a combination thereof may be used during maintenance and service 1414, or both.

[0180] For example, Figure 2 Augmented reality system 216 in can be used to provide visualizations of mission locations. These visualizations can include information about the missions to be performed at the mission locations. Augmented reality system 216 can be used by a human operator during at least one of component and subassembly manufacturing 1406, system integration 1408, certification and delivery 1410, or maintenance and service 1414. Augmented reality system 216 can be used to view larger objects, such as a partially assembled aircraft, compared to using current techniques for augmenting live views of objects.

[0181] Use of the various illustrative embodiments may substantially speed up assembly of aircraft 1500, reduce the cost of aircraft 1500, or both speed up assembly of aircraft 1500 and reduce the cost of aircraft 1500. For example, the amount of processor resources required for notification operations may be reduced while reducing the amount of time required to generate a model for performing operations on an object, such as aircraft 1500 or a portion thereof.

[0182] Now go to Fig.16, a block diagram of a product management system depicted according to an illustrative embodiment is shown. Product management system 1600 is a physical hardware system. In this illustrative example, product management system 1600 includes at least one of manufacturing system 1602 or maintenance system 1604.

[0183] Manufacturing system 1602 is configured to manufacture products, such as Fig.15 16. As shown, manufacturing system 1602 includes manufacturing equipment 1606. Manufacturing equipment 1606 includes at least one of fabrication equipment 1608 or assembly equipment 1610.

[0184] The manufacturing device 1608 is used to manufacture Fig.15 The equipment for making parts of the aircraft 1500 is shown in FIG. 1608. For example, the equipment 1608 may include machines and tools. The machines and tools may be drills, hydraulic presses, furnaces, molds, composite tape laying machines, vacuum systems, lathes, Figure 2 The fabrication device 1608 may be used to fabricate at least one of a metal part, a composite part, a semiconductor, a circuit, a fastener, a rib, a skin panel, a wing spar, an antenna, or other suitable types of parts.

[0185] Assembly equipment 1610 is used to assemble components to form Fig.15 In particular, assembly equipment 1610 is used to assemble components and parts to form aircraft 1500. Assembly equipment 1610 may also include machines and tools. These machines and tools may be robotic arms, crawlers, rapid installation systems, track-based drilling systems, Figure 2 The assembly equipment 1610 can be used to assemble components such as seats, horizontal tails, wings, engines, engine housings, landing gear systems, and Fig.15 Other parts of the AVIC 1500.

[0186] In this illustrative example, maintenance system 1604 includes maintenance equipment 1612. Maintenance equipment 1612 may include any equipment needed to perform maintenance on aircraft 1500. Maintenance equipment 1612 may include tools for performing various operations on parts on aircraft 1500. These operations may include disassembling a part, refurbishing a part, inspecting a part, reworking a part, manufacturing a replacement part, or performing other operations on a part. Fig.15 At least one of other operations for performing maintenance on aircraft 1500. These operations can be for routine maintenance, inspection, upgrade, refurbishment or other types of maintenance operations.

[0187] In the illustrative example, maintenance equipment 1612 may include ultrasonic inspection equipment, x-ray imaging systems, vision systems, drills, crawlers, and other suitable equipment. In some cases, maintenance equipment 1612 may include fabrication equipment 1608, assembly equipment 1610, or both to produce and assemble parts required for maintenance.

[0188] Product management system 1600 also includes control system 1614. Control system 1614 is a hardware system and may also include software or other types of components. Control system 1614 is configured to control the operation of at least one of manufacturing system 1602 or maintenance system 1604. Specifically, control system 1614 may control the operation of at least one of fabrication equipment 1608, assembly equipment 1610, or maintenance equipment 1612.

[0189] The hardware in control system 1614 may be implemented using hardware that may include computers, circuits, networks, and other types of devices. The control may take the form of direct control of manufacturing equipment 1606. For example, robots, computer-controlled machines, and other equipment may be controlled by control system 1614. In other illustrative examples, control system 1614 may manage operations performed by human operators 1616 in manufacturing or maintaining aircraft 1500. For example, control system 1614 may assign tasks, provide instructions, display models, or perform other operations to manage operations performed by human operators 1616. In these illustrative examples, Figure 2 The augmented reality system 216 in the embodiment may be implemented in or used in the control system 1614 to manage Fig.15 At least one of manufacturing or maintenance of aircraft 1500.

[0190] For example, control system 1614 may assign tasks, such as assembling or maintaining an object, such as an aircraft, a building, a dam, or some other suitable object, to one or more human operators 1616. Control system 1614 may assign tasks, such as assembling or maintaining an object, such as an aircraft, a building, a dam, or some other suitable object, to one or more human operators 1616. Figure 2 The portable computing device in the augmented reality system 216 sends task information to enhance the live view.

[0191] In different illustrative examples, human operator 1616 may operate or interact with at least one of manufacturing equipment 1606, maintenance equipment 1612, or control system 1614. Such interaction may be used to manufacture Fig.15 There are 1500 aircraft in the market.

[0192] Of course, the product management system 1600 can be configured to manage Fig.15Although product management system 1600 has been described for manufacturing in the aerospace industry, product management system 1600 can be configured to manage products for other industries. For example, product management system 1600 can be configured to manufacture products for the automotive industry and any other suitable industry.

[0193] Furthermore, the present disclosure includes embodiments according to the following items:

[0194] Item 1. An augmented reality system (216), comprising:

[0195] an unmanned vehicle assembly (220) operable to move relative to a physical object (204), generate an image (222) of the physical object (204), and generate scan data (224) of spatial points describing an area (226) of the physical object (204);

[0196] A computer system (218) in communication with the unmanned vehicle array (220) using a communication link (228), wherein the computer system (218) is operative to:

[0197] Receiving an image (222) of the physical object (204) from an unmanned vehicle group (220) moving relative to the physical object (204);

[0198] Receiving scan data (224) of an area (226) of the physical object (204) from a plurality of unmanned vehicles (220) in a group (220) of unmanned vehicles moving relative to the physical object (204);

[0199] creating an enhanced model (232) of the physical object (204) using the image (222) and the scan data (224), wherein regions (226) of the physical object (204) in the enhanced model (232) have greater detail than other regions (234) of the physical object (204) in the enhanced model (232); and

[0200] A portable computing device (208) operable to:

[0201] Positioning relative to the physical object (204) using the augmented model (232); and

[0202] Information (202) is displayed on a live view (212) of a physical object (204) viewed through a portable computing device (208), wherein the information (202) is identified using an augmented model (232) of the physical object (204).

[0203] Item 2. An augmented reality system (216) as described in Item 1, wherein a computer system (218) controls the movement of an unmanned vehicle group (220) relative to a physical object (204) and generates an image (222) of the physical object (204) and scan data (224) of spatial points describing an area (226) of the physical object (204).

[0204] Item 3. The augmented reality system (216) of any one of Items 1-2, wherein when creating an augmented model (232) of a physical object (204) using the image (222) and the scan data (224), the computer system (218) operates to:

[0205] creating a model (302) of the physical object (204) using the image (222);

[0206] Creating a plurality of point clouds (302) from scan data (224) generated by a plurality of unmanned vehicles (220) in the unmanned vehicle group (220); and

[0207] The model (302) of the physical object (204) is modified using the plurality of point clouds (302) to form an enhanced model (232).

[0208] Item 4. An augmented reality system (216) as described in any of Items 1-3, wherein the unmanned vehicle group (220) operates to generate images (222) and scan data (224) when a human operator (206) views a live view (212) of a physical object (204) through a portable computing device (208).

[0209] Item 5. An augmented reality system (216) as described in any of Items 1-4, wherein the computer system (218) selects an area (226) of the physical object (204) and controls multiple unmanned vehicles (220) in the unmanned vehicle group (220) to generate scanning data (224) of the area (226) of the physical object (204).

[0210] Item 6. An augmented reality system (216) as described in Item 5, wherein, when selecting an area (226) of a physical object (204), the computer system (218) selects the area (226) of the physical object (204) based on a gaze point (400) of a human operator (206) using a portable computing device (208).

[0211] Item 7. An augmented reality system (216) as described in Item 5, wherein, when selecting the area (226) of the physical object (204), the computer system (218) selects the area (226) of the physical object (204) based on the location (402) of the task (214) performed by a human operator (206) using a portable computing device (208), wherein the area (226) surrounds the location (402).

[0212] Item 8. An augmented reality system (216) as described in any of Items 1-7, wherein the computer system (218) receives at least one of an additional image (228) of the physical object (204) or additional scan data (240) of the physical object (204) from the portable computing device (208); and

[0213] Wherein, when creating an enhanced model (232) of a physical object (204) using the image (222) and the scan data (224), the computer system (218) creates the enhanced model (232) of the physical object (204) using the image (222), the additional image (228), the scan data (224), and the additional scan data (240).

[0214] Item 9. An augmented reality system (216) as described in any of Items 1-8, wherein the information (202) is selected from at least one of task information, an assembly, a video, a non-conformance indication, a work instruction, an exploded view of an assembly, or a schematic diagram.

[0215] Item 10. An augmented reality system (216) as described in any of Items 1-9, wherein the physical object (204) is selected from the group consisting of an airplane, a building, a bridge, a dam, a vehicle, a field, a lake, a mountain, an engine, a fuselage portion, and a runway.

[0216] Item 11. An augmented reality system (216) as described in any of Items 1-10, wherein the portable computing device (208) is selected from the group consisting of smart glasses, a mobile phone, a tablet computer, and a head-mounted display.

[0217] Item 12. An augmented reality system as described in any of Items 1-11, wherein the unmanned vehicle group (220) is selected from at least one of an unmanned aerial vehicle, an unmanned aerial vehicle, an unmanned ground vehicle, or an unmanned water vehicle.

[0218] Item 13. An augmented reality system (216), comprising:

[0219] A computer system (218), wherein the computer system (218) communicates with the unmanned vehicle group (220) using a communication link (228) during operation of the computer system (218) and during operation of the unmanned vehicle group (220);

[0220] A visualizer (230) in a computer system (218) operable to:

[0221] Receiving an image (222) of the physical object (204) from an unmanned vehicle group (220) moving relative to the physical object (204);

[0222] Receiving scan data (224) of an area (226) of the physical object (204) from a plurality of unmanned vehicles (220) in a group (220) of unmanned vehicles moving relative to the physical object (204);

[0223] creating an enhanced model (232) of the physical object (204) using the image (222) and the scan data (224), wherein regions (226) of the physical object (204) in the enhanced model (232) have greater detail than other regions (234) of the physical object (204) in the enhanced model (232); and

[0224] Information (202) is sent to a portable computing device (208), wherein the information (202) is displayable by the portable computing device (208) on a live view (212) of a physical object (204) viewed through the portable computing device (208), wherein the information (202) is identified using an augmented model (232) of the physical object (204).

[0225] Item 14. The augmented reality system (216) of Item 13, wherein the visualizer (230) operates to:

[0226] Controlling the unmanned vehicle group (220) to move relative to the physical object (204); and

[0227] The unmanned vehicle group (220) is controlled to generate an image (222) of the physical object (204) and scan data (224) of spatial points describing an area (226) of the physical object (204).

[0228] Item 15. An augmented reality system (216) as described in any of Items 13-14, wherein the visualizer (230) operates to select an area (226) of a physical object (204) and control multiple unmanned vehicles (220) in an unmanned vehicle group (220) to generate scan data (224) of the area (226) of the physical object (204).

[0229] Item 16. An augmented reality system (216) as described in Item 15, wherein, when selecting an area (226) of a physical object (204), the visualizer (230) operates to select the area (226) of the physical object (204) based on a gaze point (400) of a human operator (206) using a portable computing device (208).

[0230] Item 17. An augmented reality system (216) as described in Item 15, wherein, when selecting the area (226) of the physical object (204), the visualizer (230) operates to select the area (226) of the physical object (204) based on the location (402) of the task (214) performed by a human operator (206) using a portable computing device (208), wherein the area (226) surrounds the location (402).

[0231] Item 18. An augmented reality system as described in any of the preceding items, wherein the unmanned vehicle group (220) is selected from at least one of an unmanned aerial vehicle, an unmanned aerial vehicle, an unmanned ground vehicle, or an unmanned water vehicle.

[0232] Item 19. A method for visualizing information (202) on a live view (212) of a physical object (204), the method comprising:

[0233] Receiving, by a computer system (218), an image (222) of a physical object (204) from an unmanned vehicle group (220) moving relative to the physical object (204), wherein the computer system (218) communicates with the unmanned vehicle group (220) using a communication link (228);

[0234] Receiving, by a computer system (218), scan data (224) for a region (226) of a physical object (204);

[0235] creating, by a computer system (218), an enhanced model (232) of the physical object (204) using the image (222) and the scan data (224), wherein the region (226) in the enhanced model (232) has greater detail than other regions (234) of the physical object (204) in the enhanced model (232);

[0236] sending, by the computer system (218), at least a portion of the augmented model (232) to the portable computing device (208), wherein the portable computing device (208) is positioned relative to the physical object (204) using at least a portion of the augmented model (232); and

[0237] Information (202) displayable by a portable computing device (208) is sent by a computer system (218), wherein the portable computing device (208) displays the information (202) on a live view (212) of a physical object (204) seen by the portable computing device (208), and wherein the information (202) is identified using an augmented model (232) of the physical object (204).

[0238] Item 20. The method of Item 19, further comprising:

[0239] Controlling the unmanned vehicle group (220) to move relative to the physical object (204) by a computer system (218); and

[0240] The unmanned vehicle group (220) is controlled to generate an image (222) of the physical object (204) and scan data (224) of spatial points describing an area (226) of the physical object (204).

[0241] Item 21. The method of any one of Items 19-20, wherein creating an enhanced model (232) of the physical object (204) using the image (222) and the scan data (224) comprises:

[0242] creating a model (302) of the physical object (204) using the image (222);

[0243] Creating a plurality of point clouds (302) from scan data (224) generated by a plurality of unmanned vehicles (220) in the unmanned vehicle group (220); and

[0244] A model (302) of a physical object (204) is modified using the plurality of point clouds (302) to form an enhanced model (232).

[0245] Item 22. A method as described in any of Items 19-21, wherein the unmanned vehicle group (220) operates to generate images (222) and scan data (224) when a human operator (206) views a live view (212) of a physical object (204) via a portable computing device (208).

[0246] Item 23. The method of any one of Items 19-22, further comprising:

[0247] selecting, by the computer system (218), a region (226) of the physical object (204); and

[0248] A plurality of unmanned vehicles (220) in an unmanned vehicle group (220) are controlled by a computer system (218) to generate scan data (224) of an area (226) of a physical object (204).

[0249] Item 24. The method of Item 23, wherein selecting, by the computer system (218), the region (226) of the physical object (204) comprises:

[0250] A region (226) of a physical object (204) is selected by a computer system (218) based on a gaze point (400) of a human operator (206) using a portable computing device (208).

[0251] Item 25. The method of Item 23, wherein selecting, by the computer system (218), the region (226) of the physical object (204) comprises:

[0252] A region (226) of a physical object (204) is selected by a computer system (218) based on a location (402) of a task (214) performed by a human operator (206) using a portable computing device (208), wherein the region (226) encompasses the location (402).

[0253] Item 26. The method of any one of Items 19-24, further comprising:

[0254] receiving, by the computer system (218), at least one of an additional image (228) of the physical object (204) or additional scan data (240) of the physical object (204) from the portable computing device (208); and

[0255] Wherein creating, by the computer system (218), an enhanced model (232) of the physical object (204) using the image (222) and the scan data (224) includes:

[0256] An enhanced model (232) of the physical object (204) is created by a computer system (218) using the image (222), the additional image (228), the scan data (224), and the additional scan data (240).

[0257] Item 27. The method of any of Items 19-26, wherein the information (202) is selected from at least one of task information, an assembly, a video, a nonconformance indication, a work instruction, an exploded view of an assembly, or a schematic diagram.

[0258] Item 28. A method as described in any of Items 19-27, wherein the physical object (204) is selected from the group consisting of an airplane, a building, a bridge, a dam, a vehicle, a field, a lake, a mountain, an engine, a fuselage portion, and a runway.

[0259] Item 29. A method as described in any of Items 19-28, wherein the portable computing device (208) is selected from the group consisting of smart glasses, a mobile phone, a tablet computer, and a head-mounted display.

[0260] Thus, one or more illustrative examples overcome the technical problem of displaying information to enhance a live view of a physical object, reducing the amount of processing resources used compared to currently used techniques for generating point clouds of the physical object. Thus, one or more illustrative examples may provide the technical effect of reducing the amount of computing resources used to create a model of the physical object using two types of data. In the illustrative examples, using image and scan data reduces the amount of computing resources used compared to current techniques that use only point clouds.

[0261] The descriptions of different illustrative embodiments are presented for the purpose of illustration and description and are not intended to be exhaustive or limited to the embodiments of the disclosed forms. The different illustrative examples describe components that perform actions or operations. In the illustrative embodiments, the components can be configured to perform the described actions or operations. For example, the components can have a construction or design that enables the components to perform the actions or operations described as being performed by the components in the illustrative examples.

[0262] Many modifications and variations will be apparent to those of ordinary skill in the art. Different illustrative embodiments may provide different features compared to other desired embodiments. The selected one or more embodiments are chosen and described in order to best explain the principles of the embodiments, the practical application, and to enable others of ordinary skill in the art to understand the disclosure of various embodiments with various modifications suitable for the intended specific use.

Claims

1. An augmented reality system (216), comprising: an unmanned vehicle assembly (220) operable to move relative to a physical object (204), generate an image (222) of the physical object (204), and generate scan data (224) of spatial points describing an area (226) of the physical object (204); A computer system (218) in communication with the unmanned vehicle array (220) using a communication link (228), wherein the computer system (218) is operative to: Receiving the image (222) of the physical object (204) from the unmanned vehicle group (220) moving relative to the physical object (204); Receiving the scan data (224) for the area (226) of the physical object (204) from a plurality of unmanned vehicles (220) in the group (220) of unmanned vehicles moving relative to the physical object (204); creating an enhanced model (232) of the physical object (204) using the image (222) and the scan data (224), wherein the region (226) of the physical object (204) in the enhanced model (232) has greater detail than other regions (234) of the physical object (204) in the enhanced model (232); identifying nonconformities in the regions and updating a model of the physical object generated prior to the augmenting the model based on a scan of the altered structure of the physical object; classifying said nonconformity of said altered structure; and A portable computing device (208) operable to: positioning relative to the physical object (204) using the augmented model (232); and Information (202) is displayed on a live view (212) of the physical object (204) as viewed through the portable computing device (208), wherein the information (202) is identified using the augmented model (232) of the physical object (204), the information (202) identifying a nonconformity in the area and an action to be performed on the nonconformity in the area.

2. The augmented reality system (216) of claim 1, wherein: The computer system (218) controls the movement of the unmanned vehicle group (220) relative to the physical object (204) and generates the image (222) of the physical object (204) and the scan data (224) describing the spatial points of the area (226) of the physical object (204).

3. The augmented reality system (216) of any one of claims 1-2, wherein: In creating the augmented model (232) of the physical object (204) using the image (222) and the scan data (224), the computer system (218) operates to: creating a model (300) of the physical object (204) using the image (222); Creating a plurality of point clouds (302) from the scan data (224) generated by a plurality of unmanned vehicles (220) in the set of unmanned vehicles (220); and The model (300) of the physical object (204) is modified using the plurality of point clouds (302) to form the enhanced model (232).

4. The augmented reality system (216) of any one of claims 1 to 3, wherein: The unmanned vehicle assembly (220) operates to generate the image (222) and the scan data (224) while a human operator (206) views a live view (212) of the physical object (204) via the portable computing device (208).

5. The augmented reality system (216) of any one of claims 1 to 4, wherein: The computer system (218) selects the area (226) of the physical object (204) and controls a plurality of unmanned vehicles (220) in the unmanned vehicle group (220) to generate the scan data (224) of the area (226) of the physical object (204).

6. The augmented reality system (216) of claim 5, wherein: In selecting the region (226) of the physical object (204), the computer system (218) selects the region (226) of the physical object (204) based on a gaze point (400) of a human operator (206) using the portable computing device (208).

7. The augmented reality system (216) of claim 5, wherein: In selecting the area (226) of the physical object (204), the computer system (218) selects the area (226) of the physical object (204) based on a location (402) of a task (214) performed by a human operator (206) using the portable computing device (208), wherein the area (226) surrounds the location (402).

8. The augmented reality system (216) of any one of claims 1 to 7, wherein: The computer system (218) receives at least one of an additional image (238) of the physical object (204) and additional scan data (240) of the physical object (204) from the portable computing device (208); and Wherein, when creating the enhanced model (232) of the physical object (204) using the image (222) and the scan data (224), the computer system (218) creates the enhanced model (232) of the physical object (204) using the image (222), the additional image (238), the scan data (224), and the additional scan data (240).

9. The augmented reality system (216) of any one of claims 1 to 8, wherein: The information (202) is selected from at least one of task information, an assembly, a video, a nonconformance indication, a work instruction, an exploded view of an assembly, and a schematic diagram; wherein the physical object (204) is selected from the group consisting of an airplane, a building, a bridge, a dam, a vehicle, a field, a lake, a mountain, an engine, a fuselage section, and a runway; Wherein, the portable computing device (208) is selected from the group including smart glasses, mobile phones, tablet computers and head-mounted displays; and wherein, the unmanned vehicle group (220) is selected from at least one of an unmanned aerial vehicle, a drone, an unmanned ground vehicle and an unmanned water vehicle.

10. A method for visualizing information (202) on a live view (212) of a physical object (204), the method comprising: Receiving, by a computer system (218), an image (222) of a physical object (204) from an unmanned vehicle group (220) moving relative to the physical object (204), wherein the computer system (218) communicates with the unmanned vehicle group (220) using a communication link (228); Receiving, by the computer system (218), scan data (224) of an area (226) of the physical object (204); creating, by the computer system (218), an enhanced model (232) of the physical object (204) using the image (222) and the scan data (224), wherein the region (226) in the enhanced model (232) has greater detail than other regions (234) of the physical object (204) in the enhanced model (232); sending, by the computer system (218), at least a portion of the augmented model (232) to a portable computing device (208), wherein the portable computing device (208) is positioned relative to the physical object (204) using at least a portion of the augmented model (232); identifying, by the computer system (218), nonconformities in the region and updating a model of the physical object generated prior to the augmenting the model based on a scan of the altered structure of the physical object; categorizing, by the computer system (218), the nonconformity of the altered structure; and The information (202) is sent by the computer system (218) to be displayed by the portable computing device (208), wherein the portable computing device (208) displays the information (202) on a live view (212) of the physical object (204) as seen by the portable computing device (208), and wherein the information (202) is identified using the augmented model (232) of the physical object (204), the information (202) identifying a nonconformity in the area and an action to be performed on the nonconformity in the area.

11. The method of claim 10, further comprising: Controlling the unmanned vehicle group (220) to move relative to the physical object (204) by the computer system (218); as well as The unmanned vehicle group (220) is controlled to generate the image (222) of the physical object (204) and the scan data (224) describing spatial points of the area (226) of the physical object (204).

12. The method according to any one of claims 10 to 11, wherein: Creating the enhanced model (232) of the physical object (204) using the image (222) and the scan data (224) includes: creating a model (300) of the physical object (204) using the image (222); Creating a plurality of point clouds (302) from the scan data (224) generated by a plurality of unmanned vehicles (220) in the set of unmanned vehicles (220); and The model (300) of the physical object (204) is modified using the plurality of point clouds (302) to form the enhanced model (232).

13. The method according to any one of claims 10 to 12, further comprising: The area (226) of the physical object (204) is selected by the computer system (218) based on at least one of: a gaze point (400) of a human operator (206) using the portable computing device (208), or a location (402) based on a task (214) performed by the human operator (206) using the portable computing device (208); and The computer system (218) controls a plurality of unmanned vehicles (220) in the unmanned vehicle group (220) to generate the scan data (224) of the area (226) of the physical object (204).

14. The method according to any one of claims 10 to 13, further comprising: receiving, by the computer system (218), from the portable computing device (208), at least one of an additional image (238) of the physical object (204) and additional scan data (240) of the physical object (204); and Wherein, creating, by the computer system (218), the enhanced model (232) of the physical object (204) using the image (222) and the scan data (224) comprises: The enhanced model (232) of the physical object (204) is created by the computer system (218) using the image (222), the additional image (238), the scan data (224), and the additional scan data (240).

15. The method according to any one of claims 10 to 14, wherein: The information (202) is selected from at least one of task information, an assembly, a video, a nonconformance indication, a work instruction, an exploded view of an assembly, and a schematic diagram; wherein the physical object (204) is selected from the group consisting of an airplane, a building, a bridge, a dam, a vehicle, a field, a lake, a mountain, an engine, a fuselage section, and a runway; and Wherein, the portable computing device (208) is selected from the group consisting of smart glasses, mobile phones, tablet computers, and head-mounted displays.

Images