Automatic monitoring and inspection of assembly processes

Abstract

The present invention relates to automatic supervision and inspection of assembly processes. A method and apparatus for performing automatic supervision and inspection of assembly processes. The method is implemented using a computer system. A sensor system positioned relative to an assembly site is used to generate sensor data at the assembly site. The sensor data is used to generate a three-dimensional global map of the assembly site and assemblies being built at the assembly site. The three-dimensional global map is used to identify the current stage of the assembly process for building assemblies at the assembly site. A context for the current stage is identified. A quality report for the assembly is generated based on the three-dimensional global map and the context for the current stage.

Description

Technical Field

[0001] The present disclosure generally relates to inspecting assembly sites. More specifically, the present disclosure relates to methods and systems for automated supervision and inspection of assembly processes at an assembly site. Background Art

[0002] Composite structures, such as aircraft structures, may require the assembly of hundreds, thousands, tens of thousands, or even hundreds of thousands of parts. The assembly of these types of composite structures may present certain challenges in a manufacturing environment. For example, different types or levels of inspection may be required at various stages of the assembly process. One inspection step may include determining whether the parts being assembled at a particular stage are indeed the parts expected to be assembled. Another inspection step may include determining whether the parts have been assembled in the correct position. Yet another inspection step may include determining whether the parts have been assembled at the correct time during the assembly process. One inspection step may include determining whether the assembly of the parts meets safety requirements, quality requirements, or both.

[0003] Some currently available inspection methods involve manual inspection of the assembly by a human operator. However, this type of manual inspection can be prone to human error and accuracy can be lower than desired. In addition, manual inspection can take longer than desired and can be more expensive than desired. For example, when an undesirable feature is identified via manual inspection, disassembly may be required in order to identify the cause of the undesirable feature. This disassembly process can be more time consuming and more expensive than desired. Therefore, one or more devices and methods for addressing the above problems would be desirable. Summary of the invention

[0004] In one example embodiment, a method for performing automatic supervision and inspection of an assembly process is provided. The method is implemented using a computer system. Sensor data at the assembly site is generated using a sensor system positioned relative to the assembly site. A three-dimensional global map is generated using the sensor data for the assembly site and the assemblies being built at the assembly site. The three-dimensional global map is used to identify the current stage of the assembly process for building assemblies at the assembly site. The context for the current stage is identified. A quality report for the assembly is generated based on the three-dimensional global map and the context for the current stage.

[0005] In another example embodiment, a method for performing automatic supervision and inspection of an assembly process is provided. The method is implemented using a computer system. Sensor data at the assembly site is generated using a sensor system positioned relative to the assembly site. A three-dimensional global map of the assembly site and the assembly parts being built at the assembly site is generated using the sensor data. The current stage of the assembly process for building the assembly parts at the assembly site is identified using the three-dimensional global map and the assembly information stored in a data repository. The background is identified for the current stage. A quality report corresponding to the current stage of the assembly process is generated based on the three-dimensional global map, the assembly information, and the background for the current stage. Records corresponding to the current stage of the assembly process are stored in the data repository to thereby build a history of records for the assembly process. The record includes at least a portion of the sensor data and the quality report.

[0006] In another example embodiment, a system for automatic supervision and inspection of an assembly process includes a sensor system and a computer system. The sensor system is positioned relative to an assembly site and an assembly being built at the assembly site. The computer system uses sensor data generated by the sensor system to generate a three-dimensional global map of the assembly site and the assembly being built at the assembly site. The computer system identifies a current stage of the assembly process for building the assembly at the assembly site based on the three-dimensional global map. The computer system identifies a context for the current stage. The computer system generates a quality report for the assembly based on the three-dimensional global map and the context for the current stage.

[0007] The features and functions can be achieved independently in various embodiments of the present disclosure or may be combined in yet other embodiments, further details of which can be seen with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The features believed to be novel features of the example embodiments are set forth in the appended claims. However, the example embodiments together with the preferred modes of use, further objects and features thereof will be best understood by reference to the following detailed description of example embodiments when read in conjunction with the accompanying drawings, wherein:

[0009] Figure 1 is a block diagram of a manufacturing environment according to an example embodiment.

[0010] Figure 2 According to an example embodiment, Figure 1 Block diagram of the training manager and data repository.

[0011] Figure 3 According to an example embodiment, Figure 1 Block diagram of the assembly manager.

[0012] Figure 4 is an illustration of an assembly site according to an example embodiment.

[0013] Figure 5 is a flow chart of a process for performing automated supervision and inspection of an assembly process according to an example embodiment.

[0014] Figure 6 is a flow chart of a process for performing automated supervision and inspection of an assembly process according to an example embodiment.

[0015] Figure 7 is a flow diagram of a process for identifying a current stage of an assembly process according to an example embodiment.

[0016] Figure 8 is a flow chart of a process for performing rollback verification according to an example embodiment.

[0017] Fig. 9 is a flow chart of a process for automated supervision and inspection of an assembly process according to an example embodiment.

[0018] Fig.10 is a flow chart of a process for performing automated supervision and inspection of an assembly process according to an example embodiment.

[0019] Fig.11 is a flow chart of a process for performing automated supervision and inspection of an assembly process according to an example embodiment.

[0020] Fig.12 is a flow chart of a process for generating a three-dimensional global map of assembly sites and assemblies according to an example embodiment.

[0021] Fig.13 is a flow chart of a process for performing automated supervision and inspection of an assembly process according to an example embodiment.

[0022] Fig.14 is an illustration of a three-dimensional point cloud according to an illustrative embodiment.

[0023] Fig.15 According to the illustrative embodiment, Fig.14 Illustration of points in a three-dimensional point cloud where the points are clustered.

[0024] Fig.16 According to the illustrative embodiment, the use of Fig.15 Illustration of a surface patch formed by clusters of .

[0025] Fig.17 is an illustration of a three-dimensional global map according to an illustrative embodiment.

[0026] Fig.18 is a diagram of objects and location elements that have been detected in a three-dimensional global map in accordance with an illustrative embodiment.

[0027] Fig.19 is a block diagram of a data processing system according to an example embodiment.

[0028] Fig. 20 is an illustration of an aircraft manufacturing and service method according to an illustrative embodiment of an example embodiment.

[0029] Fig.21 is a block diagram of an aircraft according to an example embodiment. DETAILED DESCRIPTION

[0030] The example embodiments described below provide methods and systems for automating the supervision and inspection of assembly processes at an assembly site. These methods and systems use machine learning, computer vision, and other artificial intelligence techniques to automate the supervision and inspection of assembly processes. In addition, these methods and systems provide an automatic way to identify parts that are assembled during the assembly process according to predetermined quality requirements, in the correct position, and at the correct time.

[0031] The automated monitoring and inspection system provided by the example embodiments improves the quality control of the assembly process in a manner that reduces the need for human intervention in the inspection process. By reducing this need for human intervention, human-related errors can be reduced. In addition, the automated monitoring and inspection system provides cost and time savings. For example, the automated monitoring and inspection system provides a method for identifying the root cause of a problem without requiring extensive disassembly or reversal of actual assembly steps.

[0032] Figure 1 1 is a block diagram of a manufacturing environment according to an example embodiment. Manufacturing environment 100 is an example of one type of manufacturing environment in which plurality of parts 102 may be assembled to form structure 104. Structure 104 may take many different forms. For example, structure 104 may be an aircraft, a wing, a wing structure, a fuselage, a fuselage structure, a frame, a stabilizer, or some other type of structure 104. In some cases, parts in plurality of parts 102 may also be referred to as pieces, components, elements, members, or some other type of units.

[0033] Plurality of parts 102 may be assembled at an assembly location 105 within manufacturing environment 100. In some illustrative examples, assembly location 105 may be one of a plurality of assembly locations within manufacturing environment 100. Assembly location 105 may include, for example, a floor area, a platform, a stand, or some other type of location on which plurality of parts 102 are assembled.

[0034] The plurality of parts 102 may be assembled to form the structure 104 according to an assembly process 106. The assembly process 106 may include a plurality of stages 108. As used herein, a stage in the plurality of stages 108 may be a single stage of the assembly process 106, a series of stages, a series of sub-stages, a stage in a sub-assembly process, a step in the assembly process 106 or a sub-assembly process, or two or more steps to be performed in parallel during the assembly process 106. At any given stage of the assembly process 106, before and until completion, the structure 104 may be referred to as an assembly 110. In some cases, the assembly 110 may also be referred to as a "build."

[0035] The monitoring and inspection system 112 is used to monitor the assembly process 106 and perform inspections during the assembly process 106. The monitoring and inspection system 112 can be an automatic system. Specifically, the monitoring and inspection system 112 automates the monitoring of the assembly process 106 and automates the inspections during the assembly process 106 to reduce or in some cases even eliminate the need for human intervention.

[0036] In these illustrative examples, the supervision and inspection system 112 includes a sensor system 114, a training manager 116, an assembly manager 118, a rollback verifier 120, a data repository 121, and an output system 122. The sensor system 114 may include any number of sensors or combinations of sensors. For example, the sensor system 114 may include at least one of an imaging device, a radar sensor, a sonar sensor, a light detection and ranging (LiDAR) sensor, or some other type of sensor. The imaging device may include, for example, but not limited to, a camera, a stereo camera, a video camera, an infrared camera, some other type of imaging device, or a combination thereof. In an illustrative example, the sensor system 114 includes a group of imaging systems, each of which includes one or more imaging devices. A group of items (such as a group of imaging systems) may include one or more items.

[0037] The training manager 116, the assembly manager 118, and the rollback validator 120 may be implemented using software, hardware, firmware, or a combination thereof. When software is used, the operations performed by each of the training manager 116, the assembly manager 118, or the rollback validator 120 may be implemented using, for example, but not limited to, program code configured to run on a processor unit. When firmware is used, the operations performed by each of the training manager 116, the assembly manager 118, or the rollback validator 120 may be implemented using, for example, but not limited to, program code and data and stored in a persistent memory to run on a processor unit.

[0038] When hardware is used, the hardware may include one or more circuits that operate to perform the operations performed by each of the training manager 116, the assembly manager 118, or the rollback validator 120. Depending on the implementation, the hardware may take the form of a circuit system, an integrated circuit, an application specific integrated circuit (ASIC), a programmable logic device, or some other suitable type of hardware device configured to perform any number of operations. A programmable logic device may be configured to perform certain operations. The device may be permanently configured to perform these operations, or may be configurable. A programmable logic device may take the form of, for example, but not limited to, a programmable logic array, a programmable array logic, a field programmable logic array, a field programmable gate array, or some other type of programmable hardware device.

[0039] In these illustrative examples, assembly manager 118 and rollback validator 120 may be implemented using computer system 124. Computer system 124 may include a single computer or multiple computers in communication with each other. In some embodiments, computer system 124 may include a laptop, a tablet computer, a smart phone, or some other type of mobile device.

[0040] In some embodiments, a portion of computer system 124 may be remotely located relative to another portion of computer system 124. For example, computer system 124 may include a computer located at or around assembly site 105 within manufacturing environment 100 and a server system remotely located relative to the computer. The server system may be located within or outside of manufacturing environment 100. In some cases, training manager 116 may be implemented using a first computer, assembly manager 118 may be implemented using a second computer, and rollback validator 120 may be implemented using a third computer. Thus, computer system 124 may be used to implement assembly manager 118 and rollback validator 120 in many different ways.

[0041] Data repository 121 may include one or more databases, federated memories, some other type of data structure, or a combination thereof. Data repository 121 may communicate with training manager 116, assembly manager 118, rollback validator 120, or a combination thereof. In some examples, data repository 121 may be part of computer system 124. In some examples, data repository 121 or at least a portion of data repository 121 is remotely located relative to assembly site 105.

[0042] The data repository 121 stores assembly information 126. Figure 2 Examples of the types of assembly information 126 that may be stored in data repository 121 are described in .

[0043] Output system 122 may communicate with training manager 116, assembly manager 118, rollback validator 120, or a combination thereof. Output system 122 may include one or more output devices. In some illustrative examples, a portion or all of output system 122 may be considered to be part of computer system 124. In these illustrative examples, output system 122 may include at least one of display system 128, audio system 130, or some other type of output device. Display system 128 may include, for example, at least one of a screen, a touch screen, a monitor, a head mounted display device, or some other type of display device. Audio system 130 may include, for example, at least one of a microphone, a speaker, or some other type of audio device.

[0044] Sensor system 114 may be used to generate sensor data 132 before assembly process 106, during assembly process 106, or both. Sensor data 132 may include two-dimensional (2D) imaging data, three-dimensional (3D) imaging data, one or more other types of sensor data, or a combination thereof. Sensor data 132 is sent to training manager 116 and assembly manager 118 for processing. In some illustrative examples, a portion or all of sensor data 132 is stored in data repository 121.

[0045] Before assembly process 106 begins, training manager 116 trains assembly manager 118 using assembly information 126 and, optionally, sensor data 132 to identify various parts in plurality of parts 102 and various stages in plurality of stages 108 that are to be assembled to form structure 104. Additionally, training manager 116 trains assembly manager 118 using assembly information 126 and, optionally, sensor data 132 to evaluate quality metrics for assembly 110.

[0046] The assembly manager 118 processes the sensor data 132 to identify a current stage 134 of the assembly process 106. The assembly manager 118 then identifies a context 136 for the current stage 134. The context 136 may include, for example, at least one of a nominal condition for the assembly 110 at the current stage 134, a nominal condition for each part expected to be present at the current stage 134, a tolerance selected for the nominal condition for the assembly 110 at the current stage 134, a tolerance selected for a previously identified quality metric for the assembly 110, an identification of the step(s) completed prior to the current stage 134, an identification of the step(s) to be completed during the current stage 134, an identification of the next step(s) to be completed after the current stage 134, or other information.

[0047] The context 136 may include information that a human operator would not be able to recognize at all or would not be able to quickly and easily recognize based on observing or visualizing the assembly site 105. For example, when the assembly process 106 includes numerous stages, the human operator may not be able to easily identify the current stage 134 or the tolerances selected for the quality metrics previously identified for the assembly 110 for the current stage 134, the step(s) completed before the current stage 134, the step(s) to be completed during the current stage 134, or the next step(s) to be completed after the current stage 134.

[0048] Based on the context 136 identified for the current stage 134, the assembly manager 118 can output guidance information 138 via the output system 122. The guidance information 138 can include, for example, a list of steps to be performed during the current stage 134, a list of next steps to be performed after the current stage 134, or both. The guidance information 138 is used to inform one or more human operators of the tasks to be performed during the assembly process 106.

[0049] Additionally, based on the context 136 identified for the current stage 134, the assembly manager 118 can evaluate the assembly 110 at the current stage 134 and generate a quality report 140. The quality report 140 may also be referred to as a quality assessment. The quality report 140 indicates any relevant issues of interest related to the quality of the assembly 110.

[0050] For each stage of the assembly process 106 identified by the assembly manager 118, the assembly manager 118 stores a corresponding quality report 140 in the data repository 121 to build a history 142 of reports for the assembly process 106. If, at the current stage 134, the assembly manager 118 detects a problem of interest, the rollback validator 120 can roll back through the history 142 of quality reports stored in the data repository 121 in reverse order to identify the root cause of the problem of interest.

[0051] Figure 2 According to an example embodiment, Figure 1 A block diagram of the training manager 116 and data repository 121 of FIG. Figure 1 Before the assembly process 106 of the assembly site 105 is executed, the training manager 116 uses the assembly information 126 stored in the data repository 121 to train the assembly process 106 of the assembly site 105. Figure 1 Assembly Manager 118 in Figure 1 Each of the plurality of stages 108 is evaluated.

[0052] The training manager 116 may use any number of algorithms and techniques to perform such training. For example, the training manager 116 may use at least one of machine learning, deep learning, computer vision, some other type of computational learning, some other type of artificial intelligence learning, or a combination thereof.

[0053] The assembly information 126 stored in the data repository 121 may include various types of information. For example, the assembly information 126 may include a part image library 202, a part model library 204, assembly process flow data 206, assembly site model data 208, assembly model data 210, and assembly video data 212, feature data 214, or a combination thereof.

[0054] The part image library 202 may include, for example, Figure 1 One or more images of at least a portion of a part in plurality of parts 102. In some cases, part image library 202 may include an image for each of plurality of parts 102. In these illustrative examples, part image library 202 may also include images of parts captured from previously completed assembly processes.

[0055] The part model library 204 may include, for example, Figure 1 One or more models of at least a portion of a part in plurality of parts 102. In some cases, part model library 204 may include a model for each of plurality of parts 102. The model of a part may be, for example, a computer-aided design (CAD) model.

[0056] The assembly process flow data 206 may include information about Figure 1 The assembly process flow data 206 includes information about the flow or order of the plurality of stages 108 in the assembly process 106. In one illustrative example, the assembly process flow data 206 takes the form of a directed assembly graph 216 that represents the flow or order according to which the plurality of stages 108 of the assembly process 106 should proceed. In some cases, the directed assembly graph 216 may include sub-flows that represent sub-stages that are performed in parallel with each other.

[0057] Assembly site model data 208 may include, for example, Figure 1 The assembly site model data 208 is a reference model of the assembly site 105 of the assembly process 106, and is updated to the reference model for each of the plurality of stages 108 of the assembly process 106. In some illustrative examples, the assembly site model data 208 includes a model of the assembly site 105 for each of the plurality of stages 108. The models included in the assembly site model data 208 may be, for example, three-dimensional models. In addition, in some cases, the models may include representations of the parts involved and the preparation parts for each of the plurality of stages 108. In other cases, the models are linked to the CAD models of the parts.

[0058] Assembly model data 210 may include any number of models representing various stages of assembly 110 in plurality of stages 108. In one illustrative example, assembly model data 210 includes a baseline model for assembly 110 at a first stage in plurality of stages 108, and updates to the baseline model to reflect the addition of one or more parts to assembly 110 at each of plurality of stages 108. Assembly video data 212 may include a video of the entire assembly process 106, or a plurality of videos for various stages of assembly process 106 captured during previously executed assembly process 106.

[0059] Feature data 214 may include information about various features. Features may include cracks, dents, distortions, surface level features, other types of features, or combinations thereof. Feature data 214 may include images of these features. In these illustrative examples, feature data 214 includes selected tolerances for each of the features. As an illustrative example, feature data 214 may include a selected tolerance for the length of a crack, a selected tolerance for the depth of a dent, and other types of selected tolerances.

[0060] In these illustrative examples, training manager 116 may use sensor data 132 generated over time to update assembly information 126. This type of updating improves the overall accuracy and efficiency with which analysis of sensor data 132 may be performed.

[0061] Figure 3 According to an example embodiment, Figure 1 The assembly manager 118 may include a data analyzer 302 and an output generator 304.

[0062] Data analyzer 302 uses sensor data 132 and assembly information 126 stored in data repository 121 to identify each of plurality of stages 108 during assembly process 106. For example, at any given point in time during assembly process 106, data analyzer 302 can use any number of algorithms or techniques to identify current stage 134 of assembly process 106.

[0063] In one illustrative example, the sensor data 132 includes imaging data 306 for the assembly site 105. The imaging data 306 includes a two-dimensional image, a three-dimensional image, a video, or a combination thereof of the assembly site 105. For example, the sensor data 132 may include imaging data 306 in the form of two-dimensional imaging data 307, which may include a two-dimensional image, a two-dimensional video, or both. In other illustrative examples, the sensor data 132 includes three-dimensional data 305, which may include data generated by at least one of a LiDAR sensor, a radar sensor, a sonar sensor, or a stereo camera. The three-dimensional data 305 may include range data. Therefore, any number of these types of sensors or a combination of these types of sensors can be positioned relative to the assembly site 105 and used to generate the three-dimensional data 305.

[0064] When imaging data 306 includes two-dimensional imaging data 307 , data analyzer 302 may detect and identify object 308 in imaging data 306 using at least one of machine learning, deep learning, computer vision, customized learning techniques, some other type of computational learning, some other type of artificial intelligence learning, or a combination thereof in conjunction with assembly information 126 .

[0065] Data analyzer 302 matches detected object 308 to a part based on assembly information 126. For example, data analyzer 302 may use part image library 202, part model library 204, or both to match object 308 to a part. This matching may be performed based on, for example, form, size, color, identifying marks, or some other type of comparison point.

[0066] Once objects 308 are part matched, they can be referred to as matched objects 309. In some cases, not all objects 308 can be matched to corresponding parts. Therefore, matched objects 309 can include all or a subset of objects 308.

[0067] The data analyzer 302 identifies the position 310 and orientation 312 of the matched objects 309 relative to the reference coordinate system for the assembly site 105. For example, the data analyzer 302 may first identify the position 310 of the matched objects 309 in the specific image. The data analyzer 302 may synchronize the perspective at which the image is generated to the corresponding position of the imaging device that generates the image relative to the model of the assembly site 105. Ray tracing or some other technique may then be used to determine the position of each matched object. The orientation of the matched object may then be determined. In other illustrative examples, the data analyzer 302 identifies the position 310 and orientation 312 of the matched objects 309 relative to the reference coordinate system for the assembly 110.

[0068] Data analyzer 302 can identify one or more features 314 in imaging data 306. Features 314 can be features of a specific part, surface, or other portion of assembly 110. Features 314 can include, for example, but not limited to, cracks, dents, distortions, surface level features, or some other type of features. Surface level features can be, for example, flush levels, smooth levels, or some other type of surface features. When the feature is outside of a selected tolerance, the feature can be considered an undesirable feature or inconsistency.

[0069] Data analyzer 302 may detect and identify object 308 in imaging data 306 using at least one of machine learning, deep learning, computer vision, a customized learning technique, some other type of computational learning, some other type of artificial intelligence learning, or a combination thereof in conjunction with assembly information 126 .

[0070] In some illustrative examples, data analyzer 302 also identifies site elements 315 in imaging data 306. Site elements 315 may include tools, identification marks, labels, robots, and other elements other than parts that may be present at assembly site 105.

[0071] Using some combination of the identification of the matched objects 309, the locations 310 of the matched objects 309, the orientations 312 of the matched objects 309, and the place elements 315, the data analyzer 302 can then identify the current stage 134 of the assembly process 106. For example, the data analyzer 302 can use the assembly information 126 in conjunction with the above information to identify the current stage 134. More specifically, the data analyzer 302 can use the information from Figure 2 The current stage 134 may be identified using assembly process flow data 206, assembly site model data 208, assembly model data 210, assembly video data 212, some other type of information, or a combination thereof. In some cases, identification of the current stage 134 may be referred to as assembly position calibration or assembly stage calibration.

[0072] The current stage 134 may also be identified using the sensor data 132 in the form of the three-dimensional data 305. When the sensor data 132 includes the three-dimensional data 305, the data analyzer 302 uses the three-dimensional data 305 to generate a three-dimensional point cloud 317 of the assembly site 105 and the assembly 110 being built at the assembly site 105. The data analyzer 302 then uses the three-dimensional point cloud 317 to generate a three-dimensional global map 319. The three-dimensional global map 319 may be a representation of the assembly site 105 and the assembly 110 being built at the assembly site 105. This representation integrates information from the three-dimensional data 305 with one or more three-dimensional models (e.g., CAD models) included in the assembly information 126 stored in the data repository 121, for example.

[0073] The data analyzer 302 uses the three-dimensional global map 319 to detect objects 308 and, in some cases, place elements 315. The data analyzer 302 can then match the objects 308 using processes similar to those described above to identify matching objects 309. The data analyzer 302 can then identify the current stage 134 based on the matching objects 309 in a manner similar to that described above.

[0074] In these example embodiments, objects 308, features 314, and place elements 315 may be identified using a three-dimensional global map 319 and at least one three-dimensional comparison algorithm. For example, three-dimensional global map 319 may be compared to Figure 2 The three-dimensional global map 319 can be compared with the assembly site model data 208, the assembly model data 210, or a portion of both stored in ... Figure 2 The assembly video data 212, the feature data 214, or both are analyzed.

[0075] Use of three-dimensional global map 319 may allow different or additional objects, features, or elements of a location to be identified compared to use of two-dimensional imaging data 307. For example, some features that are more difficult to detect through processing of two-dimensional imaging data 307 may be more easily or more easily detectable through use of three-dimensional global map 319 generated through processing of three-dimensional data 305.

[0076] As an illustrative example, certain types of dents, shims, connections, fasteners, smoothing, and welds may be more easily or better detected using three-dimensional global map 319. In some cases, at least one of the objects, features, or site elements that cannot be detected through processing of two-dimensional imaging data 307 may be detected using three-dimensional global map 319.

[0077] Once current stage 134 has been identified, data analyzer 302 uses context information 316 stored in data repository 121 to identify context 136 for current stage 134. Context information 316 may also include a portion of assembly information 126. For example, context information 316 may include quality metrics for each of stages 108 of assembly process 106.

[0078] As discussed above, the context 136 for the current stage 134 may include at least one of a nominal condition for the assembly 110 at the current stage 134, a nominal condition for each part expected to exist at the current stage 134, tolerances selected for the nominal condition for the assembly 110 at the current stage 134, tolerances selected for pre-identified quality metrics for the assembly 110, identification of step(s) completed prior to the current stage 134, identification of step(s) to be completed during the current stage 134, identification of next step(s) to be completed after the current stage 134, or other information.

[0079] Once context 136 has been identified, output generator 304 generates guidance information 138 and quality report 140. Guidance information 138 is used to inform one or more human operators involved in assembly process 106. Guidance information 138 may include, for example, current stage guidance 318, next stage guidance 320, or both.

[0080] The current stage guidance 318 may include, for example, a list of steps to be performed for the current stage 134. In some cases, the current stage guidance 318 includes instructions on how to perform certain tasks for the current stage 134. The current stage guidance 318 may include one or more images, videos, or both to help guide a human operator on how to perform specific tasks. The guidance information 138 may be provided by a user interface. Figure 1 In some cases, the current stage guide 318 includes audio guidance, such as audio instructions on how to perform certain tasks during the current stage 134 of the assembly process 106. These audio instructions can be directed to a human operator through an audio signal from Figure 1 The audio system 130 is played.

[0081] The next phase guide 320 may include, for example, a list of steps to be performed during the next phase that will follow the current phase 134. The data analyzer 302, the output generator 304, or both may use, for example, Figure 2The next stage can be identified by the directional assembly diagram 216 of the embodiment. In some illustrative examples, the next stage guidance 320 includes instructions on how to perform certain tasks for the next stage. The next stage guidance 320 can include one or more images, videos, or both to help guide a human operator on how to perform a specific task.

[0082] The boot information 138 can be obtained by Figure 1 The output system 122 described in the above is presented to the human operator. For example, the guidance information 138 can be presented from Figure 1 In some cases, the current stage guide 318 may include audio guidance, such as audio instructions on how to perform certain tasks during the current stage 134 of the assembly process 106. These audio instructions may be directed to a human operator via an audio signal from Figure 1 The audio system 130 is played.

[0083] Additionally, based on the context 136, matched objects 309, positions 310, orientations 312, and locale elements 315 identified for the current stage 134, the output generator 304 evaluates the assembly 110 at the current stage 134 and generates a quality report 140. The quality report 140 indicates whether any relevant issues of interest exist regarding the quality of the assembly 110. For example, the quality report 140 includes at least one of a match evaluation 322, a position evaluation 324, an inconsistency evaluation 326, or some other type of evaluation.

[0084] Match evaluation 322 may indicate whether one or more of objects 308 cannot be matched to a corresponding part. In some illustrative examples, match evaluation 322 may also indicate whether one or more of matched objects 309 do not match a part that is expected to be present at the current stage 134 of assembly process 106. Position evaluation 324 indicates whether matched objects 309 have been assembled in a correct or incorrect position relative to assembly 110. Inconsistency evaluation 326 indicates whether feature 314 has been identified, and if so, whether feature 314 is outside of a selected tolerance. Data analyzer 302, output generator 304, or both may use information from Figure 2 The feature data 214 is used to determine whether the feature 314 is outside of a selected tolerance.

[0085] In some cases, the quality report 140 may include an overall stage evaluation 328. The overall stage evaluation 328 may be, for example, an indication of whether the current stage 134 is progressing within a selected tolerance. In some cases, the overall stage evaluation 328 may be a grade selected from one of "pass", "borderline", and "fail".

[0086] The quality report 140 can be presented to the human operator in different ways via the output system 122. In one illustrative example, the quality report 140 can be presented using a graphic indicator superimposed on an image of the assembly site 105 or assembly 110 displayed in the display system 128. For example, an object identified in the image that has been matched to the correct part and matched to the correct position and orientation can be indicated using a first type of graphic indicator (e.g., a green circle around the object, a green arrow, etc.). An object identified in the image that has been mismatched, in the wrong position, or has a wrong orientation can be indicated using a second type of graphic indicator (e.g., a red circle around the object, a red arrow, etc.). As described above, a mismatched object can be an object that is matched to a part that should not exist at the current stage 134 or an object that is not matched to any part.

[0087] Sensor data 132 and various pieces of information generated by assembly manager 118 are stored in data repository 121. For example, a stage record 330 can be stored in data repository 121 in association with current stage 134. Stage record 330 can include sensor data 132, identification of object 308, identification of matched object 309, position 310, orientation 312, and quality report 140. In some cases, stage record 330 can include guidance information 138. Once stored in data repository 121, stage record 330 becomes part of history 142. Thus, this type of history 142 of reports includes a report for each of the plurality of stages 108 identified and analyzed.

[0088] Figure 1-Figure 3 The illustrations in the drawings are not meant to imply physical or architectural limitations on the manner in which the example embodiments may be implemented. Other components in addition to or in place of the components illustrated may be used. Some components may be optional. In addition, the blocks are presented to illustrate some functional components. When implemented in different example embodiments, one or more of these blocks may be combined, separated, or combined and divided into different blocks.

[0089] Figure 4 is a diagram of an assembly site according to an example embodiment. Assembly site 400 may be for Figure 1 An example of an embodiment of assembly site 105 described in FIG. 4. Wing assembly 402 is being constructed at assembly site 400 to ultimately form a wing. Wing assembly 402 is a wing assembly for a wing assembly that is being constructed at assembly site 400. Figure 1 An example of one implementation for assembly 110 is depicted in FIG.

[0090] As depicted, monitoring and inspection system 406 resides at least partially at assembly site 400. Monitoring and inspection system 406 is directed to Figure 1 An example of an implementation of the monitoring and inspection system 112 in FIG. The monitoring and inspection system 406 includes an inspection platform 408 and an inspection platform 410. In this illustrative example, the inspection platform 408 and the inspection platform 410 can be implemented using a fixed platform. In other illustrative examples, the inspection platform 408 and the inspection platform 410 can be implemented using a mobile platform. Although in Figure 4 Only two platforms are shown in FIG. 4 , and any number of platforms may be present at assembly site 400 .

[0091] In this illustrative example, inspection platform 408 includes rotating camera 412 and control system 414. Similarly, inspection platform 410 includes rotating camera 416 and control system 418. Rotating cameras 412 and 416 are for Figure 1 4. Rotating camera 412 and rotating camera 416 can generate two-dimensional images. In other illustrative examples, inspection platform 408 and inspection platform 410 can be equipped with a three-dimensional camera system, a LiDAR sensor system, or other types of sensors.

[0092] Control system 414 and control system 418 may each include a processor, a memory, and a communication unit. Control system 414 and control system 418 may control the operation of rotating camera 412 and rotating camera 416, respectively. In addition, control system 414 and control system 418 may process the imaging data generated by these cameras and send the processed data to a remote computer system (e.g., Figure 1 In other illustrative examples, the processors of control system 414 and control system 418 are used as processors in Figure 1 Part of the computer system 124 in communication with each other.

[0093] In some cases, lighting devices (not shown) may be attached to or integral with each of inspection platform 408 and inspection platform 410. The lighting devices may help rotating camera 412 and rotating camera 416 capture higher quality images.

[0094] The imaging data generated by rotating camera 412 and rotating camera 416 include images of assembly site 400 and wing assembly 402 that are used to perform automated inspection during construction of wing assembly 402. Additionally, the imaging data is used to provide guidance to human operator 420 based on the current stage of assembly.

[0095] Figure 5 is a flow chart of a process for performing automated supervision and inspection of an assembly process according to an example embodiment. Figure 5 The process 500 illustrated in FIG. 5 may be used in Figure 1-Figure 3 The process 500 may be used to enable Figure 1 The supervision and inspection of the assembly process 106 at the assembly site 105 in the system is automated.

[0096] Process 500 can start with generating sensor data at an assembly site using a sensor system positioned relative to the assembly site (operation 502). In one illustrative example, the sensor system includes a plurality of cameras positioned relative to the assembly site and the assembly being built at the assembly site. In some cases, one or more of the cameras may be a rotating camera.

[0097] The current stage of an assembly process for building an assembly at an assembly site is identified using the sensor data (operation 504). Thereafter, a context is identified for the current stage of the assembly process (operation 506). As described above, the stage of an assembly process can be a single stage, a sequence of stages, a sequence of sub-stages, a stage in a sub-assembly process, a step in an assembly process, a step in a sub-assembly process, or two or more steps or sub-stages to be performed in parallel during an assembly process.

[0098] A quality report is then generated for the assembly based on the sensor data and the context for the current stage (operation 508), with the process terminating thereafter. The quality report is an automatic report that includes an evaluation indicating whether at least one problem of interest exists. The problem of interest may be an improper part, improper part location, improper orientation, cracks, dents, distortions, surface inconsistencies, or a combination thereof. Process 500 may be repeated for each stage of the assembly process.

[0099] Figure 6 is a flow chart of a process for performing automated supervision and inspection of an assembly process according to an example embodiment. Figure 6 The process 600 illustrated in FIG. 6 can be used in Figure 1-Figure 3 The process 600 may be used to enable Figure 1 The supervision and inspection of the assembly process 106 at the assembly site 106 in the system is automated.

[0100] Process 600 begins with generating sensor data at the assembly site using a sensor system positioned relative to the assembly site (operation 602). In these illustrative examples, the sensor system includes multiple cameras placed at predetermined fixed points relative to the assembly parts being built at the assembly site. These positions are selected to provide images that provide the most information and the best angle for evaluating the assembly process. The camera can generate images, videos, or both.

[0101] Next, a current stage of an assembly process for building an assembly at an assembly site is identified using sensor data, assembly information stored in a data repository, and at least one of machine learning, deep learning, computer vision, customized learning techniques, computational learning, or artificial intelligence learning (operation 604). Thereafter, a context is identified for the current stage (operation 606).

[0102] A quality report corresponding to the current stage of the assembly process is generated based on the sensor data and the context for the current stage (operation 608). A record corresponding to the current stage of the assembly process is stored in a data repository to thereby build a history of records for the assembly process (operation 610), wherein the process terminates thereafter. The record includes at least a portion of the sensor data and the quality report.

[0103] Figure 7 is a flow chart of a process for identifying a current stage of an assembly process according to an example embodiment. Figure 7 The process 700 illustrated in FIG. 7 can be used in Figure 1-Figure 3 More specifically, process 700 may be performed using the monitoring and inspection system 112 described in Figure 1 and Figure 3 Executed by the assembly manager 118 described in .

[0104] Process 700 begins by inspecting a plurality of objects using imaging data generated by an imaging system (operation 702). The imaging system may include one or more cameras positioned relative to an assembly site and assemblies at the assembly site.

[0105] Next, at least a portion of the plurality of objects is matched to a part for the assembly to thereby create a plurality of matched objects (operation 704). Operation 704 may be performed using at least one of an image of the part or a model of the part. For example, operation 704 may be performed using Figure 2202 and the part model library 204 in the assembly. In operation 704, each object is compared with a known part for the assembly and is matched to the known part or marked as unmatched. In these illustrative examples, matching at least a portion of the plurality of objects to known parts for the assembly can be performed using at least one of machine learning, deep learning, computer vision, customized learning techniques, computational learning, or artificial intelligence learning.

[0106] Thereafter, at least one of a plurality of positions or a plurality of orientations is identified for a plurality of matched objects relative to a reference coordinate system for the assembly site (operation 706). For example, for each object matched to a known part for an assembly being built, a position, an orientation, or both relative to a reference coordinate system for the assembly site is calculated. In some illustrative examples, the reference coordinate system may be relative to the assembly itself. The positions of the matched objects may be identified using a known position of the camera relative to the assembly site and therefore a known position of the camera relative to a model (e.g., a CAD model) of the assembly or assembly site.

[0107] A current stage of the assembly process is determined based on the identification of the plurality of matching objects and at least one of the plurality of positions or the plurality of orientations identified for the plurality of matching objects (operation 708), with the process terminating thereafter. Operation 708 may be performed by comparing the positions, orientations, or both of the matching objects to models of assembly parts at various stages during the assembly process, models of assembly sites at various stages during the assembly process, or both. Figure 2 The assembly process flow data 206, the assembly site model data 208, the assembly model data 210, the assembly video data 212, or a combination thereof may be used to determine the current stage of the assembly process.

[0108] Figure 8 is a flow chart of a process for performing rollback verification according to an example embodiment. Figure 8 The process 800 illustrated in FIG. 8 can be used in Figure 1-Figure 3 More specifically, process 800 may be performed using the monitoring and inspection system 112 described in Figure 1 The rollback validator 120 described in is executed.

[0109] Process 800 may begin by determining whether a quality report for the current stage of the assembly process indicates that a problem of interest exists (operation 802). The problem of interest may be one of an improper part, improper part position, improper orientation, cracks, dents, distortions, surface inconsistencies, or some other type of problem of interest. An improper part means that the wrong part has been added to the assembly. An improper part position may mean that the correct part has been assembled or added to the assembly in the wrong location. An improper orientation may mean that the correct part has the wrong orientation relative to the assembly.

[0110] If the problem of interest does not exist, the process terminates. Otherwise, if the problem of interest does exist, the current stage of the assembly process is selected for processing (operation 804). The report corresponding to the selected stage is retrieved from the history in the data repository and analyzed (operation 806).

[0111] Thereafter, a determination is made as to whether the root cause of the problem of interest occurred during the selected stage of the assembly process (operation 808). If the root cause occurred during the selected stage of the assembly process, a report is generated (operation 810), with the process terminating thereafter. The report may be an alert or warning with information about the problem of interest. In some cases, the report includes images, video, or both to better inform a human operator of the problem.

[0112] However, if the root cause does not occur during the selected phase of the assembly process, a determination is made as to whether the selected phase is an initial phase of the assembly process (operation 812). If the selected phase is an initial phase of the assembly process, the process proceeds to operation 810 described above. Here, the report may take the form of a notification, an alarm, or a substantive report indicating that the root cause could not be identified.

[0113] If the selected stage is not the initial stage of the assembly process, then a stage of the assembly process that occurred immediately prior to the selected stage is selected for processing (operation 814), wherein the process then returns to operation 806 described above. In this way, the process 800 rolls back through the history of records stored in the data repository in an efficient manner. Information about previous assembly steps can be analyzed and evaluated without requiring human intervention or requiring extensive disassembly (e.g., reversal of assembly steps that have already been performed).

[0114] Thus, this type of rollback verification allows for automated inspection and quality control of the assembly process in a manner that would be too difficult or time consuming for a human operator to perform. A human operator may have to reverse the assembly process and disassemble a portion of the assembly in order to identify the root cause of a problem. However, the rollback verification described above is an automated method for easily, quickly, and efficiently identifying the root cause of a problem that occurred prior to the current stage of assembly.

[0115] Fig. 9 is a flow chart of a process for automated supervision and inspection of an assembly process according to an example embodiment. Fig. 9 The process 900 illustrated in FIG. 1 may be used in Figure 1-Figure 3 This is performed by the supervision and inspection system 112 described in.

[0116] The process 900 may begin by identifying the current stage of the assembly process and the context for the current stage of the assembly process (operation 902). Next, an activated part is identified based on imaging data (operation 904). The imaging data may be generated by one or more cameras in a fixed orientation relative to the assembly site. The activated part may be, for example, a part being held by a human operator or a robot for installation. The identification performed in operation 904 may be performed using machine learning, computer vision, or other types of learning techniques.

[0117] Thereafter, an activated target location is identified for the activated part based on the imaging data (operation 906). The activated target location may be identified using machine learning, computer vision, or other types of learning techniques, and a 3D model of the assembly site or assembly to be built. In some illustrative examples, once the operator begins the task of installing the part, the target location may be identified by proximity-based detection combined with activated assembly detection.

[0118] Next, the activated part and the activated target location are matched to known parts for the assembly (operation 908). A determination is made as to whether this match is appropriate (operation 910). A match can be considered appropriate when the combination of the activated part and the activated target location is correct for the current stage of the assembly process.

[0119] If the match is appropriate, the imaging data is used to determine whether a problem of interest exists (operation 912). This determination can be made based on machine learning, computer vision, or other types of learning algorithms. In operation 912, the determination is made based on known types of problems and predetermined quality metrics.

[0120] If a problem exists, a rollback verification process is performed to identify the root cause of the problem, and a corresponding report is generated (operation 914). The process then determines whether the current phase of the assembly process is still active (operation 916). If there are additional steps or tasks to be performed as part of this phase of the assembly process, the current phase is considered to be still active. If the current phase is still active, the process returns to operation 904 described above. Otherwise, the process returns to operation 902 described above.

[0121] Referring again to operation 912, if a question of interest does not exist, the process proceeds to operation 916. Referring again to operation 910, if a match is not appropriate, a report is generated (operation 918), wherein the process then returns to operation 904 as described above. This allows a human operator or a robot to select a new active part or a new active target location.

[0122] By returning to operation 902 after operation 918, the process can "advance" the active stage of the assembly process and use the new context for decision making. When there is no next stage to advance to, the assembly process is complete. Thus, the process repeats until the assembly process has been completed.

[0123] Fig.10 is a flow chart of a process for performing automated supervision and inspection of an assembly process according to an example embodiment. Fig.10 The process 1000 illustrated in FIG. 1 may be used in Figure 1-Figure 3 The process 1000 may be performed by the monitoring and inspection system 112 described in . The process 1000 may be used to automate the monitoring and inspection of the assembly process 106 .

[0124] Process 1000 may begin by generating sensor data at an assembly site using a sensor system positioned relative to the assembly site (operation 1002). A three-dimensional global map of the assembly site and assemblies being built at the assembly site is generated using the sensor data (operation 1004).

[0125] Thereafter, a current stage of an assembly process for building the assembly is identified based on the three-dimensional global map (operation 1006). A context is then identified for the current stage (operation 1008). A quality report for the assembly can then be generated using the three-dimensional global map and the context for the current stage (operation 1010), with the process terminating thereafter. Process 1000 can be repeated at various stages or at each stage of the assembly process.

[0126] Fig.11 is a flow chart of a process for performing automated supervision and inspection of an assembly process according to an example embodiment. Fig.11 The process 1100 illustrated in FIG. 1 may be used in Figure 1-Figure 3 The process 1100 may be used to enable Figure 1 The supervision and inspection of the assembly process 106 at the assembly site 105 in the system is automated.

[0127] Process 1100 begins with generating sensor data at the assembly site using a sensor system positioned relative to the assembly site (operation 1102). In these illustrative examples, the sensor system includes multiple cameras placed at predetermined fixed points relative to the assembly parts being built at the assembly site. These positions are selected to provide images that provide the most information and the best angle for evaluating the assembly process. The camera can generate images, videos, or both.

[0128] Next, a three-dimensional global map of the assembly site and the assemblies being built at the assembly site is generated using the sensor data (operation 1104). Operation 1104 can be performed in many different ways. As an illustrative example, the sensor data can include point cloud data. The point cloud data can be used in its original form or processed to form a three-dimensional point cloud for the assembly site. One or more processing algorithms (e.g., edge detection, segmentation, mapping, etc.) can be used to process the point cloud data and generate a three-dimensional point cloud for use in generating a three-dimensional global map.

[0129] The current stage of the assembly process for building the assembly at the assembly site is then identified using the three-dimensional global map and the assembly information stored in the data repository (operation 1106). Operation 1106 may be performed, for example, by detecting objects and site elements in the three-dimensional global map.

[0130] Thereafter, a context is identified for the current stage (operation 1108). A quality report corresponding to the current stage of the assembly process is generated based on the three-dimensional global map, the assembly information, and the context for the current stage (operation 1110). A record corresponding to the current stage of the assembly process is stored in a data repository to thereby establish a history of records for the assembly process (operation 1112), wherein the process is terminated thereafter. The record may include at least a portion of the sensor data, the three-dimensional global map, the quality report, or a combination thereof.

[0131] Although process 1100 is described as using a three-dimensional global map to identify the current stage of assembly, other techniques may also be used. In some illustrative examples, operation 1104 may be optionally omitted, and the current stage of assembly may be identified by processing the three-dimensional point cloud described above. The three-dimensional point cloud may be processed using any number of algorithms or techniques or a combination thereof for detecting objects and place elements. For example, edge detection, segmentation, mapping, feature extraction, other such techniques or a combination thereof may be used to process the three-dimensional point cloud. The information generated by these techniques may then be compared with the expected or optimal information corresponding to the same current stage of assembly as defined by the background for the current stage. The result of this comparison may be a part of the quality report generated in operation 1110.

[0132] Fig.12 is a flow chart of a process for generating a three-dimensional global map of assembly sites and assemblies according to an example embodiment. Fig.12 The process 1200 illustrated in FIG. 1 may be used in Figure 1-Figure 3 More specifically, process 1200 may be performed using the monitoring and inspection system 112 described in Figure 3 In addition, the process 1200 is for Fig.10 Operation 1004 and Fig.11 An example of one implementation of operation 1104 in .

[0133] Process 1200 may begin by using three-dimensional data to generate a three-dimensional point cloud of an assembly site and assemblies being constructed at the assembly site (operation 1202). The three-dimensional data may be generated, for example, by at least one of a light detection and ranging sensor, a radar sensor, a sonar sensor, or a stereo camera positioned relative to the assembly site and the assemblies at the assembly site. In addition, the three-dimensional data may be generated substantially simultaneously and may be generated continuously or periodically. The three-dimensional data may be, for example, at Figure 3 Operation 1202 may include fusing data from multiple sensors to generate a three-dimensional point cloud. The three-dimensional cloud may be, for example, Figure 3 3D point cloud in 317.

[0134] Next, the points in the three-dimensional point cloud are clustered to thereby form a set of surface patches (operation 1204). Operation 1204 may be performed using, for example, at least one of a clustering algorithm or a surface reconstruction algorithm. Each surface patch is formed by a local point cluster in the three-dimensional cloud. In an illustrative example, operation 1204 is performed by clustering the points in the three-dimensional point cloud and then converting these clusters into three-dimensional surfaces. The surface patch may include one or more of these three-dimensional surfaces.

[0135] Thereafter, the set of surface patches is fitted to surfaces in the three-dimensional model of the assembly site and the assembly being constructed at the assembly site (operation 1206). In operation 1206, the three-dimensional model may include one or more CAD models. In some illustrative examples, operation 1206 may be performed by determining which surfaces in the three-dimensional model are most closely matched by the surface patches. The fitting in operation 1206 may be performed using at least one of radial basis functions, spline approximations, or mesh interpolations.

[0136] In operation 1206, fitting the surface patch to the surface of the three-dimensional model may include modifying or updating the surface of the three-dimensional model based on corresponding points in the surface patch or the three-dimensional point cloud used to form the surface patch to thereby form a fitted surface patch. In other illustrative examples, fitting may include modifying or updating the surface patch based on the three-dimensional model to thereby form a fitted surface patch. In some cases, such fitting includes identifying a position (position and orientation) of the surface patch relative to a reference coordinate system for the three-dimensional model.

[0137] The set of fitted surface patches is then organized into a three-dimensional global map (operation 1208), with the process terminating thereafter. In one illustrative example, operation 1208 may be performed by, for example, superimposing the surface patches on their corresponding surfaces in the three-dimensional model to form a three-dimensional global map. In other illustrative examples, the surface patches may be aligned relative to a reference coordinate system for the three-dimensional model to form a three-dimensional global map.

[0138] The three-dimensional global map generated in operation 1208 can be used to identify the current stage of the assembly process and generate an assessment of the quality of the assembly process and the assembly parts being built via the assembly process at the assembly site. Figure 3 Features 314 in the image may be identified using the three-dimensional global map. Using the three-dimensional global map may allow different features, additional features, or both to be identified. For example, some features that would be difficult to detect through processing of two-dimensional imaging data may be easier or more easily detectable through processing of three-dimensional data.

[0139] Fig.13 is a flow chart of a process for performing automated supervision and inspection of an assembly process according to an example embodiment. Process 1300 may be used to automate supervision and inspection of assembly process 106.

[0140] Process 1300 may begin by generating sensor data at an assembly site using a sensor system positioned relative to the assembly site, the sensor data including two-dimensional imaging data and three-dimensional data (operation 1302). The sensor system may include any number or combination of sensors positioned relative to the assembly site. The sensor may generate sensor data continuously or periodically.

[0141] For example, the sensor system may include one or more fixed platforms for generating two-dimensional imaging data at fixed points around the assembly site, within the assembly site, or both. These fixed platforms may be equipped with cameras, which in some cases may be rotating cameras. The sensor system may also include one or more sensor devices capable of generating three-dimensional data. For example, the sensor system may include at least one of a LiDAR sensor, a radar sensor, a sonar sensor, or a stereo camera.

[0142] At least one of the two-dimensional imaging data or the three-dimensional data is used to identify a current stage of the assembly process and a context for the current stage (operation 1304). Thereafter, a quality report is generated using the two-dimensional imaging data and the context for the current stage (operation 1306). The quality report is generated using the context for the current stage and a three-dimensional global map generated based on the three-dimensional data (operation 1308).

[0143] The two quality reports are then integrated to form a master quality report (operation 1310). In an illustrative example, operation 1310 can be performed by comparing the two quality reports. The various evaluations included in each quality report can be fused. For example, inconsistent evaluations in the two quality reports can be fused to remove redundant detections of undesirable features, and where appropriate, the detections can be merged into a single detection.

[0144] The master quality report is delivered for observation by at least one human operator (operation 1312). Operation 1312 can be performed in various ways. As an example, a report detailing the results of the assessment of the current stage of the assembly process can be delivered on site. In other examples, the information in the master quality report can be displayed in a graphical user interface on a display system in a manner that visually conveys the results of the assessment. For example, a graphical indicator superimposed on a three-dimensional model of an assembly at the assembly site can be used to indicate a problem of interest. For example, a colored circle, arrow, or other type of graphical indicator can be used to indicate a wrong part, a part that has been assembled in the wrong position, a dent, a crack, a distortion, a surface inconsistency, other types of problems, or a combination thereof.

[0145] In an illustrative example, the master quality report can be sent to at least one remote device via at least one communication link for observation by at least one human operator. For example, operations 1304 to 1310 can be performed on a remote server located outside the manufacturing environment. Once the master quality report has been generated, the master quality report can be sent to a device (e.g., a mobile device, such as a laptop, a tablet computer, a smart phone, etc.) located at the assembly site so that the human operator at the assembly site can use the information in the master quality report to solve any problem of interest. In other illustrative examples, operation 1310 is omitted so that operation 1312 is modified so that two quality reports are delivered to (one or more) human operators for observation.

[0146] A determination is made as to whether the assembly process is complete (operation 1314). If the assembly process is not complete, the process returns to operation 1304 described above. Otherwise, the process terminates.

[0147] The flow chart and block diagram in the different embodiments described illustrate the architecture, function and operation of some possible implementations of the equipment and method in the illustrative embodiment.In this regard, each square block in the flow chart or block diagram can represent a part of a module, section, function, and / or operation or step.

[0148] In some alternative implementations of the illustrative examples, one or more functions annotated in the blocks may not appear in the order annotated in the figure. For example, in some cases, two blocks shown in succession may be executed substantially simultaneously, or blocks may sometimes be executed in reverse order, depending on the functions involved. Moreover, blocks other than the blocks shown may be added to the flow chart or block diagram.

[0149] Fig.14 is a diagram of a three-dimensional point cloud according to an illustrative embodiment. The three-dimensional point cloud 1400 is for Figure 3 An example of an implementation of the three-dimensional point cloud 317 in FIG. In addition, the three-dimensional point cloud 1400 may be Fig.12 14. An example of a three-dimensional point cloud generated in operation 1202 of FIG. The three-dimensional point cloud 1400 is generated using data from multiple LiDAR sensors.

[0150] Fig.15 According to the illustrative embodiment, Fig.14 1400, wherein the points are clustered. Points 1500 of the three-dimensional point cloud 1400 have been clustered into clusters 1502. Clusters 1502 of points 1500 may represent clusters that may be Figure 1 The type of clustering performed in operation 1204 in .

[0151] Fig.16 According to the illustrative embodiment, the use of Fig.15 1502 is a diagram of a surface patch formed by a cluster of 1502. The surface patch 1600 may be a Figure 1 Examples of the types of surface patches formed in operation 1204 in FIG.

[0152] Fig.17 is a diagram of a three-dimensional global map according to an illustrative embodiment. The three-dimensional global map 1700 is for Figure 3 An example of an implementation of the three-dimensional global map 319 in FIG. In addition, the three-dimensional global map 1700 may be Fig.12 1700 is an example of a three-dimensional global map 1700 generated in operation 1208 in FIG. Fig.16 The surface patch 1600 in FIG. 1 is fitted to a three-dimensional model (eg, a CAD model) to form a three-dimensional global map 1700 .

[0153] Fig.18 is an illustration of objects and locale elements that have been detected in a three-dimensional global map in accordance with an illustrative embodiment. Object 1800 and locale element 1802 have been detected within three-dimensional global map 1700 .

[0154] Now turn to Fig.19 , depicts in block diagram form a diagram of a data processing system according to an illustrative embodiment. Data processing system 1900 may be used to implement Figure 1 As depicted, data processing system 1900 includes communications framework 1902, which provides communications between processor unit 1904, storage device 1906, communication unit 1908, input / output unit 1910, and display 1912. In some cases, communications framework 1902 may be implemented as a bus system.

[0155] Processor unit 1904 is configured to execute instructions for software to perform a variety of operations. Depending on the implementation, processor unit 1904 may include multiple processors, a multi-core processor, and / or some other type of processor. In some cases, processor unit 1904 may take the form of a hardware unit, such as a circuit system, an application specific integrated circuit (ASIC), a programmable logic device, or some other suitable type of hardware unit.

[0156] Instructions for the operating system, applications, and / or programs executed by processor unit 1904 may be located in storage device 1906. Storage device 1906 may communicate with processor unit 1904 via communication framework 1902. As used herein, a storage device (also referred to as a computer-readable storage device) is any piece of hardware capable of storing information on a temporary and / or permanent basis. The information may include, but is not limited to, data, program code, and / or other information.

[0157] Memory 1914 and persistent storage 1916 are examples of storage devices 1906. Memory 1914 may take the form of, for example, random access memory or some type of temporary or non-temporary storage device. Persistent storage 1916 may include any number of components or devices. For example, persistent storage 1916 may include a hard drive, a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination of the above. The media used by persistent storage 1916 may or may not be removable.

[0158] Communications unit 1908 allows data processing system 1900 to communicate with other data processing systems and / or devices. Communications unit 1908 may provide communications using physical and / or wireless communications links.

[0159] Input / output unit 1910 allows data to be received from, and output to be sent to, other devices of data processing system 1900. For example, input / output unit 1910 may allow user input to be received via a keyboard, a mouse, and / or some other type of input device. As another example, input / output unit 1910 may allow output to be sent to a printer connected to data processing system 1900.

[0160] The display 1912 is configured to display information to the user. The display 1912 may include, for example but not limited to, a monitor, a touch screen, a laser display, a holographic display, a virtual display device, and / or some other type of display device.

[0161] In this illustrative example, the processes of the different illustrative embodiments may be performed using computer-implemented instructions by processor unit 1904. These instructions may be referred to as program code, computer usable program code, or computer readable program code, and may be read and executed by one or more processors in processor unit 1904.

[0162] In these examples, program code 1918 is provided in functional form on selectively removable computer readable media 1920 and can be loaded or transferred to data processing system 1900 for execution by processor unit 1904. Program code 1918 and computer readable media 1920 together form computer program product 1922. In this illustrative example, computer readable media 1920 may be computer readable storage media 1924 or computer readable signal media 1926.

[0163] Computer readable storage media 1924 is a physical or tangible storage device used to store program code 1918 rather than a medium that propagates or transmits program code 1918. Computer readable storage media 1924 may be, for example but not limited to, an optical or magnetic disk or a persistent storage device connected to data processing system 1900.

[0164] Alternatively, program code 1918 may be transferred to data processing system 1900 using computer readable signal media 1926. Computer readable signal media 1926 may be, for example, a propagated data signal containing program code 1918. The data signal may be an electromagnetic signal, an optical signal, and / or some other type of signal that may be propagated over a physical and / or wireless communication link.

[0165] Fig.19 The illustration of data processing system 1900 in FIG. 1 is not meant to provide architectural limitations to the manner in which the illustrative embodiments may be implemented. Different illustrative embodiments may be implemented in a data processing system including components in addition to or in place of those illustrated for data processing system 1900. Additionally, Fig.19 The components shown in the figures may vary from the graphical examples shown.

[0166] Illustrative embodiments of the present disclosure may be found in Fig. 20 Aircraft manufacturing and service method 2000 as shown in Fig.21 2100 is described in the context of the aircraft 2100 shown in FIG. Fig. 20 , depicts an illustration of a block diagram of an aircraft manufacturing and service method according to an illustrative embodiment. During trial production, aircraft manufacturing and service method 2000 may include Fig.21 Specification and design of aircraft 2100 in 2002 and material procurement 2004.

[0167] During production, Fig.21 Component and subassembly manufacturing 2006 and system integration 2008 of the aircraft 2100. Thereafter, Fig.21 The aircraft in 2100 can be certified and delivered in 2010, so that it can enter service in 2021. When used by customers, Fig.21The aircraft 2100 is scheduled for routine maintenance and servicing 2014 (which may also include modification, reconfiguration, refurbishment, and other maintenance and servicing).

[0168] Each of the processes of aircraft manufacturing and service method 2000 may be performed or carried out by a system integrator, a third party, and / or an operator. In these examples, the operator may be a customer. For purposes of this description, a system integrator may include, but is not limited to, any number of aircraft manufacturers and major system subcontractors; a third party may include, but is not limited to, any number of sellers, subcontractors, and suppliers; and an operator may be an airline, a leasing company, a military entity, a service organization, etc.

[0169] Now refer to Fig.21 , depicts a diagram of an aircraft in which an illustrative embodiment may be implemented. In this example, aircraft 2100 is Fig. 20 21. The aircraft manufacturing and service method 2000 in the present invention may be produced and may include an airframe 2102 and a plurality of systems 2104 and an interior 2106. Examples of systems 2104 include one or more of a propulsion system 2108, an electrical system 2110, a hydraulic system 2121, and an environmental system 2114. Any number of other systems may be included. Although an aviation example is shown, different illustrative embodiments may be applied to other industries, such as marine and automotive industries.

[0170] The apparatus and methods embodied herein may be used in Fig. 20 During at least one of the stages of aircraft manufacturing and service method 2000, the method is employed. Specifically, Figure 1 The structure 104 may be manufactured during any of the stages of aircraft manufacturing and service method 2000. For example, without limitation, assembly 110 may be built using plurality of parts 102 during at least one of component and subassembly manufacturing 2006, system integration 2108, routine maintenance and service 2014, or some other stage of aircraft manufacturing and service method 2000. Additionally, monitoring and inspection system 112 may provide automated monitoring and inspection of assembly process 106 for building assembly 110. Monitoring and inspection system 112 may be used to automate monitoring and inspection of a structure that is part of fuselage 2102, interior 2106, or some other portion of aircraft 2100.

[0171] In an illustrative example, Fig. 20 The components and subassemblies produced in the component and subassembly manufacturing 2006 can be produced in a manner similar to that when the aircraft 2100 is in Fig. 20As another example, one or more apparatus embodiments, method embodiments, or a combination thereof may be produced or manufactured during a production phase (such as Fig. 20 One or more apparatus embodiments, method embodiments, or a combination thereof may be used when the aircraft 2100 is in service 2112, and / or during operation of the aircraft 2100. Fig. 20 Used during maintenance and service 2014. Use of a number of different illustrative embodiments may substantially expedite assembly of aircraft 2100, and / or reduce the cost of aircraft 2100, or both.

[0172] Thus, example embodiments provide automated systems and methods for monitoring and inspecting an assembly process, although the automated systems and methods for monitoring and inspecting may be used for assembly of other types of vehicles and vehicle structures (eg, automotive assembly, spacecraft assembly, etc.).

[0173] Example embodiments provide systems and methods for mapping manufacturing environments in order to understand manufacturing in three-dimensional form. In addition, the described systems and methods perform three-dimensional mapping diagram generation, three-dimensional object detection, three-dimensional field element detection, and three-dimensional feature detection based on three-dimensional sensor data to evaluate the quality of assembly processes and assemblies being built. In some illustrative examples, the use of three-dimensional data in the manner described above allows certain features that are undetectable or unconfirmably detectable using only two-dimensional imaging data to be easily and conclusively detected. Therefore, in some cases, the use of three-dimensional data can improve the overall accuracy of quality assessments of assembly processes and assemblies.

[0174] As used herein, when used in conjunction with multiple items, the phrase "at least one" means that different combinations of one or more listed items can be used and only one of the listed items is required. The item can be a specific object, article, step, operation, process, or category. In other words, "at least one" refers to any combination of the listed items or the number of listed items, but not all listed items may be required. For example, without limitation, "at least one of item A, item B, or item C" or "at least one of item A, item B, and item C" can mean item A; item A and item B; item B; item A, item B, and item C; item B and item C; item A and item C. For example, in some cases, "at least one of item A, item B, or item C" or "at least one of item A, item B, and item C" can mean, but not limited to, 2 items A, 1 item B, and 10 items C, 4 items B and 7 items C, or some other suitable combination.

[0175] Additionally, the present disclosure includes embodiments according to the following clauses:

[0176] Clause 1. A method for performing automated supervision and inspection of an assembly process, the method being implemented using a computer system and comprising:

[0177] generating sensor data at the assembly site using a sensor system positioned relative to the assembly site;

[0178] generating, by the computer system, a three-dimensional global map of the assembly site and assemblies being constructed at the assembly site using the sensor data;

[0179] identifying, by the computer system, a current stage of an assembly process for building the assembly based on the three-dimensional global map;

[0180] identifying, by the computer system, a context for the current stage; and

[0181] A quality report for the assembly is generated by the computer system using the three-dimensional global map and the context for the current stage.

[0182] Clause 2. The method according to clause 1, further comprising:

[0183] The quality report is transmitted by the computer system via at least one communication link to at least one remote device for viewing by at least one human operator.

[0184] Clause 3. A method according to any preceding clause, wherein generating the sensor data comprises:

[0185] Three-dimensional data for the assembly site and the assembly is generated using at least one of a light detection and ranging sensor, a radar sensor, a sonar sensor, or a stereo camera positioned relative to the assembly site and the assembly at the assembly site.

[0186] Clause 4. The method of clause 3, wherein generating the three-dimensional global map by the computer system comprises:

[0187] The three-dimensional data is used by the computer system to generate a three-dimensional point cloud of the assembly site and the assembly being constructed at the assembly site.

[0188] Clause 5. The method of clause 4, wherein generating the three-dimensional global map further comprises:

[0189] Points in the three-dimensional point cloud are clustered, by the computer system, to form a set of surface patches.

[0190] Clause 6. The method of clause 5, wherein generating the three-dimensional global map further comprises:

[0191] The set of surface patches is fitted, by the computer system, to surfaces in a three-dimensional model of the assembly site and the assembly being constructed at the assembly site.

[0192] Clause 7. The method of clause 6, wherein fitting the set of surface patches to the surface in the three-dimensional model comprises:

[0193] The set of surface patches is fit to the surface in the three-dimensional model by the computer system using at least one of radial basis functions, spline approximation, or mesh interpolation.

[0194] Clause 8. The method of clause 6, wherein generating the three-dimensional global map by the computer system further comprises:

[0195] The set of surface patches of the surface that have been fitted to the three-dimensional model are organized into the three-dimensional global map by the computer system.

[0196] Clause 9. The method of any preceding clause, wherein identifying, by the computer system, the current stage of the assembly process comprises:

[0197] detecting, by the computer system, a plurality of objects using the three-dimensional global map; and

[0198] At least a portion of the plurality of objects is matched, by the computer system, to a part for the assembly using at least one of an image of the part or a model of the part, wherein the portion of the plurality of objects matched to the part is a plurality of matched objects.

[0199] Clause 10. The method of clause 9, wherein identifying, by the computer system, the current stage of the assembly process based on the plurality of matching objects comprises:

[0200] Detecting, by the computer system, a plurality of venue elements within the three-dimensional global map; and

[0201] The current stage of the assembly process is identified, by the computer system, based on the plurality of matching objects and the plurality of site elements.

[0202] Clause 11. The method of clause 9, wherein identifying, by the computer system, the current stage of the assembly process further comprises:

[0203] determining, by the computer system, at least one of a plurality of positions or a plurality of orientations of the plurality of matched objects relative to a reference coordinate system for the assembly site; and

[0204] The current stage of the assembly process is determined by the computer system based on the identification of the plurality of matching objects and the at least one of the plurality of positions or the plurality of orientations of the plurality of matching objects.

[0205] Clause 12. The method of clause 11, wherein identifying the context by the computer system comprises:

[0206] The context for the current stage of the assembly process is determined by the computer system using context information stored in a data repository, wherein the context includes at least one of a nominal condition for the assembly at the current stage, a nominal condition for each part expected to be present at the current stage, tolerances selected for the nominal condition for the assembly at the current stage, identification of steps completed before the current stage, identification of steps to be completed during the current stage, or identification of steps to be completed after the current stage.

[0207] Clause 13. The method of clause 11, wherein generating the quality report by the computer system comprises:

[0208] A match assessment is generated by the computer system, the match assessment indicating whether one or more of the plurality of objects detected cannot be matched to a corresponding part and whether one or more of the plurality of matched objects do not match a part expected to be present at the current stage of the assembly process.

[0209] Clause 14. The method of clause 11, wherein generating the quality report by the computer system comprises:

[0210] A position assessment is generated, by the computer system, based on the context for the current stage of the assembly process, the position assessment indicating whether each of the plurality of matched objects has been assembled in a correct or incorrect position.

[0211] Clause 15. The method of clause 11, wherein generating the quality report by the computer system comprises:

[0212] An inconsistency assessment is generated by the computer system indicating whether any features have been identified and whether any identified features are outside of selected tolerances, wherein the features are selected from one of cracks, dents, distortions, or surface level features.

[0213] Clause 16. The method according to clause 15, further comprising:

[0214] One or more features in the assembly are detected by the computer system using at least one three-dimensional comparison algorithm.

[0215] Clause 17. The method according to any preceding clause, further comprising:

[0216] Guidance information for guiding a human operator during the assembly process is generated by the computer system based on the context for the current stage of the assembly process.

[0217] Clause 18. The method of clause 17, wherein generating the guidance information comprises:

[0218] A current phase guide is generated by the computer system for use by the human operator to perform one or more tasks during the current phase of the assembly process.

[0219] Clause 19. The method of clause 17, wherein generating the guidance information comprises:

[0220] A next phase guide is generated by the computer system for use by the human operator to perform one or more tasks during a next phase of the assembly process.

[0221] Clause 20. The method according to clause 17, further comprising:

[0222] The guidance information is displayed by the computer system on a display system positioned relative to the assembly site.

[0223] Clause 21. The method according to any preceding clause, further comprising:

[0224] The quality report is stored by the computer system in a data repository as part of a record corresponding to the current stage of the assembly process to thereby build a recorded history for the assembly process.

[0225] Clause 22. The method according to clause 21, further comprising:

[0226] determining, by the computer system, whether the quality report for the current stage of the assembly process indicates that a problem of interest exists; and

[0227] The history recorded is rolled back via the computer system to identify a root cause of the problem of interest.

[0228] Clause 23. A method for performing automated supervision and inspection of an assembly process, the method being implemented using a computer system and comprising:

[0229] generating sensor data at the assembly site using a sensor system positioned relative to the assembly site;

[0230] generating, by the computer system, a three-dimensional global map of the assembly site and assemblies being constructed at the assembly site using the sensor data;

[0231] identifying, by the computer system, a current stage of an assembly process for building an assembly at the assembly site using the three-dimensional global map and assembly information stored in a data repository;

[0232] identifying, by the computer system, a context for the current stage;

[0233] generating, by the computer system, a quality report corresponding to the current stage of the assembly process based on the three-dimensional global map, the assembly information, and the context for the current stage; and

[0234] A record corresponding to the current stage of the assembly process is stored in the data repository by the computer system to thereby build a history of records for the assembly process, wherein the record includes at least one of the three-dimensional global map, at least a portion of the sensor data, or the quality report.

[0235] Clause 24. The method according to clause 23, further comprising:

[0236] A determination is made by the computer system whether the quality report for the current stage of the assembly process indicates that at least one issue of interest exists, wherein the at least one issue of interest is selected from one of an improper part, an improper part position, an improper orientation, a crack, a dent, a torque, and a surface inconsistency.

[0237] Clause 25. The method according to clause 24, further comprising:

[0238] The history of records is rolled back via the computer system to identify a root cause of the at least one problem of interest.

[0239] Clause 26. The method according to any one of clauses 23-25, further comprising:

[0240] At least one of a current stage guide or a next stage guide is generated by the computer system for use by a human operator during the assembly process.

[0241] Clause 27. The method according to clause 26, further comprising:

[0242] At least one of the current stage boot or the next stage boot is displayed on a display system through the computer system.

[0243] Clause 28. The method according to any one of clauses 23-27, further comprising:

[0244] A human operator is notified of the quality report by the computer system presenting at least a portion of the quality report on a display system.

[0245] Clause 29. A system for automatic supervision and inspection of an assembly process, the system comprising:

[0246] a sensor system positioned relative to an assembly site and an assembly being constructed at the assembly site; and

[0247] A computer system that uses sensor data generated by the sensor system to generate a three-dimensional global map of the assembly site and assemblies being built at the assembly site; identifies a current stage of the assembly process for building the assemblies at the assembly site based on the three-dimensional global map; identifies a context for the current stage; and generates a quality report for the assemblies using the three-dimensional global map and the context for the current stage.

[0248] Clause 30. The system of clause 29, wherein the sensor system comprises at least one of a light detection and ranging sensor, a radar sensor, a sonar sensor, or a stereo camera, and wherein the sensor system generates three-dimensional data.

[0249] Clause 31. The system of any of Clauses 29-30, wherein the computer system uses the three-dimensional global map and assembly information stored in a data repository to identify the current stage of the assembly process.

[0250] Clause 32. The system of clause 31, wherein the assembly information comprises at least one of a part image library, a part model library, assembly process flow data, assembly site model data, assembly model data, assembly video data, or feature data.

[0251] Clause 33. A system according to any of clauses 29-32, wherein the context includes at least one of a nominal condition for the assembly at the current stage, a nominal condition for each part expected to exist at the current stage, tolerances selected for the nominal condition of the assembly at the current stage, identification of steps completed before the current stage, identification of steps to be completed during the current stage, or identification of steps to be completed after the current stage.

[0252] For the purpose of illustration and description, the description of different illustrative embodiments has been shown, which is not intended to be exhaustive or limited to the embodiments of the disclosed form. Many modifications and variations will be apparent to those skilled in the art. In addition, different illustrative embodiments may provide different features compared to other desired embodiments. The selected embodiment or embodiments are selected and described to better explain the principles of the embodiments, practical applications, and to enable other persons skilled in the art to understand the disclosure of the various embodiments, wherein various modifications suitable for specific purposes are contemplated.

Claims

1. A method for performing automatic supervision and inspection of an assembly process, the method being implemented using a computer system and comprising: generating sensor data at the assembly site using a sensor system positioned relative to the assembly site; Generating, by the computer system, a three-dimensional global map of the assembly site and assemblies being constructed at the assembly site using the sensor data, wherein generating the three-dimensional global map comprises: generating, by the computer system, a set of surface patches using the sensor data, each surface patch in the set of surface patches comprising one or more three-dimensional surfaces; fitting, by the computer system, the set of surface patches to surfaces in a three-dimensional model of the assembly site and the assembly being constructed at the assembly site; identifying, by the computer system, a current stage of an assembly process for building the assembly based on the three-dimensional global map; identifying, by the computer system, a context for the current stage, The context includes a first identification of steps completed before the current phase, a second identification of steps to be completed during the current phase, and a third identification of subsequent steps to be completed after the current phase; and A quality report for the assembly is generated by the computer system using the three-dimensional global map and the context for the current stage, the quality report including an overall stage assessment indicating the quality of the current stage of the assembly process.

2. The method according to claim 1, wherein: The sensor data includes two-dimensional imaging data and three-dimensional data, and the method further includes: generating, by the computer system, a quality report using the two-dimensional imaging data and the background of the current stage; The quality report generated using the three-dimensional global map and the quality report generated using the two-dimensional imaging data are integrated into a master quality report by the computer system, wherein the integration includes merging redundant detections of undesirable features into a single detection.

3. The method according to claim 1, further comprising: The quality report is transmitted by the computer system via at least one communication link to at least one remote device for viewing by at least one human operator.

4. The method of claim 1 , wherein generating the sensor data comprises: Three-dimensional data for the assembly site and the assembly is generated using at least one of a light detection and ranging sensor, a radar sensor, a sonar sensor, or a stereo camera positioned relative to the assembly site and the assembly at the assembly site.

5. The method of claim 4, wherein generating the three-dimensional global map by the computer system comprises: The three-dimensional data is used by the computer system to generate a three-dimensional point cloud of the assembly site and the assembly being constructed at the assembly site.

6. The method of claim 5, wherein generating the three-dimensional global map further comprises: Points in the three-dimensional point cloud are clustered, by the computer system, to form the set of surface patches.

7. The method of claim 4, wherein fitting the set of surface patches to the surface in the three-dimensional model comprises: Fitting, by the computer system, the set of surface patches to the surface in the three-dimensional model using at least one of radial basis functions, spline approximation, or mesh interpolation, and / or wherein generating, by the computer system, the three-dimensional global map further comprises: The set of surface patches of the surface that have been fitted to the three-dimensional model are organized by the computer system into the three-dimensional global map.

8. The method of claim 1 , wherein identifying, by the computer system, the current stage of the assembly process comprises: detecting, by the computer system, a plurality of objects using the three-dimensional global map; and At least a portion of the plurality of objects are matched to a part of the assembly using at least one of an image of the part or a model of the part by the computer system, wherein the at least a portion of the plurality of objects matched to the part is a plurality of matched objects.

9. The method of claim 8, wherein identifying, by the computer system, the current stage of the assembly process based on the plurality of matched objects comprises: detecting, by the computer system, a plurality of venue elements within the three-dimensional global map; as well as identifying, by the computer system, the current stage of the assembly process based on the plurality of matched objects and the plurality of location elements, and / or wherein identifying, by the computer system, the current stage of the assembly process further comprises: determining, by the computer system, at least one of a plurality of positions or a plurality of orientations of the plurality of matched objects relative to a reference coordinate system for the assembly site; as well as The current stage of the assembly process is determined by the computer system based on the identification of the plurality of matching objects and the at least one of the plurality of positions or the plurality of orientations of the plurality of matching objects.

10. The method of claim 8, wherein identifying, by the computer system, the context comprises: determining, by the computer system, the context for the current stage of the assembly process using context information stored in a data repository, wherein the context also includes at least one of a nominal condition for the assembly at the current stage, or a nominal condition for each part expected to be present at the current stage, a tolerance selected for the nominal condition for the assembly at the current stage, an identification of steps completed before the current stage, an identification of steps to be completed during the current stage, or an identification of steps to be completed after the current stage, and / or wherein generating, by the computer system, the quality report comprises: generating, by the computer system, a match assessment indicating whether one or more of the plurality of objects detected cannot be matched to a corresponding part and whether one or more of the plurality of matched objects do not match a part expected to be present at the current stage of the assembly process, and / or wherein generating, by the computer system, the quality report comprises: generating, by the computer system, a position assessment based on the context for the current stage of the assembly process, the position assessment indicating whether each of the plurality of matched objects has been assembled in a correct or incorrect position, and / or wherein generating, by the computer system, the quality report comprises: An inconsistency assessment is generated by the computer system indicating whether any features have been identified and whether any identified features are outside of a selected tolerance, wherein the features Choose from cracks, dents, distortions, or one of the surface level features.

11. The method according to claim 10, further comprising: One or more features in the assembly are detected by the computer system using at least one three-dimensional comparison algorithm.

12. The method according to claim 1, further comprising: Guidance information for guiding a human operator during the assembly process is generated by the computer system based on the context for the current stage of the assembly process.

13. The method of claim 12, wherein generating the guidance information comprises: generating, by the computer system, current stage guidance for use by the human operator to perform one or more tasks during the current stage of the assembly process, and / or wherein generating the guidance information comprises: generating, by the computer system, a next stage guide for use by the human operator to perform one or more tasks during a next stage of the assembly process, and / or the method further comprises: The guidance information is displayed by the computer system on a display system positioned relative to the assembly site.

14. The method according to claim 1, further comprising: The quality report is stored by the computer system in a data repository as part of a record corresponding to the current stage of the assembly process to thereby build a recorded history for the assembly process.

15. The method according to claim 14, further comprising: determining, by the computer system, whether the quality report for the current stage of the assembly process indicates that a problem of interest exists; and The history recorded is rolled back via the computer system to identify a root cause of the problem of interest.

16. The method according to claim 1: wherein identifying, by the computer system, a current stage of an assembly process for building an assembly at the assembly site is performed using the three-dimensional global map and assembly information stored in a data repository; the method further comprising: generating, by the computer system, a quality report corresponding to the current stage of the assembly process based on the three-dimensional global map, the assembly information, and the context for the current stage; and A record corresponding to the current stage of the assembly process is stored in the data repository by the computer system to thereby build a history of records for the assembly process, wherein the record includes at least one of the three-dimensional global map, at least a portion of the sensor data, or the quality report.

17. The method according to claim 16, further comprising: A determination is made by the computer system whether the quality report for the current stage of the assembly process indicates that at least one issue of interest exists, wherein the at least one issue of interest is selected from one of an improper part, an improper part position, an improper orientation, a crack, a dent, a torque, and a surface inconsistency.

18. The method according to claim 17, further comprising: rolling back, via the computer system, the history recorded to identify a root cause of the at least one issue of interest, and / or the method further comprising: At least one of a current stage guide or a next stage guide is generated by the computer system for use by a human operator during the assembly process.

19. The method according to claim 18, further comprising: A human operator is informed of the quality report by the computer system displaying at least one of the current stage guide or the next stage guide on a display system and / or by the computer system presenting at least a portion of the quality report on a display system.

20. A system for automatic supervision and inspection of an assembly process, the system comprising: a sensor system positioned relative to an assembly site and an assembly being constructed at the assembly site; as well as a computer system that uses sensor data generated by the sensor system to generate a three-dimensional global map of the assembly site and assemblies being built at the assembly site; identifying a current stage of the assembly process for building the assembly at the assembly site based on the three-dimensional global map; identifying a context for the current stage, the context comprising a first identification of steps completed before the current stage, a second identification of steps to be completed during the current stage, and a third identification of subsequent steps to be completed after the current stage; and generating a quality report for the assembly using the three-dimensional global map and the context for the current stage, the quality report comprising an overall stage assessment indicating the quality of the current stage of the assembly process; The generating of the three-dimensional global map comprises: generating, by the computer system, a set of surface patches using the sensor data, each surface patch in the set of surface patches comprising one or more three-dimensional surfaces; The set of surface patches are fitted, by the computer system, to surfaces in a three-dimensional model of the assembly site and the assembly being constructed at the assembly site.

21. The system of claim 20, wherein: The sensor system comprises at least one of a light detection and ranging sensor, a radar sensor, a sonar sensor, or a stereo camera, and wherein the sensor system generates three-dimensional data; Wherein generating the set of surface patches comprises: generating, by the computer system, a three-dimensional point cloud of the assembly site and the assembly being constructed at the assembly site using the three-dimensional data; and Points in the three-dimensional point cloud are clustered, by the computer system, to form the set of surface patches.

22. The system of claim 20, wherein the computer system uses the three-dimensional global map and assembly information stored in a data repository to identify the current stage of the assembly process.

23. The system of claim 22, wherein the assembly information comprises at least one of a part image library, a part model library, assembly process flow data, assembly site model data, assembly model data, assembly video data, or feature data.

24. The system of claim 20, wherein the context comprises a nominal condition for the assembly at the current stage, a nominal condition for each part expected to be present at the current stage, or a tolerance selected for the nominal condition for the assembly at the current stage.

Images