Computerized image-dependent inspection of aircraft structures

Abstract

This invention provides a computer-automated, visual image-based method for inspecting aircraft structures to track their assembly status. [Solution] The build plan 110 for the aircraft structure 104 is received. An image 116 of the aircraft structure 104 is received. Based on the image 116 and the build plan 110, a composite image 122 of the aircraft structure 104 is generated. A region of interest 136 of the composite image 122 is determined. By inputting the region of interest 136 into an object detection machine learning model 138, the actual locations of holes, tacks, and / or fasteners on the aircraft structure 104 within the region of interest 136 are identified. An assembly status report 140 is generated, including the discrepancy between the actual locations and the planned locations 112 specified in the build plan 110. Based on the assembly status report 140, an event notification 168 is generated indicating corrective actions 170 to be taken based on the discrepancy. The event notification 168 is output.

Description

Technical Field

[0001] The present disclosure relates generally to inspecting aircraft structures, and in particular, to inspecting the assembled state of aircraft structures in a computer-automated manner. Background Art

[0002] During aircraft assembly, aircraft structures such as wings and fuselages are inspected to verify that all components and systems meet predetermined criteria before being incorporated into larger structures. The results of such inspections are recorded in a Condition of Assembly (CoA) report that confirms the aircraft complies with regulatory standards, design specifications, and safety requirements. More specifically, the CoA report is used to identify inconsistencies between the actual aircraft structure and the build plan for said aircraft structure.

[0003] Traditionally, aircraft structures during assembly are inspected manually by inspectors comparing them to a hard copy of the build plan for that structure. More specifically, for example, inspectors can verify that the aircraft structure has the number, type, and location of holes, tacks, and / or fasteners specified in the build plan. By inspecting the aircraft structure and confirming that the holes, tacks, and / or fasteners are provided as specified in the build plan, the aircraft structure can be kept within the specifications of the build plan throughout the entire assembly process. Inspectors manually record the inspection results on a hard copy of the CoA report for that aircraft structure. This traditional method of manually inspecting aircraft structures and manually creating CoA reports with recorded inspection results is time-consuming, leading to prolonged downtime between stages in the aircraft assembly process and increasing the overall cost of the aircraft. Furthermore, this traditional manual inspection of aircraft structures, carried out by inspectors, is prone to errors, which could lead to further delays in aircraft assembly and potentially cause problems with the manufacturing quality of the aircraft. [Overview of the project]

[0004] Disclosed is an example of a computer-automated, image-based method for inspecting an aircraft structure to track its assembly status. In one example, a build plan for the aircraft structure is received, specifying multiple planned locations for multiple holes, tacks, and / or fasteners on the aircraft structure. Multiple digital images of the aircraft structure are received from one or more cameras. Based on at least the multiple digital images and the build plan, a composite digital image of the aircraft structure is generated by a composite image generation module. Multiple regions of interest (ROIs) are determined in the composite digital image. For each of the multiple ROIs, the actual locations of multiple actual holes, tacks, and / or fasteners on the aircraft structure within the ROI are identified by inputting the ROI into an object detection machine learning (ML) model. An assembly status report of the aircraft structure is generated, including discrepancies between the multiple actual locations and the multiple planned locations. Based on at least the assembly status report, one or more event notifications regarding the aircraft structure are generated. The one or more event notifications indicate one or more corrective actions to be taken based on at least the discrepancies. The one or more event notifications are output.

[0005] This summary is provided to introduce some of the concepts further described in the detailed description below in a simplified form. This summary is not intended to identify any major or essential features of the subject matter of the claims, nor is it intended to be used to limit the scope of the subject matter of the claims. Furthermore, the subject matter of the claims is not limited to embodiments that resolve any or all of the defects pointed out in any part of this disclosure. [Brief explanation of the drawing]

[0006] [Figure 1] This figure schematically illustrates an exemplary computing environment in which a computer-automated, image-based method for inspecting aircraft structures to track their assembly state is implemented, according to one embodiment of the present disclosure. [Figure 2] This figure schematically illustrates an exemplary process for generating a composite digital image of an aircraft structure according to one embodiment of the present disclosure. [Figure 3] This figure schematically illustrates an exemplary process for generating multiple regions of interest (ROIs) in a composite digital image of an aircraft structure shown in Figure 2, according to one embodiment of the present disclosure. [Figure 4] This figure shows an example of a Condition of Assembly (CoA) report generated for an aircraft structure according to one embodiment of the present disclosure. [Figure 5] Figure 2 shows an illustrative marked digital image of an aircraft structure ROI, where mismatches between multiple actual and multiple planned locations of multiple holes, tacks, and / or fasteners on the aircraft structure are highlighted by visual markers. [Figure 6-7] This figure shows an example of a computer-based method for inspecting the assembly state of an aircraft structure, according to one embodiment of the present disclosure. [Figure 8] This figure schematically shows an example of a computing system that represents any computing system in the computing environment shown in Figure 1. [Modes for carrying out the invention]

[0007] Traditional manual inspections of aircraft structures, performed by one or more inspectors, are time-consuming and prone to errors, potentially leading to further delays in aircraft assembly and causing problems with the manufacturing quality of the aircraft. Furthermore, the traditional method of manually creating Condition of Assembly (CoA) reports that record the results of manual inspections is also time-consuming. These traditional manual inspection and documentation methods prolong downtime between stages in the aircraft assembly process and contribute to increased overall aircraft costs.

[0008] Accordingly, in order to address these and other issues discussed herein, embodiments of a computer-automated, image-based method for inspecting aircraft structures and tracking their Condition of Assembly (CoA) are disclosed. In one example, multiple digital images of a large aircraft structure, such as the wings or fuselage of an aircraft, are used to generate a composite digital image of the aircraft structure. Multiple regions of interest (ROIs) are determined in this composite digital image. For each of the ROIs, the actual locations of multiple actual holes, tacks, and / or fasteners on the aircraft structure within the ROI are identified by inputting the ROI into an object detection machine learning (ML) model. A CoA report is generated for the aircraft structure, including discrepancies between the actual locations of multiple actual holes, tacks, and / or fasteners on the aircraft structure and the planned locations of multiple holes, tacks, and / or fasteners on the aircraft structure as specified in the aircraft structure's build plan. Based on at least the CoA report, one or more event notifications regarding the aircraft structure are generated. The one or more event notifications indicate one or more corrective actions to be taken based on at least the above discrepancies. Output one or more event notifications.

[0009] The computer-automated, image-based method for inspecting aircraft structures disclosed herein provides a simple, rapid, reliable, and non-invasive method for automating, detecting, and inspecting drilling and fastening operations performed during the assembly of aircraft structures. Furthermore, the method leverages a specially trained ML model to identify, count, and compare openings, tacks, and / or fasteners on the aircraft structure with the aircraft structure's build plan, automatically generating a CoA report that is faster, more accurate, and more robust than conventional manual inspection and documentation methods.

[0010] Figure 1 schematically shows an example of a computing environment 100 in which a computing system 102 is configured to perform a computer-automated image-based technique to inspect the aircraft structure 104 in order to track the CoA.

[0011] The aircraft structure 104 may include any suitable aircraft components or assemblies. Typically, the aircraft structure 104 is large, and it is not possible to capture the entire structure in a single digital image with sufficient resolution to capture all the features to be tracked in the CoA inspection. In some examples, the aircraft structure 104 includes the fuselage of the aircraft. In other examples, the aircraft structure 104 includes the wings of the aircraft. In yet another example, the aircraft structure 104 includes another large component of the aircraft.

[0012] The computing system 102 comprises one or more processors 106 configured to execute instructions stored in memory 108 to perform computing operations related to a computer-automated image-based method for inspecting the aircraft structure 104 disclosed herein. In one example, one or more processors 106 are configured to execute instructions stored in memory 108 to receive a build plan 110 of the aircraft structure 104, specifying a plurality of planned locations 112 for a plurality of planned holes, tacks and / or fasteners 114 on the aircraft structure 104. In some examples, the build plan 110 includes hole system layout features, which are specific attributes or characteristics related to holes in structural components, such as fastener holes, rivet holes, or bolt holes. These hole system layout features provide a reference for aligning and joining the parts of the aircraft structure 104. The exact number, order, placement, and drilling of holes (and / or tacks / fasteners) in the aircraft structure 104 are important for maintaining tolerances and structural integrity. The size, shape, and spacing of the holes affect the load distribution and the strength of the joints. Such hole layout features enable automated drilling and inspection systems to locate and work on the predetermined locations of multiple holes, tacks, and / or fasteners 114 on the aircraft structure 104, thereby streamlining the assembly process.

[0013] One or more processors 106 are configured to execute instructions stored in memory 108 to receive multiple digital images 116 of the aircraft structure 104 from one or more cameras 118. In some examples, the cameras 118 include multiple cameras positioned at various locations within the facility where the aircraft structure 104 is being assembled. In some examples, at least one of the cameras 118 includes a camera in a fixed position (e.g., X, Y, Z) and / or attitude (e.g., pitch, roll, azimuth). For example, multiple cameras can be positioned at different locations along the assembly line of the aircraft structure 104, and the aircraft structure 104 can move relative to these multiple cameras as the assembly of the aircraft structure 104 progresses.

[0014] In some examples, at least one of the cameras 118 includes a camera that can move to different positions and / or orientations. In some examples, the movable camera is mounted on a robot that moves to different locations within a facility where the aircraft structure 104 is assembled. In other examples, the movable camera is mounted on a robotic arm whose position and / or orientation can be adjusted in order to capture digital images 116 of the aircraft structure 104 from different viewpoints. In yet another example, the movable camera may include a handheld camera held by an operator in order to capture digital images 116 of the aircraft structure 104 from different viewpoints.

[0015] In some examples, camera 118 includes cameras already used for purposes other than CoA inspection, related to the assembly of the aircraft structure 104. For example, camera 118 may already be used to allow an operator to visually inspect the aircraft structure 104 while manually operating a drilling machine, or to monitor a robot performing automated drilling into the aircraft structure 104. By utilizing cameras already used for other purposes related to the assembly of the aircraft structure 104, computer-automated image-based CoA inspection methods can be implemented in a non-invasive manner, or at least with minimal interference with assembly work performed on the aircraft structure 104.

[0016] Camera 118 can take any suitable form. For example, camera 118 may include one or more of the following: a visible light camera, a depth camera, a LiDAR sensor, another suitable type of camera, or a combination thereof.

[0017] In some embodiments, the camera 118 is configured to capture a series of digital images 116 in the form of a video stream, from which multiple digital images 116 can be extracted. In other embodiments, the entire video stream is processed and analyzed at once, which consumes more computing resources compared to processing and analyzing a subset of digital images extracted from the video stream.

[0018] Typically, because the aircraft structure 104 is quite large, it is difficult to capture the entire aircraft structure 104 (or at least the area of ​​the aircraft structure 104 being inspected) in a single digital image with sufficient detail to visualize multiple holes, tacks, and / or fasteners for automated computer inspection to evaluate the CoA of the aircraft structure 104. Therefore, the computing system 102 is configured to generate a composite digital image 122 based on at least multiple digital images 116 of the aircraft structure 104, and this composite digital image shows multiple holes, tacks, and / or fasteners of the aircraft structure 104 with sufficient image resolution for automated computer inspection to evaluate the CoA of the aircraft structure 104.

[0019] In one example, the processor 106 is configured to execute a composite digital image generation module 120, which is configured to generate a composite digital image 122 of the aircraft structure 104 based on at least a plurality of digital images 116 and a build plan 110, by executing instructions stored in memory 108. More specifically, the composite digital image generation module 120 may use the build plan 110 as a criterion for confirming that the entire aircraft structure 104 (or at least the region of the aircraft structure 104 being inspected by the CoA) is captured by the plurality of digital images 116, and that the composite digital image 122 includes the entire aircraft structure 104 (or at least the region of the aircraft structure 104 being inspected by the CoA).

[0020] In some embodiments, the synthetic digital image generation module 120 runs one or more machine learning (ML) models that have been trained to perform various computing operations related to the generation of the synthetic digital image 122. In one example, the synthetic digital image generation module 120 is configured to run a feature extraction ML model 124, a feature matching ML model 126, and an image stitching ML model 128.

[0021] The feature extraction ML model 124 is configured to identify and extract features including aircraft-specific features and more basic features from the plurality of digital images 116, and generate descriptors for the extracted features.

[0022] The feature extraction ML model 124 can be trained using various training methods. In one example, the feature extraction ML model 124 is trained using supervised learning based at least on a training dataset including digital images of a plurality of different aircraft structures, and these digital images are labeled with aircraft-specific features and hole-based layout features that are features common to the aircraft structures to be inspected by CoA. For example, the aircraft-specific features and the hole-based layout features may include wing edges, fuselage curvature, seams between panels, jig features, hole-based layout feature markings, and other aircraft-specific features. By training the feature extraction ML model 124 using labeled training data specific to aircraft structures, the feature extraction ML model 124 can identify features specific to the aircraft structure 104 in the plurality of digital images 116 with higher accuracy. In another example, the feature extraction ML model 124 is trained using unsupervised learning. In this case, the training data includes unlabeled digital images of the aircraft structures, and an unsupervised learning method is applied to refine the latent spatial representation of the training data into a meaningful compressed representation that captures the structure or pattern inherent in the training data. Furthermore, the feature extraction ML model 124 can also be trained using a self-supervised learning method that may involve generation of pseudo-labels or auxiliary objective functions to structure the latent spatial representation of the training data.

[0023] The feature extraction ML model 124 can be implemented using various types of ML models. In one example, the feature extraction ML model 124 is implemented using a convolutional neural network (CNN). In another example, the feature extraction ML model 124 is implemented using a specialized deep learning model such as D2-Net, R2D2, or SuperPoint. In yet another example, the feature extraction ML model 124 is implemented using a hybrid approach that combines deep learning-based techniques and conventional neural network techniques to further improve feature extraction from the plurality of digital images 116. In still another example, the feature extraction ML model 124 is implemented using other types of ML models.

[0024] The feature matching ML model 126 is configured to receive features and corresponding descriptors extracted from the plurality of digital images 116 by the feature extraction ML model 124. The feature matching ML model 126 is configured to match features extracted between different digital images of the plurality of digital images 116 by comparing the descriptors. In order to properly align the plurality of digital images 116 to form the composite digital image 122, the extracted features matched between the digital images are used to identify which digital image corresponds to which part of the aircraft structure.

[0025] The feature matching ML model 126 can be implemented using various types of ML models. In one example, the feature matching ML model 126 is implemented using a deep learning model (e.g., SuperGlue, LoFTR). In another example, the feature matching ML model 126 is implemented using a different type of ML model. In yet another example, the synthetic digital image generation module 120 may perform feature matching using conventional distance-based matching methods such as the Scale-Invariant Feature Transform (SIFT) method and the Speeded-Up Robust Features (SURF) method.

[0026] In some embodiments, the accuracy of feature matching can be improved based at least on the features of the camera 118 used to capture multiple digital images 116. In some embodiments, the camera 118 is configured to project multiple landmark features 130 onto the aircraft structure 104. The multiple landmark features 130 are captured within multiple digital images 116 of the aircraft structure 104. The composite digital image generation module 120 is configured to generate a composite digital image 122 of the aircraft structure 104 based at least on aligning the landmark features 130 between adjacent digital images among the multiple digital images 116. In embodiments using a feature matching ML model 126 for feature matching, the feature matching ML model 126 is configured to match the landmark features 130 between adjacent digital images among the multiple digital images 116.

[0027] The image stitching ML model 128 is configured to generate a composite digital image 122 based on aligning at least one digital image 116 that has a matched feature. In one example, the image stitching ML model 128 is configured to generate a composite digital image 122 by calculating a transformation matrix (e.g., homography) between the digital images 116 using the matching features output by the feature matching ML model 126. Furthermore, the image stitching ML model 128 is configured to warp the digital images as needed to align them with each other. In some embodiments, the image stitching ML model 128 is configured to use a blending technique to smooth the transitions between digital images. In some embodiments, the image stitching ML model 128 is configured to match the texture and color between the stitched images. In some embodiments, the image stitching ML model 128 is configured to correct parallax problems that may occur when objects at different depths appear misaligned in overlapping digital images.

[0028] In some embodiments, the composite digital image generation module 120 can improve the accuracy of the composite digital image 122 by using metadata 132 associated with at least a plurality of digital images 116. In some examples, the metadata 132 may include camera position and / or orientation data indicating the position and / or orientation of the camera when the digital images were captured by the camera. By using the camera position and / or orientation data with the feature matching ML model 126, feature matching between digital images among the plurality of digital images 116 can be supported. Furthermore, by using the camera position and / or orientation data with the image stitching ML model 128, different digital images among the plurality of digital images 116 can be aligned to generate the composite digital image 122. In some embodiments, the metadata 132 may include depth data provided by a depth camera or LiDAR sensor. By using the depth data with the feature extraction ML model 124, the location of features in the plurality of digital images 116 can be determined based at least on the depth of features in the digital images.

[0029] Figure 2 schematically illustrates an exemplary process 200 for generating a composite digital image of an aircraft structure according to one embodiment of the present disclosure. In the illustrated process 200, the aircraft structure to be inspected by the CoA is the aircraft wing 202. Due to the size of the aircraft wing 202, a single camera cannot easily capture images with sufficient resolution to capture all holes, tacks and / or fasteners on the aircraft wing 202. Therefore, instead, multiple cameras 118 (e.g., 118.1, 118.2, 118.3, 118.4, 118.5) capture multiple digital images 116 (e.g., 116.1, 116.2, 116.3, 116.4, 116.5) of different areas of the aircraft wing 202. For example, the multiple cameras 118 can be placed at various locations within the facility where the aircraft wing 202 is assembled.

[0030] In the illustrated embodiment, multiple cameras 118 are configured to project multiple landmark features 130 onto the aircraft wing 202. The multiple landmark features 130 are captured in multiple digital images 116 of the aircraft wing 202. For example, the first landmark feature set 130.1 is captured in the first digital image 116.1, the second landmark feature set 130.2 is captured in the second digital image 116.2, the third landmark feature set 130.3 is captured in the third digital image 116.3, the fourth landmark feature set 130.4 is captured in the fourth digital image 116.4, the fifth landmark feature set 130.5 is captured in the fifth digital image 116.5, and the sixth landmark feature set 130.6 is captured in the sixth digital image 116.6.

[0031] The composite digital image generation module 120 shown in Figure 1 is configured to generate a composite digital image 122 of an aircraft wing 202, based at least on aligning landmark features 130 between adjacent digital images among a plurality of digital images 116. More specifically, the landmark feature 130 provides a feature that can be used to align adjacent digital images during the digital image stitching process.

[0032] The composite digital image 122 includes the entire aircraft wing 202 (or at least the area of ​​the aircraft wing to be inspected for CoA), including all holes, tacks, and / or fasteners to be inspected for CoA. The composite digital image 122 is then used in a computer-automated inspection to determine the CoA of the aircraft wing 202.

[0033] Returning to Figure 1, in some embodiments, the processor 106 is configured to execute instructions stored in memory 108 to run the Region of Interest (ROI) determination module 134. The ROI determination module 134 receives the composite digital image 122 output from the composite digital image generation module 120 and is configured to determine a plurality of ROIs 136 that are subject to visual inspection of the CoA in the composite digital image 122. The composite digital image 122 can be sliced ​​or divided into a plurality of ROIs 136 that can be individually visually inspected for the CoA, which is more efficient in terms of computing resources overall compared to visually inspecting the entire composite digital image 122 at once. Furthermore, the accuracy of identifying the location of holes, tacks and / or fasteners on the aircraft structure 104 can be improved by limiting the visual inspection area to ROIs compared to visually inspecting the entire composite digital image 122. However, in some embodiments, the visual inspection of the CoA can also be performed on the entire composite digital image 122 without slicing or dividing it into a plurality of ROIs.

[0034] Multiple ROIs 136 can be set to any suitable size smaller than the composite digital image 122. In some examples, each ROI 136 is set to the same predetermined size / dimensions. In one example, each ROI 136 is a 400 x 400 pixel square. In other examples, the ROIs 136 have different predetermined sizes / dimensions. In yet another example, the ROIs 136 can have different sizes / dimensions from each other. In some of these examples, the ROIs 136 can be dynamically selected by the ROI determination module 134 based on the planned location of planned features on the aircraft structure 104 as specified in the build plan 110. In some examples, each of the multiple ROIs 136 does not overlap with one another. In other examples, at least one of the ROIs 136 overlaps at least partially with at least one other ROI 136.

[0035] Figure 3 schematically illustrates an exemplary process 300 for generating multiple ROIs in a composite digital image 122 of the aircraft wing 202 shown in Figure 2, according to one embodiment of the present disclosure. In the illustrated example, the multiple ROIs 136 are all squares of the same size and do not overlap each other. The ROI determination module 134 aligns the first ROI 136.1 to the upper left corner of the composite digital image 122, and then slices / divides the image into multiple ROIs 136 by moving from left to right across the composite digital image 122. The ROI determination module 134 continues slicing / dividing the composite digital image 122 until it reaches the lower right corner and the entire composite digital image 122 is sliced / divided into multiple ROIs 136. In some examples, each of the multiple ROIs 136 is assigned an index to identify the ROI's position within the composite digital image 122 and / or its relative position to other ROIs within the composite digital image 122. The process 300 for generating multiple ROIs of the composite digital image 122 is shown as a non-limiting example. In other examples, the composite digital image 122 can be sliced / divided in a different way.

[0036] In some embodiments, the ROI determination module 134 is configured to identify regions of the composite digital image 122 that are not of interest and can be excluded from the CoA inspection. For example, referring to the composite digital image 122 of an aircraft wing 202, regions of the composite digital image 122 that do not include the aircraft wing 202, such as regions that include only the background and do not include the aircraft wing 202, can be identified as regions of no interest and can be excluded from the CoA inspection.

[0037] Returning to Figure 1, the processor 106 is configured to execute an object detection ML model 138 by executing instructions stored in memory 108. The object detection ML model is configured to receive a build plan 110 of the aircraft structure 104 and a plurality of ROIs 136 as input and to generate a CoA report 140 based on performing a visual inspection of at least the plurality of ROIs 136. The CoA report 140 includes object tracking data 142 about the objects identified on the aircraft structure 104 by the object detection ML model 138.

[0038] Figure 4 schematically shows a CoA report 140 of an aircraft structure 104 generated by an object detection ML model 138 according to one embodiment of the present disclosure. The object detection ML model 138 is configured to identify the actual locations 146 of multiple actual holes, tucks and / or fasteners 148 on the aircraft structure within each of a plurality of ROIs 136. In some examples, the actual locations 146 of multiple actual holes, tucks and / or fasteners 148 are represented as a group of pixels. In other examples, the actual locations 146 of multiple actual holes, tucks and / or fasteners 148 are represented as point cloud locations. In some embodiments, a plurality of planned holes, tucks and / or fasteners 114, their corresponding planned locations 112, a plurality of actual holes, tucks and / or fasteners 148, and their corresponding actual locations 146 are listed in the CoA report.

[0039] The object detection ML model 138 is configured to identify inconsistencies 150 by comparing the multiple actual locations 146 of multiple actual holes, tucks and / or fasteners 148 with the planned locations 112 of the planned holes, tucks and / or fasteners 114 specified in the build plan. For example, inconsistencies 150 may include actual holes, tucks and / or fasteners placed in the wrong location, missing actual holes, tucks and / or fasteners, extra actual holes, tucks and / or fasteners, and actual holes, tucks and / or fasteners of a different type than planned (e.g., holes instead of planned fasteners). In some embodiments, the inconsistencies 150 are listed in the CoA report 140.

[0040] In some embodiments, the object detection ML model 138 is configured to identify specific types 152 (e.g., rivets, screws, bolts) of actual holes, tacks, and / or fasteners 148. In some embodiments, each type 152 of actual holes, tacks, and / or fasteners 148 is listed in the CoA report 140.

[0041] In some embodiments, the object detection ML model 138 is configured to track the total number 154 actual holes, tacks, and / or fasteners 148 identified on the aircraft structure 104. The total number 154 can be used as summary information for quickly evaluating the global CoA of the aircraft structure 104. In some embodiments, the total number 154 actual holes, tacks, and / or fasteners 148 are listed in the CoA report 140.

[0042] In some embodiments, the object detection ML model 138 is further configured to identify the location 158 of foreign debris 160 on the aircraft structure 104 within each of a plurality of ROIs 136. The foreign debris 160 includes objects that are not intended to be present on the aircraft structure 104. For example, the foreign debris 160 may include material fragments (e.g., metal shavings in holes), tools, and other foreign matter (e.g., sealant). In some embodiments, the foreign debris 160 may also be included in the mismatch 150. In some embodiments, the identified locations 158 of the foreign debris 160 on the aircraft structure 104 are listed in the CoA report 140.

[0043] In some embodiments, the object detection ML model 138 is further configured to identify the locations of defects 156 in multiple actual holes, tucks, and / or fasteners 148 within each of a plurality of ROIs 136. Defects may include holes, tucks, and / or fasteners that are incorrectly positioned, holes with inappropriate shape and / or depth, and other defects. In some examples, the object detection ML model 138 is configured to identify defects 156, including the installation of the wrong type or part number of fasteners on the aircraft structure 104. In some embodiments, the identified locations 156 of defects in actual holes, tucks, and / or fasteners on the aircraft structure 104 are listed in the CoA report 140.

[0044] In some embodiments, the CoA report 140 generated by the object detection ML model 138 includes a timestamp 162 indicating the time the CoA report was generated. The object detection ML model 138 can be configured to generate CoA reports based on different composite digital images of the aircraft structure 104 at different points in time, indicated by different timestamps, throughout the entire assembly process, in order to track the CoA of the aircraft structure 104.

[0045] In some embodiments, the object detection ML model 138 is further configured to generate one or more marked digital images 164 of the aircraft structure 104 in which at least one inconsistency 150 is highlighted by a visual marker. The marked digital images 164 can be displayed to technicians performing assembly work on the aircraft structure 104 so that they can quickly identify the inconsistency 150 and take corrective action to correct the inconsistency 150.

[0046] Figure 5 shows an exemplary marked digital image 164 of the ROI of a composite digital image 122 of the aircraft wing 202 shown in Figure 2, in which inconsistencies 150 between multiple actual locations 146 and multiple planned locations 112 of multiple holes, tacks and / or fasteners on the aircraft wing 202 are highlighted by multiple visual markers 300 (e.g., 300.1, 300.2, 300.3, 300.4). In the illustrated example, the marked digital image 164 includes actual holes, tacks and / or fasteners identified by the object detection ML model 138. In some examples, the actual holes, tacks and / or fasteners can be color-coded by type within the marked digital image 164 (e.g., holes are shown in red, tacks in orange, and fasteners in green). The marked digital image 164 contains several inconsistencies 150 (e.g., 150.1, 150.2, 150.3, 150.4) identified by the object detection ML model 138. As an example, the first inconsistency 150.1 is highlighted by the first visual marker 300.1. The first inconsistency 150.1 is identified as a fastener that is incorrectly attached where a tack should be. As another example, the second inconsistency 150.2 is highlighted by the second visual marker 300.2. The second inconsistency 150.2 is identified as a location where a hole should be drilled according to the build plan 110, but where no hole currently exists. As yet another example, the third inconsistency 150.3 is highlighted by the third visual marker 300.3. The third inconsistency 150.3 is identified as a hole that is in a different location than indicated in the build plan 110, or that is formed out of order as indicated in the build plan. As yet another example, the fourth inconsistency 150.4 is highlighted by the fourth visual marker 300.4. The fourth inconsistency 150.4 is identified as a defective hole because it is elliptical rather than circular.

[0047] The marked digital image 164 may include any of the object tracking data 142 identified by the object detection ML model 138 based on processing at least multiple ROIs 136 of the composite digital image 122.

[0048] Returning to Figure 1, the object detection ML model 138 is configured to output the marked digital image 164 to an output device such as the display subsystem 808 (see Figure 8) and / or one or more technician devices 176. This allows technicians involved in the assembly process of the aircraft structure 104 to refer to the marked digital image 164 when addressing the inconsistencies 150 or other issues identified by the object detection ML model 138.

[0049] The object detection ML model 138 can be implemented using various types of ML models. In one example, the object detection ML model 138 is implemented using a CNN. In some examples, the object detection ML model 138 is implemented as a single-stage object detection model (e.g., YOLO, SSD). In other examples, the object detection ML model 138 is implemented as a two-stage object detection model (e.g., RCNN, Faster R-CNN). In yet another example, the object detection ML model 138 is implemented using a semantic segmentation-based model (e.g., DeepLab, U-Net) that classifies image pixels into object categories. In yet another example, the object detection ML model 138 is implemented using an instance segmentation-based model (e.g., Mask R-CNN) that detects objects and defines their pixel-level boundaries. In yet another example, the object detection ML model 138 can be implemented using a different type of machine learning model.

[0050] The processor 106 is configured to execute an event notification module 166, which is configured to execute instructions stored in memory 108 and receive a CoA report 140 generated by the object detection ML model 138. The event notification module 166 is configured to generate one or more event notifications 168 concerning the aircraft structure 104, based on at least the CoA report 140. The event notifications 168 indicate one or more corrective actions 170 to be taken, based on at least the inconsistencies 150 listed in the CoA report 140. In some examples, the corrective action 170 is taken to correct the inconsistency 150. For example, if the inconsistency indicates that a planned hole does not exist at a planned location on the aircraft structure, the corrective action 170 includes sending a command to drill a hole at that planned location on the aircraft structure. In other examples, other corrective actions 170 may be taken to correct other types of inconsistencies identified in the CoA report 140, which may include, for example, sending a command to replace an incorrect tack / fastener with a correct tack / fastener or sending a command to fill a hole that has been incorrectly formed in the aircraft structure 104.

[0051] In embodiments where the object detection ML model 138 is configured to identify defects 156 on the aircraft structure 104, the corrective action 170 includes sending a command to correct the defect 156 of a hole, tack and / or fastener at the identified location on the aircraft structure 104. For example, such a corrective action 170 to correct the defect 156 may include correcting the shape of a hole, filling a misformed hole, or replacing a broken tack / fastener with a properly functioning tack / fastener.

[0052] In an embodiment where the object detection ML model 138 is configured to identify foreign debris 160 on the aircraft structure 104, the corrective action 170 includes sending a command to remove the foreign debris 160 from its identified location 158.

[0053] In some examples, corrective action 170 is taken to prevent further inconsistencies from occurring downstream in the assembly process due to an inconsistency 150 identified in the CoA report 140. In one example, if the CoA report 140 identifies an inconsistency 150 indicating that a planned hole is missing, corrective action 170 includes sending a command to stop the filling operation that would otherwise be performed at the planned location of the hole until the hole can be formed at the planned location. In other examples, other corrective actions may be taken to prevent further inconsistencies from occurring downstream.

[0054] In embodiments where CoA report 140 is generated along with a timestamp 162, the event notification module 166 is configured to compare the CoA report 140 of the aircraft structure 104 with a previous CoA report (not shown) of the aircraft structure 104 having an earlier timestamp to verify whether the previous inconsistencies identified in the previous CoA report have been corrected since the generation of the previous CoA report. Based on the identification that the previous inconsistencies have not been corrected, the event notification module 166 is configured to generate one or more additional event notifications indicating one or more additional corrective actions to correct the previous inconsistencies. The event notification module 166 is also configured to generate an event notification confirming that the previous inconsistencies have been corrected, based on the identification that the previous inconsistencies have been corrected in the current CoA report 140. This functionality provides the ability to track the current "as-built" state of the aircraft structure 104 throughout the assembly process and provides verification feedback, which is retained as data, to mitigate quality deviations caused by inconsistencies that may occur during the assembly process.

[0055] The event notification module 166 is configured to output event notifications 168 to different destinations. In some embodiments, the event notification module 166 is configured to output event notifications 168 to a data warehouse 172 configured to store event notifications 168. In such embodiments, the object detection ML model 138 is also configured to output CoA reports 140 to the data warehouse 172. The data warehouse 172 is configured to store CoA reports 140 (and other CoA reports for the aircraft structure 104 generated throughout the assembly process) and event notifications 168 for the aircraft structure 104 in memory (e.g., a database). The storage of CoA reports and event notifications in the data warehouse 172 allows the information to be referenced as part of a feedback / verification function to confirm that identified inconsistencies and / or defects have been corrected and / or foreign debris has been removed from the aircraft structure. The storage of CoA reports and event notifications in the data warehouse 172 also allows the information to be referenced to audit and improve the assembly process used for future aircraft structure assembly.

[0056] In some embodiments, the event notification module 166 is configured to output an event notification 168 to the assembly robot 174. The assembly robot 174 is configured to perform computer-automated tasks related to the assembly of the aircraft structure 104, such as forming and filling holes, correcting defects, and removing foreign debris. The assembly robot 174 is configured to perform the corrective action 170 specified in the event notification 168. In this way, the inconsistencies 150 identified by the object detection ML model 138 can be automatically corrected by the assembly robot 174 with minimal or no human intervention.

[0057] In some embodiments, the assembly robot 174 is configured to send a confirmation notice to the computing system 102 based on the completion of at least the corrective action 170 specified in the event notification 168. This confirmation notice may be sent to the event notification module 166 as part of a process to track whether the inconsistency has been corrected.

[0058] In some embodiments, the event notification module 166 is configured to output an event notification 168 to one or more technician devices 176 (e.g., portable computing devices, smartphones, tablet computers). The technician device 176 can present the event notification 168, including a corrective action 170, to the relevant technician, so that the technician can perform the corrective action 170 manually if the assembly robot 174 cannot perform the corrective action 170 in a computer-automated manner.

[0059] Figures 6 and 7 show an example of a computer-based method 600 for inspecting the assembly condition of an aircraft structure, according to one embodiment of the present disclosure. For example, method 600 can be performed by the computing system 102 shown in Figure 1 or by another suitable computing system.

[0060] In 602, method 600 includes receiving a build plan for an aircraft structure, specifying a plurality of planned locations for a plurality of holes, tacks and / or fasteners on the aircraft structure. In 604, method 600 includes receiving a plurality of digital images of the aircraft structure from one or more cameras. In some embodiments, in 606, one or more cameras may be configured to project a plurality of landmark features onto the aircraft structure, and the plurality of landmark features may be captured in the plurality of digital images of the aircraft structure. In 608, method 600 includes generating a composite digital image of the aircraft structure using a composite digital image generation module based on at least a plurality of digital images and the build plan. In some embodiments, in 610, the composite digital image may be generated based on at least aligning landmark features between adjacent digital images. In 612, method 600 includes determining a plurality of regions of interest (ROIs) of the composite digital image. In 614, Method 600 includes inputting each of a plurality of ROIs into an object detection machine learning (ML) model to identify multiple actual locations of multiple actual holes, tacks and / or fasteners on the aircraft structure within the ROI. In some embodiments, in 616, Method 600 may include identifying the location of foreign debris on the aircraft structure within each of a plurality of ROIs. In some embodiments, in 618, Method 600 may include identifying the location of defects in multiple holes, tacks and / or fasteners within each of a plurality of ROIs. In some embodiments, in 620, Method 600 may include generating marked digital images of one or more ROIs of the aircraft structure in which inconsistencies are highlighted with visual markers.

[0061] In Figure 7, in some embodiments, method 600 may include outputting one or more marked digital images in 622. In 624, method 600 includes generating an assembly condition (CoA) report of the aircraft structure, which includes discrepancies between a plurality of actual locations and a plurality of planned locations. In some embodiments, in 626, the CoA report further includes identified locations of foreign debris on the aircraft structure. In some embodiments, in 628, the CoA report further includes identified locations of defects in holes, tacks and / or fasteners on the aircraft structure. In 630, method 600 includes generating one or more event notifications relating to the aircraft structure based on at least the assembly condition report, which indicate one or more corrective actions to be taken based on at least the above discrepancies. In some embodiments, in 632, one or more corrective actions include sending one or more commands to remove foreign debris from identified locations on the aircraft structure. In some embodiments, in 634, one or more corrective actions include sending one or more commands to correct defects in holes, tacks and / or fasteners at identified locations on the aircraft structure. In 636, method 600 includes outputting one or more event notifications. In some embodiments, in 638, method 600 may include comparing a CoA report of an aircraft structure with a previous CoA report of the aircraft structure to verify whether previous inconsistencies have been corrected since the generation of the previous CoA report. In some embodiments, in 640, method 600 may, based on the identification that previous inconsistencies have not been corrected, generate one or more additional event notifications indicating one or more additional corrective actions to correct those previous inconsistencies.

[0062] By implementing Method 600, a computer-automated, image-based approach can be realized for inspecting aircraft structures in a simple, rapid, reliable, and non-invasive manner. Furthermore, Method 600 leverages a specially trained ML model to identify, count, and compare openings, tacks, and / or fasteners on the aircraft structure with the aircraft structure's build plan, automatically generating a CoA report that is faster, more accurate, and more robust than conventional manual inspection and documentation methods. In addition, an event notification is output, containing corrective actions that can be taken to correct any inconsistencies identified in the CoA report. Such corrective actions can be performed in an automated manner in some embodiments, minimizing human intervention throughout the entire process.

[0063] In some embodiments, the methods and processes described herein may be associated with a computing system comprising one or more computing devices. In particular, such methods and processes may be implemented as computer application programs or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0064] Figure 8 schematically illustrates a non-limiting embodiment of the computing system 800 capable of performing one or more of the methods and processes described above. The computing system 800 is shown in a simplified form. The computing system 800 may embody the computing system 102, camera 118, data warehouse 172, assembly robot 174, and technician device 176 described above and shown in Figure 1. The computing system 800 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, game devices, mobile computing devices, mobile communication devices (e.g., smartphones), edge devices configured to perform real-time analysis, and / or other computing devices, as well as wearable computing devices such as smartwatches and head-mounted augmented reality devices.

[0065] The computing system 800 includes a logical processor 802, a volatile memory 804, and a non-volatile storage device 806. The computing system 800 may optionally include a display subsystem 808, an input subsystem 810, a communication subsystem 812, and / or other components not shown in Figure 8.

[0066] The logical processor 802 includes one or more physical devices configured to execute instructions. For example, the logical processor 802 may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or achieve other desired results.

[0067] A logic processor may include one or more physical processors (hardware) configured to execute software instructions. In addition, or alternatively, a logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processors of the logic processor 802 may be, for example, single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The individual components of the logic processor may optionally be distributed across two or more separate devices, which may be located remotely or configured for cooperative processing. Embodiments of the logic processor may be executed by remotely accessible network-attached computing devices that are virtualized and configured in a cloud computing configuration. In such cases, it will be understood that these virtualized embodiments run on different physical logic processors on various different machines.

[0068] The non-volatile storage device 806 includes one or more physical devices configured to hold instructions executable by a logic processor for implementing the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 806 may be transformed, for example, to hold different data.

[0069] The non-volatile storage device 806 may include removable physical devices and / or built-in physical devices. The non-volatile storage device 806 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), or other mass storage technology. The non-volatile storage device 806 may include non-volatile, dynamic, static, read / write, read-only, sequential access, position-addressable, file-addressable, and / or content-addressable devices. The non-volatile storage device 806 is configured to retain instructions even when the power supply to the non-volatile storage device 806 is cut off.

[0070] The volatile memory 804 may include a physical device containing random access memory. The volatile memory 804 is typically used by the logical processor 802 to temporarily store information during the processing of software instructions. However, the volatile memory 804 typically does not retain instructions if the power supply to the volatile memory 804 is cut off.

[0071] Embodiments of the logic processor 802, volatile memory 804, and non-volatile storage device 806 may be integrated into one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), program and application-specific integrated circuits (PASICs / ASICs), program and application-specific standard products (PSSPs / ASSPs), systems-on-a-chip (SoCs), and complex programmable logic devices (CPLDs).

[0072] The terms “module,” “program,” and “engine” may be used to describe a mode of computing system 800 that is typically implemented as software by a processor and uses a portion of volatile memory to perform a specific function. This function includes a translation process that specially configures the processor to perform that function. Thus, a module, program, or engine may be embodied by a logical processor 802 executing instructions held in a non-volatile storage device 806 using a portion of volatile memory 804. Different modules, programs, and / or engines may be embodied from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be embodied from different applications, services, code block, object, library, routine, API, function, etc. The terms “module,” “program,” and “engine” may encompass individual or group executable files, data files, libraries, drivers, scripts, database records, etc.

[0073] If a display subsystem 808 is included, it may be used to present a visual representation of the data held in the non-volatile memory 806. This visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein modify the data held in the non-volatile memory, thereby transforming the state of the non-volatile memory, the state of the display subsystem 808 is also transformed, allowing for a visual representation of the changes in the underlying data. The display subsystem 808 may include one or more display devices utilizing substantially any type of technology. Such display devices may be integrated into a common enclosure with the logical processor 802, the volatile memory 804, and / or the non-volatile memory 806, or such display devices may be peripheral display devices.

[0074] If an input subsystem 810 is included, it may comprise or interface with one or more user input devices such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may comprise or interface with selected natural user input (NUI) components. Such components may be built-in or peripheral, and the conversion and / or processing of input behavior may be onboard or offboard. Examples of NUI components include microphones for speech and / or speech recognition, infrared cameras, color cameras, stereoscopic cameras and / or depth cameras for machine vision and / or gesture recognition, head trackers, eye trackers, accelerometers and / or gyroscopes for motion detection and / or intent recognition, and / or other appropriate sensors.

[0075] If a communication subsystem 812 is included, it may be configured to communicate with each other or with other devices, as described herein. The communication subsystem 812 may include wired and / or wireless communication devices compliant with one or more different communication protocols. In a non-limiting example, the communication subsystem may be configured to communicate over a wireless telephone network or a wired or wireless local area network or wide area network such as an HDMI over Wi-Fi connection. In some embodiments, the communication subsystem may enable the computing system 800 to send and receive messages with other devices over a network such as the Internet.

[0076] Furthermore, this disclosure includes configurations as follows:

[0077] In one example, the computing system comprises one or more processors, which, by executing instructions stored in memory, receive a build plan for an aircraft structure specifying multiple planned locations for multiple holes, tacks, and / or fasteners on the aircraft structure; receive multiple digital images of the aircraft structure from one or more cameras; generate a composite digital image of the aircraft structure using a composite image generation module based on at least the multiple digital images and the build plan; determine multiple regions of interest (ROIs) in the composite digital image; for each of the multiple ROIs, input the ROI into an object detection machine learning (ML) model to identify multiple actual locations for multiple holes, tacks, and / or fasteners on the aircraft structure within the ROI; generate an assembly status report of the aircraft structure, including inconsistencies between the multiple actual locations and the multiple planned locations; generate one or more event notifications regarding the aircraft structure, based on at least the assembly status report, indicating one or more corrective actions to be taken based on at least the inconsistencies; and output the one or more event notifications. In this example and / or other examples, for example, the one or more cameras are configured to project a plurality of landmark features onto the aircraft structure, the plurality of landmark features are captured in the plurality of digital images of the aircraft structure, and the composite digital image generation module is configured to generate the composite digital image of the aircraft structure based at least on aligning the plurality of landmark features between adjacent digital images of the plurality of digital images.In this example and / or other examples, for example, the synthetic digital image generation module is configured to run a feature extraction ML model configured to extract aircraft-specific features from the plurality of digital images, the feature extraction ML model is trained on a set of training digital images of a plurality of different aircraft structures labeled with aircraft-specific features and hole-system layout features, and the synthetic digital image generation module is configured to generate the synthetic digital image based on at least the aircraft-specific features output by the feature extraction ML model. In this example and / or other examples, for example, the object detection ML model is further configured to identify the location of foreign debris on the aircraft structure within each of the plurality of ROIs, the assembly status report further includes the identified location of the foreign debris on the aircraft structure, and the one or more corrective actions include sending one or more commands to remove the foreign debris from the identified location. In this example and / or other examples, for example, the object detection ML model is further configured to identify the location of defects in the holes, tucks, and / or fasteners within each of the plurality of ROIs, the assembly status report further includes the identified locations of the defects in the holes, tucks, and / or fasteners on the aircraft structure, and the one or more corrective actions include sending one or more commands to correct the defects in the holes, tucks, and / or fasteners at the identified locations on the aircraft structure. In this example and / or other examples, the object detection ML model is further configured to generate marked digital images of one or more ROIs of the aircraft structure, with the inconsistencies highlighted by visual markers, and the one or more processors are configured to execute instructions stored in memory to output the one or more marked digital images.In the example and / or other example, for example, the inconsistency includes identifying the absence of a hole at a planned location on the aircraft structure, and the one or more corrective actions include sending a command to drill a hole at the planned location on the aircraft structure. In the example and / or other example, for example, the one or more corrective actions include sending a command to stop the filling work being performed on the hole at the planned location until the hole can be drilled at the planned location. In the example and / or other example, for example, the one or more processors are configured to execute instructions stored in memory to compare the assembly status report of the aircraft structure with a previous assembly status report of the aircraft structure to verify whether the previous inconsistency has been corrected since the generation of the previous assembly status report, and to generate one or more additional event notifications indicating one or more additional corrective actions to correct the previous inconsistency based on the identification that the previous inconsistency has not been corrected.

[0078] In another example, a computer-based method receives a build plan for an aircraft structure specifying multiple planned locations for multiple holes, tacks, and / or fasteners on the aircraft structure; receives multiple digital images of the aircraft structure from one or more cameras; generates a composite digital image of the aircraft structure using a composite image generation module based on at least the multiple digital images and the build plan; determines multiple regions of interest (ROIs) in the composite digital image; identifies multiple actual locations for multiple holes, tacks, and / or fasteners on the aircraft structure within each of the multiple ROIs by inputting the ROI into an object detection machine learning (ML) model; generates an assembly status report of the aircraft structure, including inconsistencies between the multiple actual locations and the multiple planned locations; generates one or more event notifications regarding the aircraft structure, based on at least the assembly status report, indicating one or more corrective actions to be taken based on at least the inconsistencies; and outputs the one or more event notifications. In this example and / or other examples, for example, the one or more cameras are configured to project a plurality of landmark features onto the aircraft structure, the plurality of landmark features are captured in the plurality of digital images of the aircraft structure, and the composite digital image generation module is configured to generate the composite digital image of the aircraft structure based at least on aligning the plurality of landmark features between adjacent digital images of the plurality of digital images.In this example and / or other examples, the synthetic digital image generation module is configured to run a feature extraction ML model configured to extract aircraft-specific features from the plurality of digital images, the feature extraction ML model being trained on a set of training digital images of a plurality of different aircraft structures labeled with aircraft-specific features and hole-system layout features, and the synthetic digital image generation module is configured to generate the synthetic digital image based on the aircraft-specific features output by the feature extraction ML model. In this example and / or other examples, the object detection ML model is further configured, for example, to identify the location of foreign debris on the aircraft structure within each of the plurality of ROIs, the assembly status report further includes the identified location of the foreign debris on the aircraft structure, and the one or more corrective actions include sending one or more commands to remove the foreign debris from the identified location. In this example and / or other examples, for example, the object detection ML model is further configured to identify the location of defects in the holes, tucks, and / or fasteners within each of the plurality of ROIs, the assembly status report further includes the identified locations of the defects in the holes, tucks, and / or fasteners on the aircraft structure, and the one or more corrective actions include sending one or more commands to correct the defects in the holes, tucks, and / or fasteners at the identified locations on the aircraft structure. In this example and / or other examples, for example, the object detection ML model is further configured to generate marked digital images of one or more ROIs of the aircraft structure in which the inconsistencies are highlighted by visual markers, and the method performed by the computer further includes outputting the one or more marked digital images.In the example and / or other example, for example, the inconsistency includes identifying that there is no hole at a planned location on the aircraft structure, and the one or more corrective actions include sending a command to drill a hole at the planned location on the aircraft structure. In the example and / or other example, for example, the one or more corrective actions include sending a command to stop the filling work being performed on the hole at the planned location until the hole can be drilled at the planned location. In the example and / or other example, the method implemented by the computer further includes, for example, comparing the assembly status report of the aircraft structure with a previous assembly status report of the aircraft structure to verify whether the previous inconsistency has been corrected since the generation of the previous assembly status report, and, based on the identification that the previous inconsistency has not been corrected, generating one or more additional event notifications indicating one or more additional corrective actions to correct the previous inconsistency.

[0079] In yet another example, the computing system comprises one or more processors, which, by executing instructions stored in memory, receive a build plan for an aircraft structure specifying multiple planned locations for multiple holes, tacks, and / or fasteners on the aircraft structure; receive multiple digital images of the aircraft structure from one or more cameras; generate a composite digital image of the aircraft structure by a composite image generation module based on at least the multiple digital images and the build plan; input the composite digital image into an object detection machine learning (ML) model to identify multiple actual locations for multiple holes, tacks, and / or fasteners on the aircraft structure in the composite digital image; generate an assembly status report of the aircraft structure, including inconsistencies between the multiple actual locations and the multiple planned locations; generate one or more event notifications regarding the aircraft structure, based on at least the assembly status report, indicating one or more corrective actions to be taken based on at least the inconsistencies; and output the one or more event notifications. In the present and / or other examples, for example, the one or more processors determine multiple regions of interest (ROIs) of the composite digital image by executing instructions stored in memory, and for each of the multiple ROIs, the ROI is input into the object detection ML model to identify multiple actual locations of multiple actual holes, tacks, and / or fasteners on the aircraft structure within the ROI, and the assembly status report of the aircraft structure includes inconsistencies between the multiple actual locations and the multiple planned locations in the multiple ROIs.

[0080] Furthermore, the configurations and / or methods described herein are illustrative in nature, and these specific embodiments or examples should not be construed as limiting, as numerous modifications are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Accordingly, the various operations illustrated and / or described may be performed in the order illustrated and / or described, in other orders, in parallel, or omitted. Similarly, the order of the processing described above may be changed.

[0081] The subject matter of this disclosure includes any novel and non-obvious combinations and partial combinations of the various processes, systems and configurations disclosed herein, as well as any other features, functions, operations and / or characteristics, and all equivalents thereof.

Claims

1. A computing system comprising one or more processors, wherein the processor executes instructions stored in memory, A build plan for an aircraft structure is received, specifying multiple planned locations for multiple holes, tacks, and / or fasteners on the aircraft structure. Multiple digital images of the aircraft structure are received from one or more cameras. Based on at least the plurality of digital images and the build plan, a composite image generation module generates a composite digital image of the aircraft structure. Multiple regions of interest (ROIs) of the aforementioned synthesized digital image are determined, For each of the aforementioned multiple ROIs, the ROI is input into an object detection machine learning (ML) model to identify multiple actual locations of multiple actual holes, tacks, and / or fasteners on the aircraft structure within the ROI. A report on the assembly status of the aircraft structure is generated, including the inconsistencies between the multiple actual positions and the multiple planned positions. Based on the assembly status report, generate one or more event notifications relating to the aircraft structure, indicating at least one or more corrective actions to be taken based on the aforementioned inconsistency. A computing system configured to output one or more of the aforementioned event notifications.

2. The computing system according to claim 1, wherein one or more cameras are configured to project a plurality of landmark features onto the aircraft structure, the plurality of landmark features are captured in the plurality of digital images of the aircraft structure, and the composite digital image generation module is configured to generate the composite digital image of the aircraft structure based on aligning the plurality of landmark features between adjacent digital images of the plurality of digital images.

3. The computing system according to claim 1, wherein the composite digital image generation module is configured to run a feature extraction ML model configured to extract aircraft-specific features from the plurality of digital images, the feature extraction ML model is trained on a set of training digital images of a plurality of different aircraft structures labeled with aircraft-specific features and hole-type layout features, and the composite digital image generation module is configured to generate the composite digital image based on the aircraft-specific features output by the feature extraction ML model.

4. The computing system according to claim 1, wherein the object detection ML model is further configured to identify the location of foreign debris on the aircraft structure within each of the plurality of ROIs, the assembly status report further includes the identified location of the foreign debris on the aircraft structure, and the one or more corrective actions include sending one or more commands to remove the foreign debris from the identified location.

5. The computing system according to claim 1, wherein the object detection ML model is further configured to identify the location of defects in the plurality of holes, tucks, and / or fasteners within each of the plurality of ROIs, the assembly status report further includes the identified locations of the defects in the holes, tucks, and / or fasteners on the aircraft structure, and the one or more corrective actions include transmitting one or more commands to correct the defects in the holes, tucks, and / or fasteners at the identified locations on the aircraft structure.

6. The computing system according to claim 1, wherein the object detection ML model is further configured to generate marked digital images of one or more ROIs of the aircraft structure in which the inconsistencies are highlighted by visual markers, and the one or more processors are configured to execute instructions stored in memory to output the one or more marked digital images.

7. The computing system according to claim 1, wherein the inconsistency includes identifying the absence of a hole at a planned location on the aircraft structure, and the one or more corrective actions include transmitting a command to drill a hole at the planned location on the aircraft structure.

8. The computing system according to claim 7, wherein the one or more corrective measures include sending a command to stop the filling operation being performed on the hole at the planned location until the hole can be drilled at the planned location.

9. The aforementioned one or more processors execute instructions stored in memory, The assembly status report of the aircraft structure is compared with a previous assembly status report of the aircraft structure to verify whether the previous inconsistency was corrected after the generation of the previous assembly status report. The computing system according to any one of claims 1 to 8, configured to generate one or more additional event notifications indicating one or more additional corrective actions to correct a previous inconsistency, based on the identification that a previous inconsistency has not been corrected.

10. A method performed by a computer, A build plan for an aircraft structure is received, specifying multiple planned locations for multiple holes, tacks, and / or fasteners on the aircraft structure. Multiple digital images of the aircraft structure are received from one or more cameras. Based on at least the plurality of digital images and the build plan, a composite image generation module generates a composite digital image of the aircraft structure. Multiple regions of interest (ROIs) of the aforementioned synthesized digital image are determined, For each of the aforementioned multiple ROIs, the ROI is input into an object detection machine learning (ML) model to identify multiple actual locations of multiple actual holes, tacks, and / or fasteners on the aircraft structure within the ROI. A report on the assembly status of the aircraft structure is generated, including the inconsistencies between the multiple actual positions and the multiple planned positions. Based on the assembly status report, generate one or more event notifications relating to the aircraft structure, indicating at least one or more corrective actions to be taken based on the aforementioned inconsistency. A method for outputting one or more of the aforementioned event notifications.

11. The computer-based method according to claim 10, wherein one or more cameras are configured to project a plurality of landmark features onto the aircraft structure, the plurality of landmark features are captured within the plurality of digital images of the aircraft structure, and the composite digital image generation module is configured to generate the composite digital image of the aircraft structure based on aligning the plurality of landmark features between adjacent digital images of the plurality of digital images.

12. The aforementioned composite digital image generation module is configured to execute a feature extraction ML model configured to extract aircraft-specific features from the plurality of digital images. The computer-based method according to claim 10, wherein the feature extraction ML model is trained on a set of training digital images of a plurality of different aircraft structures labeled with aircraft-specific features and hole-type layout features, and the synthetic digital image generation module is configured to generate the synthetic digital image based on at least the aircraft-specific features output by the feature extraction ML model.

13. The computer-based method according to claim 10, wherein the object detection ML model is further configured to identify the location of foreign debris on the aircraft structure within each of the plurality of ROIs, the assembly status report further includes the identified location of the foreign debris on the aircraft structure, and the one or more corrective actions include transmitting one or more commands to remove the foreign debris from the identified location.

14. The computer-based method according to claim 10, wherein the object detection ML model is further configured to identify the location of defects in the plurality of holes, tucks, and / or fasteners within each of the plurality of ROIs, the assembly status report further includes the identified locations of the defects in the holes, tucks, and / or fasteners on the aircraft structure, and the one or more corrective actions include transmitting one or more commands to correct the defects in the holes, tucks, and / or fasteners at the identified locations on the aircraft structure.

15. The object detection ML model is further configured to generate marked digital images of one or more ROIs of the aircraft structure in which the inconsistencies are highlighted by visual markers, and the computer-implemented method further comprises outputting the one or more marked digital images, according to claim 10.

16. The computer-based method according to claim 10, wherein the inconsistency includes identifying the absence of a hole at a planned location on the aircraft structure, and the one or more corrective actions include transmitting a command to drill a hole at the planned location on the aircraft structure.

17. The one or more corrective actions include sending a command to stop the filling operation being performed at the planned location of the hole until the hole can be drilled at the planned location. The computer-based method described in claim 16.

18. Furthermore, the assembly status report of the aircraft structure is compared with a previous assembly status report of the aircraft structure to verify whether the previous inconsistency was corrected after the generation of the previous assembly status report. A computer-based method according to any one of claims 10 to 17, which, based on the identification that a previous inconsistency has not been corrected, generates one or more additional event notifications indicating one or more additional corrective actions to correct said previous inconsistency.

19. A computing system comprising one or more processors, wherein the processor executes instructions stored in memory, A build plan for an aircraft structure is received, specifying multiple planned locations for multiple holes, tacks, and / or fasteners on the aircraft structure. Multiple digital images of the aircraft structure are received from one or more cameras. Based on at least the plurality of digital images and the build plan, a composite image generation module generates a composite digital image of the aircraft structure. By inputting the aforementioned synthesized digital image into an object detection machine learning (ML) model, the actual locations of multiple actual holes, tacks, and / or fasteners on the aircraft structure within the synthesized digital image are identified. A report on the assembly status of the aircraft structure is generated, including the inconsistencies between the multiple actual positions and the multiple planned positions. Based on the assembly status report, generate one or more event notifications relating to the aircraft structure, indicating at least one or more corrective actions to be taken based on the aforementioned inconsistency. A computing system configured to output one or more of the aforementioned event notifications.

20. The aforementioned one or more processors execute instructions stored in memory, Multiple regions of interest (ROIs) of the aforementioned synthesized digital image are determined, The computing system according to claim 19, wherein for each of the plurality of ROIs, the ROI is input into the object detection ML model to identify a plurality of actual locations of a plurality of actual holes, tacks, and / or fasteners on the aircraft structure within the ROI, and the assembly status report of the aircraft structure includes any inconsistencies between the plurality of actual locations and the plurality of planned locations in the plurality of ROIs.

Images