Computer-automated image-based detection of aircraft structures

Abstract

This application relates to computer automated image based inspection of aircraft structures. Examples are disclosed relating to computer automated vision based methods for inspecting aircraft structures to track assembly status. In one example, a build plan for an aircraft structure is received. An image of the aircraft structure is received. A stitched image of the aircraft structure is generated based on the image and the build plan. Regions of interest (ROIs) of the stitched image are determined. For each ROI, the ROI is input into an object detection machine learning model to identify actual locations of actual holes, pins, and / or fasteners on the aircraft structure in the corresponding ROI. An assembly status report is generated, the assembly status report including differences between the actual locations and expected locations specified in the build plan. An event notification is generated indicating to take remedial action according to the differences from the assembly status report. The event notification is output.

Description

Technical Field

[0001] This disclosure generally relates to inspecting aircraft structures, and more specifically, to inspecting the assembly status of aircraft structures in a computer-automated manner. Background Technology

[0002] During aircraft assembly, structural elements such as wings or fuselages are inspected to verify that all components and systems meet specific standards before being assembled into a larger structure. The results of these inspections are documented in an Assembly Status (CoA) report to ensure the aircraft complies with regulatory standards, design specifications, and safety requirements. More specifically, the CoA report is used to identify discrepancies between the actual aircraft structure and the planned aircraft structure construction.

[0003] Traditionally, the assembled aircraft structure is manually inspected by human inspectors who compare it to a hard copy of the aircraft structure's build plan. More specifically, in one example, the human inspector might check whether the correct number, type, and location of holes, nails, and / or fasteners are installed on the aircraft structure as specified in the build plan. By inspecting the aircraft structure to ensure that holes, nails, and / or fasteners are installed as specified in the build plan, the aircraft structure can be maintained within the specifications of the build plan throughout the assembly process. The human inspector manually records the inspection results in a corresponding hard copy of the aircraft structure's CoA report. This conventional method of manually inspecting the aircraft structure and manually creating corresponding CoA reports recording the inspection results is time-consuming, resulting in extended downtime between steps in the aircraft assembly process and increasing the overall cost of the aircraft. Furthermore, this conventional method of manually inspecting the aircraft structure by human inspectors can be error-prone, potentially leading to additional delays in aircraft assembly and build quality issues. Summary of the Invention

[0004] Examples of a computer-automated, vision-based method for inspecting an aircraft structure to track its assembly status are disclosed. In one example, a build plan for the aircraft structure is received, specifying multiple expected locations of a plurality of holes, nails, and / or fasteners on the aircraft structure. Multiple digital images of the aircraft structure are received from one or more cameras. A stitched digital image of the aircraft structure is generated via a stitching digital image generation module, based at least on the plurality of digital images and the build plan. A plurality of regions of interest (ROIs) in the stitched digital image are determined. For each of the plurality of ROIs, the corresponding ROI is input into an object detection machine learning (ML) model to identify multiple actual locations of a plurality of actual holes, nails, and / or fasteners on the aircraft structure within the corresponding ROI. An assembly status report of the aircraft structure is generated, including differences between the multiple actual locations and the multiple expected locations. One or more event notifications for the aircraft structure are generated, based at least on the assembly status report. The one or more event notifications indicate one or more remedial actions to be taken, at least based on the differences. One or more event notifications are output.

[0005] This summary is provided to present, in a simplified form, the selection of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to the implementation of solutions to any or all the shortcomings mentioned in any part of this disclosure. Attached Figure Description

[0006] Figure 1 An example computing environment according to one embodiment of the present disclosure is illustrated, wherein a computer-automated vision-based method for inspecting an aircraft structure to track its assembly status is implemented.

[0007] Figure 2 An example processing for generating a stitched digital image of an aircraft structure according to an embodiment of the present disclosure is illustrated schematically.

[0008] Figure 3 A generation according to an embodiment of the present disclosure is illustrated schematically. Figure 2 Example processing of multiple regions of interest (ROIs) in a stitched digital image of the aircraft structure shown.

[0009] Figure 4 An example assembly status (CoA) report for an aircraft structure generated according to one embodiment of this disclosure is shown.

[0010] Figure 5 It shows Figure 2An exemplary masked digital image of the ROI of the aircraft structure shown, wherein the differences between multiple actual locations and multiple expected locations of multiple holes, nails and / or fasteners on the aircraft structure are highlighted with visual markers.

[0011] Figures 6 to 7 An example computer-implemented method for inspecting the assembly status of an aircraft structure, according to an embodiment of this disclosure, is shown.

[0012] Figure 8 The illustration shows the representation. Figure 1 Example computing systems for any computing system in the computing environment shown.

[0013] Parts list:

[0014] Computing environment 100

[0015] Computing System 102

[0016] Aircraft Structure 104

[0017] More processors 106

[0018] Memory 108

[0019] Construction Plan 110

[0020] Expected position 112

[0021] Fastener 114

[0022] Digital Image 116

[0023] Camera 118

[0024] 120 stitched digital image generation module

[0025] Stitching digital images 122

[0026] Feature Extraction ML Model 124

[0027] Feature matching ML model 126

[0028] Image stitching ML model 128

[0029] Landmark Features 130

[0030] Metadata 132

[0031] ROI Determination Module 134

[0032] ROI 136

[0033] Object detection ML model 138

[0034] CoA Report 140

[0035] Object tracking data 142

[0036] Actual location 146

[0037] Fastener 148

[0038] Difference 150

[0039] Type 152

[0040] Total 154

[0041] Defect 156

[0042] Position 158

[0043] 160 foreign object fragments

[0044] Timestamp 162

[0045] Masking digital images 164

[0046] Event Notification Module 166

[0047] Event Notification 168

[0048] Remedial Action 170

[0049] Data Warehouse 172

[0050] Assembly Robot 174

[0051] Human technicians and equipment 176

[0052] Process 200

[0053] Aircraft Wing 202

[0054] Process 300

[0055] Visual markers 300.1 to 300.4

[0056] Computer Implementation Method 600

[0057] Computing System 800

[0058] Logic Processor 802

[0059] 804 Volatile Memory

[0060] Non-volatile storage device 806

[0061] Display Subsystem 808

[0062] Input Subsystem 810

[0063] Communication Subsystem 812. Detailed Implementation

[0064] Routine manual inspections of aircraft structures performed by one or more human inspectors are time-consuming and error-prone, potentially leading to additional delays in aircraft assembly and issues with the aircraft's build quality. Furthermore, the conventional method of manually generating corresponding CoA (Content of Assembly) reports documenting the results of these manual inspections consumes significant time. These traditional manual inspection and documentation methods result in prolonged downtime between steps in the aircraft assembly process and increase the overall cost of the aircraft.

[0065] Therefore, to address these and other issues discussed herein, examples of a vision-based, computer-automated method for inspecting aircraft structures to track CoA are disclosed. In one example, multiple digital images of a large aircraft structure (such as an aircraft wing or fuselage) are used to generate a stitched digital image of the aircraft structure. Multiple regions of interest (ROIs) are determined from the stitched digital image. For each of the multiple ROIs, the corresponding ROI is fed into an object detection machine learning (ML) model to identify multiple actual locations of multiple actual holes, pins, and / or fasteners on the aircraft structure within the corresponding ROI. A CoA report for the aircraft structure is generated, comprising the differences between the multiple actual locations of the multiple actual holes, pins, and / or fasteners on the aircraft structure and the multiple expected locations of the multiple expected holes, pins, and / or fasteners on the aircraft structure as specified in the aircraft structure's construction plan. One or more event notifications for the aircraft structure are generated based at least on the CoA report. The one or more event notifications indicate one or more remedial actions to be taken, at least based on the differences. The one or more event notifications are output.

[0066] The computer-automated vision-based method for inspecting aircraft structures disclosed in this paper provides a simple, fast, reliable, and non-invasive way to automate, detect, and inspect drilling and fastening operations performed during the assembly of aircraft structures. Furthermore, the method leverages the use of an ML model specifically trained to identify, count, and compare holes, nails, and / or fasteners on the aircraft structure against the aircraft structure's build plan, thereby automatically generating a CoA report that is faster, more accurate, and more robust than traditional manual inspection and recording methods.

[0067] Figure 1 An example computing environment 100 is schematically shown, in which computing system 102 is configured to perform a computer-automated vision-based method for inspecting aircraft structure 104 to track CoA.

[0068] Aircraft structure 104 may include any suitable aircraft part or component. Typically, aircraft structure 104 is large enough that the entire structure cannot be captured in a single digital image with sufficient resolution to capture all features tracked via CoA inspection. In some examples, aircraft structure 104 includes the fuselage of an aircraft. In other examples, aircraft structure 104 includes the wings of an aircraft. In still other examples, aircraft structure 104 includes another large component of the aircraft.

[0069] The computing system 102 includes one or more processors 106 configured to execute instructions stored in memory 108 to perform computational operations related to a computer-automated, vision-based approach for examining the aircraft structure 104 disclosed herein. In one example, the processors 106 are configured to execute instructions stored in memory 108 to receive a build plan 110 for the aircraft structure 104, specifying multiple expected locations 112 of expected holes, pins, and / or fasteners 114 on the aircraft structure 104. In some examples, the build plan 110 includes hole-based layout features, which are specific properties or characteristics associated with holes (such as fastener holes, rivet holes, or bolt holes) in structural components. These hole-based layout features provide a reference for aligning and connecting components of the aircraft structure 104. The precise number, sequence, placement, and drilling of holes (and / or pins / fasteners) on the aircraft structure 104 are critical for maintaining tolerances and structural integrity. The size, shape, and spacing of the holes affect the load distribution and strength of the joint. Hole-based layout features simplify the assembly process by allowing automated drilling and inspection systems to locate and work with predefined positions of the expected multiple holes, pins, and / or fasteners 114 on the aircraft structure 104.

[0070] One or more processors 106 are configured to execute instructions stored in memory 108 to receive multiple digital images 116 of the aircraft structure 104 from one or more cameras 118. In some examples, the cameras 118 include multiple cameras positioned throughout the facility where the aircraft structure 104 is being assembled. In some examples, at least one of the cameras 118 includes a camera in a fixed position (e.g., X, Y, Z) and / or orientation (e.g., pitch, roll, azimuth). For example, the multiple cameras may be positioned at different locations along the assembly line of the aircraft structure 104, and the aircraft structure 104 may move relative to the multiple cameras while it is being assembled.

[0071] In some examples, at least one of the cameras 118 includes a camera that can be moved to different positions and / or orientations. In some examples, the movable camera is attached to a robot that moves to different positions within a facility where the aircraft structure 104 is being assembled. In other examples, the movable camera is attached to a robotic arm that can adjust the position and / or orientation of the camera to capture digital images 116 of the aircraft structure 104 from different perspectives. In still other examples, the movable camera may include a handheld camera held by a human operator to capture digital images 116 of the aircraft structure 104 from different perspectives.

[0072] In some examples, camera 118 includes cameras already used for other purposes (not just CoA inspection) related to the assembly of aircraft structure 104. For example, camera 118 may have already been used by a human operator to observe aircraft structure 104 while manually controlling a drilling machine, or to supervise a robot performing automated drilling on aircraft structure 104. By employing cameras already used for other purposes related to the assembly of aircraft structure 104, computer-automated vision-based CoA inspection methods can be implemented in a non-invasive manner or otherwise with minimal interference to the assembly operations performed on aircraft structure 104.

[0073] Camera 118 can take any suitable form. For example, camera 118 may include one or more of a visible light camera, a depth camera, a LiDAR sensor, another suitable type of camera, or a combination thereof.

[0074] In some embodiments, camera 118 is configured to capture a sequence of digital images 116 as a video stream, and multiple digital images 116 can be extracted from the video stream. In other embodiments, the video stream is processed and analyzed as a whole; however, this would consume more computational resources than processing and analyzing a subset of the digital images extracted from the video stream.

[0075] Typically, the size of the aircraft structure 104 is quite large, and capturing the entire aircraft structure 104 (or at least the area of ​​the aircraft structure 104 being inspected for CoA) in a single digital image with sufficiently high image resolution makes it impractical for multiple holes, pins, and / or fasteners to be perceived with sufficient detail for computer-automated inspection to assess the CoA of the aircraft structure 104. Therefore, the computing system 102 is configured to generate a stitched digital image 122 based at least on multiple digital images 116 of the aircraft structure 104, which shows multiple holes, pins, and / or fasteners of the aircraft structure 104 at a suitably high enough image resolution for computer-automated inspection to assess the CoA of the aircraft structure 104.

[0076] In one example, processor 106 is configured to execute instructions stored in memory 108 to execute a stitched digital image generation module 120, which is configured to generate a stitched digital image 122 of the aircraft structure 104 based at least on a plurality of digital images 116 and a build plan 110. More specifically, the stitched digital image generation module 120 may use the build plan 110 as a reference to verify that the entire aircraft structure 104 (or at least the area of ​​the aircraft structure 104 being checked against CoA) is captured by the plurality of digital images 116, and that the stitched digital image 122 includes the entire aircraft structure 104 (or at least the area of ​​the aircraft structure 104 being checked against CoA).

[0077] In some embodiments, the stitched digital image generation module 120 executes one or more machine learning (ML) models trained to perform various computational operations related to generating the stitched digital image 122. In one example, the stitched digital image generation module 120 is configured to execute a feature extraction ML model 124, a feature matching ML model 126, and an image stitching ML model 128.

[0078] The feature extraction ML model 124 is configured to identify and extract features (including aircraft-specific features as well as more primitive features) from multiple digital images 116 and generate descriptors for the extracted features.

[0079] Different training methods can be used to train the feature extraction ML model 124. In one example, supervised learning based at least on a training dataset is used to train the feature extraction ML model 124, which includes digital images of multiple different aircraft structures labeled with aircraft-specific features and aperture-based layout features that are common features of the aircraft structures for CoA checking. For example, aircraft-specific features and aperture-based layout features may include wing edges, fuselage curvature, seams between panels, tool features, aperture-based layout feature markers, and other aircraft-specific features. By training the feature extraction ML model 124 with aircraft structure-specific labeled training data, the feature extraction ML model 124 can more accurately identify aircraft structure-specific features 104 in multiple digital images 116. In other examples, unsupervised learning is used to train the feature extraction ML model 124, where the training data includes unlabeled digital images of aircraft structures, and unsupervised techniques are applied to refine the latent spatial representation of the training data into a meaningful and compressed representation that captures the underlying structure or patterns in the training data. In addition, self-supervised learning techniques can be used to train the feature extraction ML model 124, which may involve generating pseudo-labels or auxiliary targets to construct a latent spatial representation of the training data.

[0080] The feature extraction ML model 124 can be implemented using various types of ML models. In one example, a convolutional neural network (CNN) is used to implement the feature extraction ML model 124. In another example, a specialized deep learning model (such as D2-Net, R2D2, or SuperPoint) is used to implement the feature extraction ML model 124. In yet another example, a hybrid approach combining deep learning-based methods with traditional neural network methods is used to further enhance feature extraction from multiple digital images 116 to implement the feature extraction ML model 124. In other examples, different types of ML models are used to implement the feature extraction ML model 124.

[0081] Feature matching ML model 126 is configured to receive features and corresponding descriptors extracted from multiple digital images 116 by feature extraction ML model 124. Feature matching ML model 126 is configured to match the extracted features among different digital images of the multiple digital images 116 by comparing the descriptors. The extracted features matched among the digital images are used to identify which digital images correspond to which parts of the aircraft structure, so as to correctly align the multiple digital images 116 to form a stitched digital image 122.

[0082] The feature matching ML model 126 can be implemented using various types of ML models. In one example, a deep learning model (e.g., SuperGlue, LoFTR) is used to implement the feature matching ML model 126. In other examples, different types of ML models are used to implement the feature matching ML model 126. In still other examples, the stitching digital image generation module 120 can perform feature matching using classic distance-based matching techniques such as Scale Invariant Feature Transform (SIFT) and Speed-Up Robust Feature Transform (SURF) techniques.

[0083] In some embodiments, the accuracy of feature matching can be enhanced at least based on features of a camera 118 used to capture multiple digital images 116. In some embodiments, the camera 118 is configured to project multiple landmark features 130 onto the aircraft structure 104. Multiple landmark features 130 are captured in the multiple digital images 116 of the aircraft structure 104. A stitched digital image generation module 120 is configured to generate a stitched digital image 122 of the aircraft structure 104 based at least on aligning the landmark features 130 between adjacent digital images in the multiple digital images 116. In embodiments employing a feature matching ML model 126 for feature matching, the feature matching ML model 126 is configured to match landmark features 130 between adjacent digital images of the multiple digital images 116.

[0084] Image stitching ML model 128 is configured to generate stitched digital image 122 based at least on digital images with matching features aligned among a plurality of digital images 116. In one example, image stitching ML model 128 is configured to generate stitched digital image 122 by calculating a transformation matrix (e.g., homography) between the digital images of the plurality of digital images 116 using matching features output by feature matching ML model 126. Furthermore, image stitching ML model 128 is configured to warp the digital images as needed to align them with each other. In some embodiments, image stitching ML model 128 is configured to use blending techniques to produce smooth transitions between digital images. In some embodiments, image stitching ML model 128 is configured to match textures and colors between stitched images. In some embodiments, image stitching ML model 128 is configured to correct for parallax problems that may occur when objects at different depths are misaligned in overlapping digital images.

[0085] In some embodiments, the digital image generation module 120 may enhance the accuracy of stitching the digital image 122 based at least on metadata 132 associated with the plurality of digital images 116. In some examples, the metadata 132 may include camera position and / or orientation data indicating the position and / or orientation of the camera when the digital image is captured by the camera. The feature matching ML model 126 may use the camera position and / or orientation data to facilitate feature matching between the digital images of the plurality of digital images 116. Furthermore, the image stitching ML model 128 may use the camera position and / or orientation data to align the different digital images in the plurality of digital images 116 to generate the stitched digital image 122. In some embodiments, the metadata 132 includes depth data provided by a depth camera or LIDAR sensor. The depth data may be used by the feature extraction ML model 124 to identify the location of features in the plurality of digital images 116 based at least on the depth of features in the digital images.

[0086] Figure 2 An example process 200 for generating stitched digital images of an aircraft structure according to one embodiment of the present disclosure is illustrated. In the illustrated process 200, the aircraft structure being inspected for the CoA is an aircraft wing 202. The aircraft wing 202 is large enough that it is not easily captured with a single camera at a sufficiently high image resolution to capture all the holes, pins, and / or fasteners on the aircraft wing 202. Instead, multiple cameras 118 (e.g., 118.1, 118.2, 118.3, 118.4, 118.5) capture multiple digital images 116 (e.g., 116.1, 116.2, 116.3, 116.4, 116.5) of different areas of the aircraft wing 202. For example, the multiple cameras 118 may be positioned throughout the facility where the aircraft wing 202 is being assembled.

[0087] In the illustrated embodiment, a plurality of cameras 118 are configured to project a plurality of landmark features 130 onto the aircraft wing 202. The plurality of landmark features 130 are captured in a plurality of digital images 116 of the aircraft wing 202. For example, a first set 130.1 of landmark features is captured in a first digital image 116.1, a second set 130.2 of landmark features is captured in a second digital image 116.2, a third set 130.3 of landmark features is captured in a third digital image 116.3, a fourth set 130.4 of landmark features is captured in a fourth digital image 116.4, a fifth set 130.5 of landmark features is captured in a fifth digital image 116.5, and a sixth set 130.6 of landmark features is captured in a sixth digital image 116.6.

[0088] Figure 1The stitched digital image generation module 120 shown is configured to generate a stitched digital image 122 of the aircraft wing 202 based at least on alignment landmark features 130 between adjacent digital images in a plurality of digital images 116. More specifically, landmark features 130 provide features that can be used to align adjacent digital images during digital image stitching processing.

[0089] The stitched digital image 122 includes the entirety of the aircraft wing 202 (or at least the area of ​​the aircraft wing being inspected for the CoA), including all holes, pins, and / or fasteners to be inspected for the CoA. The stitched digital image 122 is then used for computer-automated inspection to determine the CoA of the aircraft wing 202.

[0090] Return to Figure 1 In some embodiments, processor 106 is configured to execute instructions stored in memory 108 to perform a Region of Interest (ROI) determination module 134. ROI determination module 134 is configured to receive a stitched digital image 122 output from stitched digital image generation module 120 and determine multiple ROIs 136 of the stitched digital image 122 for visual inspection of the CoA. The stitched digital image 122 can be sliced / subdivided into multiple ROIs 136, which can be visually inspected individually for each CoA, which is generally more computationally efficient than visually inspecting the entire stitched digital image 122 at once. Furthermore, limiting the visual inspection area to ROIs can increase the accuracy of identifying the locations of holes, nails, and / or fasteners on the aircraft structure 104 compared to visually inspecting the entire stitched digital image 122. However, in some embodiments, the CoA of the entire stitched digital image 122 can be visually inspected without slicing / subdividing it into multiple ROIs.

[0091] Multiple ROIs 136 can be specified as any suitable size smaller than the stitched digital image 122. In some examples, each of the multiple ROIs 136 is set to the same specified size / dimension. In one example, ROI 136 is a 400x400 pixel square. In other examples, ROIs 136 are different specified sizes / dimensions. In other examples, ROIs 136 can be different sizes / dimensions. In some such examples, ROIs 136 can be dynamically selected by the ROI determination module 134 based at least on the expected locations of expected features on the aircraft structure 104 as specified by the construction plan 110. In some examples, each of the multiple ROIs 136 does not overlap with each other. In other examples, at least one of the ROIs 136 at least partially overlaps with at least one other ROI among the ROIs 136.

[0092] Figure 3A generation according to an embodiment of the present disclosure is illustrated schematically. Figure 2 The process 300 illustrates an example of multiple Regions of Interest (ROIs) in a stitched digital image 122 of the aircraft wing 202 shown. In the example shown, the multiple ROIs 136 are all non-overlapping squares of the same size. The ROI determination module 134 aligns a first ROI 136.1 with the upper left corner of the stitched digital image 122 and then moves from left to right across the stitched digital image 122, slicing / subdividing it into ROIs 136. The ROI determination module 134 continues slicing / subdividing the stitched digital image 122 until it reaches the lower right corner and the entire stitched digital image 122 is sliced / subdivided into multiple ROIs 136. In some examples, each of the multiple ROIs 136 is indexed to identify the corresponding ROI within the stitched digital image 122 and / or its position relative to other ROIs within the stitched digital image 122. The process 300 for generating multiple ROIs of the stitched digital image 122 is provided as a non-limiting example. In other examples, the stitched digital image 122 can be sliced / subdivided in different ways.

[0093] In some embodiments, the ROI determination module 134 is configured to identify regions in the stitched digital image 122 that are not of interest and can be omitted from the CoA check. For example, referring to the stitched digital image 122 of an aircraft wing 202, regions of the stitched digital image 122 that do not include the aircraft wing 202 (such as regions that only include the background and do not include the aircraft wing 202) can be identified as not of interest and can be omitted from the CoA check.

[0094] Return to Figure 1 The processor 106 is configured to execute instructions stored in memory 108 to execute an object detection ML model 138, which is configured to receive a build plan 110 for the aircraft structure 104 and multiple ROIs 136 as input, and to generate a CoA report 140 based at least on visual inspections of the multiple ROIs 136. The CoA report 140 includes object tracking data 142 for objects identified by the object detection ML model 138 on the aircraft structure 104.

[0095] Figure 4A CoA report 140 of an aircraft structure 104 generated by an object detection ML model 138 according to one embodiment of the present disclosure is illustrated schematically. The object detection ML model 138 is configured to identify, for each of a plurality of ROIs 136, a plurality of actual locations 146 of actual holes, pins, and / or fasteners 148 on the aircraft structure 104 within the corresponding ROI. In some examples, the actual locations 146 of the plurality of actual holes, pins, and / or fasteners 148 are represented as pixel groups. In other examples, the actual locations 146 of the plurality of actual holes, pins, and / or fasteners 148 are represented as point cloud locations. In some embodiments, the CoA report lists the expected plurality of holes, pins, and / or fasteners 114, the corresponding plurality of expected locations 112, the actual plurality of holes, pins, and / or fasteners 148, and the corresponding plurality of actual locations 146.

[0096] The object detection ML model 138 is configured to compare multiple actual locations 146 of multiple actual holes, nails, and / or fasteners 142 with expected locations 112 of expected holes, nails, and / or fasteners 114 specified in the build plan to identify discrepancies 150. For example, discrepancies 150 may include actual holes, nails, and / or fasteners being placed in the wrong location, missing actual holes, nails, and / or fasteners, additional actual holes, nails, and / or fasteners, or actual holes, nails, and / or fasteners of a different type than expected (e.g., actual holes instead of expected fasteners). In some embodiments, discrepancies 150 are listed in a CoA report 140.

[0097] In some embodiments, the object detection ML model 138 is configured to identify a specific type 152 of actual holes, nails, and / or fasteners 148 (e.g., rivets, screws, bolts). In some embodiments, the type 152 of each of the actual holes, nails, and / or fasteners 148 is listed in the CoA report 140.

[0098] In some embodiments, the object detection ML model 138 is configured to track the total number 154 of actual holes, pins, and / or fasteners 148 identified on the aircraft structure 104. The total number 154 can be used as an overview to quickly evaluate the advanced CoA of the aircraft structure 104. In some embodiments, the total number 154 of actual holes, pins, and / or fasteners 148 is listed in the CoA report 140.

[0099] In some embodiments, the object detection ML model 138 is further configured to, for each of the plurality of ROIs 136, identify the location 158 of a foreign object debris 160 on the aircraft structure 104 within the corresponding ROI. The foreign object debris 160 includes objects not expected to be located on the aircraft structure 104. For example, the foreign object debris 160 may include material debris (e.g., metal scrapers in holes), tools, and other foreign objects (e.g., sealant). In some embodiments, the foreign object debris 160 may also be included in a difference 150. In some embodiments, the identified locations 158 of the foreign object debris 160 on the aircraft structure 104 are listed in the CoA report 140.

[0100] In some embodiments, the object detection ML model 138 is also configured to, for each of a plurality of ROIs, identify the location of a defect 156 in a plurality of actual holes, pins, and / or fasteners 148 within the corresponding ROI. Defects may include holes, pins, and / or fasteners placed in the wrong location, holes having incorrect shape and / or depth, and / or other defects. In some examples, the object detection ML model 138 is configured to identify defects 156 including incorrect fastener types / part numbers being installed on the aircraft structure 104. In some embodiments, the locations of the identified defects 156 in actual holes, pins, and / or fasteners on the aircraft structure 104 are listed in the CoA report 140.

[0101] In some embodiments, the CoA report 140 generated by the object detection ML model 138 includes a timestamp 162 indicating the time when the CoA report was generated. The object detection ML model 138 can be configured to generate CoA reports at least based on different stitched digital images of the aircraft structure 104 indicated by different timestamps throughout the assembly process in order to track the CoA of the aircraft structure 104.

[0102] In some embodiments, the object detection ML model 138 is also configured to generate one or more masked digital images 164 of the aircraft structure 104, wherein at least differences 150 are highlighted with visual markers. The masked digital images 164 can be shown to human technicians assembling the aircraft structure 104 to quickly identify differences 150 and take remedial actions to correct them.

[0103] Figure 5 It shows Figure 2An example masked digital image 164 of the ROI of the stitched digital image 122 of the aircraft wing 202 shown is provided, in which differences 150 between multiple actual locations 146 and multiple expected locations 112 of multiple holes, pins, and / or fasteners on the aircraft wing 202 are highlighted with multiple visual markers (e.g., 300.1, 300.2, 300.3, 300.4). In the example shown, the masked digital image 164 includes actual holes, pins, and / or fasteners identified by the object detection ML model 138. In some examples, the actual holes, pins, and / or fasteners may be color-coded by type in the masked digital image 164 (e.g., holes are shown in red, pins in orange, and fasteners in green). The masked digital image 164 includes multiple differences 150 (e.g., 150.1, 150.2, 150.3, 150.4) identified by the object detection ML model 138. As an example, the first difference 150.1 is highlighted by the first visual marker 300.1. The first difference 150.1 is identified as an incorrectly installed fastener (which should be a nail). As another example, the second difference 150.2 is highlighted by the second visual marker 300.2. The second difference 150.2 is identified as a hole that should be drilled according to build plan 110 but is not currently present. As yet another example, the third difference 150.3 is highlighted by the third visual marker 300.3. The third difference 150.3 is identified as a hole drilled in the wrong location and / or out of sequence as indicated by build plan 110. As yet another example, the fourth difference 150.4 is highlighted by the fourth visual marker 300.4. The fourth difference 150.4 is identified as a defective hole due to being oval instead of round.

[0104] The masked digital image 164 may include any one of the object tracking data 142 identified by the object detection ML model 138 based at least on a plurality of ROIs 136 processed from the stitched digital image 122.

[0105] Return to Figure 1 The object detection ML model 138 is configured to output the masked digital image 164 to an output device (such as the display subsystem 808). Figure 8 The device 176, shown in the figure, and / or one or more human technicians, allows human technicians involved in assembling the aircraft structure 104 to refer to the masked digital image 164 when resolving discrepancies 150 or other issues identified by the object detection ML model 138.

[0106] The object detection ML model 138 can be implemented using various types of ML models. In one example, a CNN is used to implement the object detection ML model 138. In some examples, the object detection ML model 138 is implemented as a single-stage object detection model (e.g., YOLO, SSD). In other examples, the object detection ML model 138 is implemented as a two-stage object detection model (e.g., RCNN, Faster R-CNN). In other examples, the object detection ML model 138 is implemented using a semantic segmentation-based model that classifies the pixels of an image into object categories (e.g., DeepLab, U-Net). In other examples, the object detection ML model 138 is implemented using an instance segmentation-based model that detects objects and delineates their pixel-level boundaries (e.g., Mask R-CNN). In other examples, different types of machine learning models can be used to implement the object detection ML model 138.

[0107] Processor 106 is configured to execute instructions stored in memory 108 to execute event notification module 166, which is configured to receive a CoA report 140 generated by object detection ML model 138. Event notification module 166 is configured to generate one or more event notifications 168 for aircraft structure 104 based at least on CoA report 140. Event notification 168 indicates one or more remedial actions 170 to be taken based at least on discrepancies 150 listed in the CoA report. In some examples, remedial action 170 is performed to correct discrepancies 150. In one example where the discrepancy includes identifying the absence of a desired hole at a desired location on the aircraft structure, remedial action 170 includes sending a command to drill a hole at the desired location on the aircraft structure. In other examples, other remedial actions 170 may be performed to correct other types of discrepancies identified in CoA report 140, such as sending a command to replace an incorrect nail / fastener with a correct nail / fastener or sending a command to fill a hole that has been incorrectly drilled in aircraft structure 104.

[0108] In an embodiment where the object detection ML model 138 is configured to identify a defect 156 on the aircraft structure 104, remedial action 170 includes sending a command to repair the defect 156 in a hole, pin, and / or fastener at the identified location on the aircraft structure 104. For example, such remedial action 170 to repair defect 156 may include reshaping a hole, filling an improperly drilled hole, or replacing a broken pin / fastener with a working pin / fastener.

[0109] In an embodiment where the object detection ML model 138 is configured to identify foreign object debris 160 on the aircraft structure 104, remedial action 170 includes sending a command to remove the foreign object debris 160 from the identified location 158.

[0110] In some examples, remedial action 170 is performed to prevent further variations downstream of the assembly process due to the difference 150 identified in the CoA report 140. In one example where the CoA report 140 identifies a difference 150 indicating a missing expected hole, remedial action 170 includes sending a command to stop the filling operation performed at the expected location of the hole until the hole can be drilled at the expected location. In other examples, other remedial actions may be performed to prevent additional downstream variations.

[0111] In some embodiments that utilize timestamp 162 to generate the CoA report 140, the event notification module 166 is configured to compare the CoA report 140 of the aircraft structure 104 with a previous CoA report (not shown) of the aircraft structure 104 having an earlier timestamp to verify that previous discrepancies identified in the previous CoA report since its generation have been corrected. The event notification module 166 is configured to generate one or more additional event notifications indicating one or more additional remedial actions for correcting previous discrepancies based on the identification that previous discrepancies have not yet been corrected. The event notification module 166 is configured to generate an event notification based on the identification that previous discrepancies have been corrected in the current CoA report 140, the event notification including verification that the previous discrepancies have been corrected. This feature provides the ability to track the current “as-built” status of the aircraft structure 104 throughout the assembly process and provides verification feedback as data retention to mitigate quality deviations due to discrepancies that may occur during the assembly process.

[0112] Event notification module 166 is configured to output event notification 168 to a different source. In some embodiments, event notification module 166 is configured to output event notification 168 to a data warehouse 172 configured to store event notification 168. In such an embodiment, object detection ML model 138 is also configured to output CoA report 140 to data warehouse 172. Data warehouse 172 is configured to store CoA report 140 (and other CoA reports of aircraft structure 104 generated throughout the assembly process) and event notification 168 of aircraft structure 104 in memory (e.g., in a database). The storage of CoA reports and event notifications in data warehouse 172 allows the information to be referenced as part of feedback / verification features to ensure that identified discrepancies and / or defects are corrected, and / or foreign object debris is removed from the aircraft structure. Furthermore, the storage of CoA reports and event notifications in data warehouse 172 allows the information to be referenced for auditing and improving future assembly processes used to assemble the aircraft structure.

[0113] In some embodiments, the event notification module 166 is configured to output event notification 168 to one or more assembly robots 174. The assembly robot 174 is configured to perform computer-automated operations related to the assembly of the aircraft structure 104, such as drilling and filling holes, correcting defects, and removing foreign debris. The assembly robot 174 is configured to perform remedial actions 170 specified in the event notification 168. In this way, discrepancies 150 identified by the object detection ML model 138 can be automatically corrected by the assembly robot 174 with minimal or no human intervention.

[0114] In some embodiments, the assembly robot 174 is configured to send an acknowledgment notification to the computing system 102 at least based on a remedial action 170 specified in a completion event notification 168. The acknowledgment notification may be sent to the event notification module 166 as part of a process to track whether discrepancies have been corrected.

[0115] In some embodiments, the event notification module 166 is configured to output an event notification 168 to one or more human technician devices 176 (e.g., handheld computing devices, smartphones, tablet computers). The human technician device 176 may present the event notification 168, including a remedial action 170, to the associated human technician, enabling the human technician to manually perform the remedial action 170 in scenarios where the assembly robot 174 cannot perform the remedial action 170 in a computer-automated manner.

[0116] Figures 6 to 7 An example computer-implemented method 600 for inspecting the assembly condition of an aircraft structure according to an embodiment of the present disclosure is shown. For example, method 600 can be implemented by… Figure 1 The computing system 102 shown or another suitable computing system shall be used to perform the operation.

[0117] At 602, method 600 includes receiving a construction plan for an aircraft structure, the construction plan specifying multiple expected locations of a plurality of holes, nails, and / or fasteners on the aircraft structure. At 604, method 600 includes receiving multiple digital images of the aircraft structure from one or more cameras. In some embodiments, at 606, one or more cameras may be configured to project a plurality of landmark features onto the aircraft structure and may capture the plurality of landmark features in the multiple digital images of the aircraft structure. At 608, method 600 includes generating a stitched digital image of the aircraft structure via a stitching digital image generation module, based at least on the plurality of digital images and the construction plan. In some embodiments, at 610, the stitched digital image may be generated at least based on aligning landmark features between adjacent digital images. At 612, method 600 includes determining a plurality of regions of interest (ROIs) of the stitched digital image. At 614, method 600 includes: for each of a plurality of ROIs, inputting the corresponding ROI into an object detection machine learning (ML) model to identify multiple actual locations of multiple actual holes, nails, and / or fasteners on the aircraft structure within the corresponding ROI. In some embodiments, at 616, method 600 may include: for each of the plurality of ROIs, identifying the location of foreign object debris on the aircraft structure within the corresponding ROI. In some embodiments, at 618, method 600 may include: for each of the plurality of ROIs, identifying the location of defects in multiple holes, nails, and / or fasteners within the corresponding ROI. In some embodiments, at 620, method 600 may include: generating one or more masked digital images of the ROIs of the aircraft structure, wherein differences are highlighted with visual markers.

[0118] exist Figure 7In some embodiments, at 622, method 600 may include: outputting one or more masked digital images. At 624, method 600 includes: generating an assembly status (CoA) report for the aircraft structure, the CoA report including differences between a plurality of actual locations and a plurality of expected locations. In some embodiments, at 626, the CoA report further includes the identified locations of foreign object debris on the aircraft structure. In some embodiments, at 628, the CoA report further includes the identified locations of defects in holes, nails, and / or fasteners on the aircraft structure. At 630, method 600 includes: generating one or more event notifications for the aircraft structure based at least on the assembly status report, wherein the one or more event notifications indicate one or more remedial actions to be taken at least based on the differences. In some embodiments, at 632, one or more remedial actions include sending one or more commands for removing foreign object debris from the identified locations on the aircraft structure. In some embodiments, at 634, one or more remedial actions include sending one or more commands for repairing defects in holes, nails, and / or fasteners at the identified locations on the aircraft structure. At 636, method 600 includes outputting one or more event notifications. In some embodiments, at 638, method 600 may include comparing a CoA report of the aircraft structure with a previous CoA report of the aircraft structure to verify that previous discrepancies have been corrected since the previous CoA report was generated. In some embodiments, at 640, method 600 may include generating one or more additional event notifications indicating one or more additional remedial actions for correcting the previous discrepancies, based on the identification that previous discrepancies have not yet been corrected.

[0119] Method 600 can be performed to provide a computer-automated, vision-based method for inspecting aircraft structures in a simple, fast, reliable, and non-invasive manner. Furthermore, method 600 utilizes an ML model specifically trained to identify, count, and compare holes, nails, and / or fasteners on the aircraft structure against the aircraft structure's construction plan, thereby automatically generating a CoA report that is faster, more accurate, and more robust than traditional manual inspection and recording methods. Additionally, the output includes event notifications of remedial actions that can be executed to correct discrepancies identified in the CoA report. In some embodiments, such remedial actions can be performed automatically to minimize human interaction throughout the process.

[0120] In some embodiments, the methods and processes described herein may be associated with a computing system comprising one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0121] Figure 8 A non-limiting embodiment of a computing system 800 that can implement one or more of the methods and processes described above is schematically illustrated. The computing system 800 is shown in a simplified form. The computing system 800 can embody the above description and... Figure 1 The diagram shows a computing system 102, a camera 118, a data warehouse 172, an assembly robot 174, and a human technician device 176. The computing system 800 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), edge devices and / or other computing devices configured to run real-time analytics, and wearable computing devices such as smartwatches and head-mounted augmented reality devices.

[0122] The computing system 800 includes a logic processor 802, volatile memory 804, and non-volatile storage device 806. The computing system 800 may optionally include a display subsystem 808, an input subsystem 810, a communication subsystem 812, and / or... Figure 8 Other components not shown.

[0123] The logic processor 802 includes one or more physical devices configured to execute instructions. For example, the logic processor may be configured to execute instructions as part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, change the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0124] A logic processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, a logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of logic processor 802 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Optionally, the various components of the logic processor may be distributed across two or more separate devices, which may be remotely located and / or configured for coordinated processing. Aspects of the logic processor may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. It should be understood that, in this case, these virtualized aspects run on different physical logic processors on various different machines.

[0125] The non-volatile storage device 806 includes one or more physical devices configured to hold instructions executable by a logic processor to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 806 can be changed, for example, to maintain different data.

[0126] Non-volatile storage device 806 may include removable and / or built-in physical devices. Non-volatile storage device 806 may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or magnetic storage (e.g., hard disk drive, floppy disk drive, magnetic tape drive, MRAM, etc.) or other high-capacity storage technologies. Non-volatile storage device 806 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that non-volatile storage device 806 is configured to retain instructions even when power to non-volatile storage device 806 is cut off.

[0127] Volatile memory 804 may include a physical device, including random access memory. Volatile memory 804 is typically used by logic processor 802 to temporarily store information during the processing of software instructions. It should be understood that when power to volatile memory 804 is cut off, volatile memory 804 typically does not continue storing instructions.

[0128] The logic processor 802, volatile memory 804, and non-volatile storage device 806 can be integrated together into one or more hardware logic components. For example, such hardware logic components may include field-programmable gate arrays (FPGAs), programmable and application-specific integrated circuits (PASIC / ASIC), programmable and application-specific standard products (PSSP / ASSP), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs).

[0129] The terms "module," "program," and "engine" can be used to describe aspects of a computing system 800 typically implemented in software by a processor to perform specific functions using portions of volatile memory, involving transformation processing specifically configured for that function. Thus, a module, program, or engine can be instantiated via logic processor 802 using portions of volatile memory 804 to execute instructions held by non-volatile storage device 806. It should be understood that different modules, programs, and / or engines can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass individual or grouped executable files, data files, libraries, drivers, scripts, database records, etc.

[0130] When included, the display subsystem 808 can be used to present a visual representation of the data held by the non-volatile storage device 806. The visual representation may take the form of a graphical user interface (GUI). When the methods and processes described herein change the data held by the non-volatile storage device and thus change the state of the non-volatile storage device, the state of the display subsystem 808 can also be changed to visually represent the changes in the underlying data. The display subsystem 808 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with a logic processor 802, volatile memory 804, and / or non-volatile storage device 806 in a shared housing, or such display devices may be peripheral display devices.

[0131] When the input subsystem 810 is included, it may include or interface with one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may include or interface with a selected Natural User Input (NUI) component. Such components may be integrated or peripheral, and the translation and / or processing of input actions may be handled on-board or off-board. Example NUI components may include a microphone for voice and / or speech recognition; an infrared, color, stereo, and / or depth camera for machine vision and / or gesture recognition; a head tracker, eye tracker, accelerometer, and / or gyroscope for motion detection and / or intent recognition; and / or another suitable sensor.

[0132] When a communication subsystem 812 is included, the communication subsystem 812 can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 812 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network (such as an HDMI connection via Wi-Fi). In some embodiments, the communication subsystem may allow the computing system 800 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0133] In addition, this disclosure includes configurations based on the following examples.

[0134] In one example, a computing system includes one or more processors configured to execute instructions stored in memory to receive a construction plan for an aircraft structure, the construction plan specifying multiple expected locations of a plurality of holes, nails, and / or fasteners on the aircraft structure; receive multiple digital images of the aircraft structure from one or more cameras; generate a stitched digital image of the aircraft structure via a stitching image generation module, based at least on the plurality of digital images and the construction plan; determine multiple regions of interest (ROIs) of the stitched digital image for each of the plurality of ROIs; input the corresponding ROIs into an object detection machine learning (ML) model to identify multiple actual locations of a plurality of actual holes, nails, and / or fasteners on the aircraft structure within the corresponding ROIs; generate an assembly status report of the aircraft structure, the assembly status report including differences between the plurality of actual locations and the plurality of expected locations; generate one or more event notifications for the aircraft structure, based at least on the assembly status report, wherein the one or more event notifications indicate one or more remedial actions to be taken based at least on the differences; and output the one or more event notifications. In this example and / or other examples, one or more cameras can be configured to project multiple landmark features onto the aircraft structure, capture multiple landmark features in multiple digital images of the aircraft structure, and the stitched digital image generation module can be configured to generate a stitched digital image of the aircraft structure based at least on aligning multiple landmark features between adjacent digital images in the multiple digital images. In this example and / or other examples, the stitched digital image generation module can be configured to execute a feature extraction ML model configured to extract aircraft-specific features from multiple digital images. The feature extraction ML model can be trained at least on a training digital image set of multiple different aircraft structures labeled with aircraft-specific features and hole-based layout features, and the stitched digital image generation module can be configured to generate a stitched digital image based at least on the aircraft-specific features output by the feature extraction ML model. In this example and / or other examples, the object detection ML model may be further configured to, for each of a plurality of ROIs, identify the location of foreign object debris on the aircraft structure in the corresponding ROI, the assembly status report may further include the identified location of the foreign object debris on the aircraft structure, and one or more remedial actions may include sending one or more commands for removing the foreign object debris from the identified location.In this example and / or other examples, the object detection ML model may be further configured to, for each of a plurality of ROIs, identify the locations of defects in a plurality of holes, nails, and / or fasteners in the corresponding ROI. The assembly status report may further include the identified locations of the defects in the holes, nails, and / or fasteners on the aircraft structure, and one or more remedial actions may include sending one or more commands to repair the defects in the holes, nails, and / or fasteners at the identified locations on the aircraft structure. In this example and / or other examples, the object detection ML model may be further configured to generate one or more masked digital images of the ROIs of the aircraft structure, wherein differences are highlighted with visual markers, and one or more processors may be configured to execute instructions stored in memory to output one or more masked digital images. In this example and / or other examples, differences may include identifying the absence of holes at expected locations on the aircraft structure, and one or more remedial actions may include sending commands to drill holes at expected locations on the aircraft structure. In this example and / or other examples, one or more remedial actions may include sending a command to stop the filling operation performed at the intended location of the hole until the hole can be drilled at the intended location. In this example and / or other examples, one or more processors may be configured to execute instructions stored in memory to compare an assembly status report of the aircraft structure with a previous assembly status report of the aircraft structure to verify that previous discrepancies have been corrected since the previous assembly status report was generated, and, based on the identification that previous discrepancies have not been corrected, to generate one or more additional event notifications indicating one or more additional remedial actions for correcting previous discrepancies.

[0135] In another example, a computer-implemented method includes the steps of: receiving a construction plan for an aircraft structure, the construction plan specifying multiple expected locations of multiple holes, nails, and / or fasteners on the aircraft structure; receiving multiple digital images of the aircraft structure from one or more cameras; generating a stitched digital image of the aircraft structure via a stitched digital image generation module, based at least on the multiple digital images and the construction plan; determining multiple regions of interest (ROIs) of the stitched digital image for each of multiple ROIs; inputting the corresponding ROIs into an object detection machine learning (ML) model to identify multiple actual locations of multiple actual holes, nails, and / or fasteners on the aircraft structure within the corresponding ROIs; generating an assembly status report of the aircraft structure, the assembly status report including differences between the multiple actual locations and the multiple expected locations; generating one or more event notifications for the aircraft structure, based at least on the assembly status report, wherein the one or more event notifications indicate one or more remedial actions to be taken based at least on the differences; and outputting the one or more event notifications. In this example and / or other examples, one or more cameras can be configured to project multiple landmark features onto the aircraft structure, and multiple landmark features can be captured in multiple digital images of the aircraft structure. The stitching digital image generation module can be configured to generate a stitched digital image of the aircraft structure based at least on aligning multiple landmark features between adjacent digital images in the multiple digital images. In this example and / or other examples, the stitching digital image generation module can be configured to execute a feature extraction ML model configured to extract aircraft-specific features from multiple digital images. The feature extraction ML model can be trained at least on a training digital image set of multiple different aircraft structures labeled with aircraft-specific features and hole-based layout features. The stitching digital image generation module can be configured to generate a stitched digital image based at least on the aircraft-specific features output by the feature extraction ML model. In this example and / or other examples, the object detection ML model may be further configured to, for each of a plurality of ROIs, identify the location of foreign object debris on the aircraft structure in the corresponding ROI, the assembly status report may further include the location of the identified foreign object debris on the aircraft structure, and one or more remedial actions may include sending one or more commands for removing the foreign object debris from the identified location.In this example and / or other examples, the object detection ML model may be further configured to, for each of a plurality of ROIs, identify the location of defects in a plurality of holes, nails, and / or fasteners in the corresponding ROI. The assembly status report may further include the identified locations of the defects in the holes, nails, and / or fasteners on the aircraft structure, and one or more remedial actions may include sending one or more commands to repair the defects in the holes, nails, and / or fasteners at the identified locations on the aircraft structure. In this example and / or other examples, the object detection ML model may be further configured to generate one or more masked digital images of the ROIs of the aircraft structure, wherein differences are highlighted with visual markers, and the computer-implemented method may further include outputting one or more masked digital images. In this example and / or other examples, differences may include identifying the absence of a hole at a desired location on the aircraft structure, and one or more remedial actions may include sending a command to drill a hole at the desired location on the aircraft structure. In this example and / or other examples, one or more remedial actions may include sending a command to stop the filling operation performed at the desired location of the hole until the hole can be drilled at the desired location. In this example and / or other examples, the computer-implemented method may further include comparing an assembly status report of the aircraft structure with a previous assembly status report of the aircraft structure to verify that previous discrepancies have been corrected since the previous assembly status report was generated, and, based on the identification that previous discrepancies have not been corrected, generating one or more additional event notifications indicating one or more additional remedial actions for correcting previous discrepancies.

[0136] In another example, a computing system includes one or more processors configured to execute instructions stored in memory to receive a construction plan for an aircraft structure, the construction plan specifying multiple expected locations of a plurality of holes, nails, and / or fasteners on the aircraft structure; receive multiple digital images of the aircraft structure from one or more cameras; generate a stitched digital image of the aircraft structure via a stitched digital image generation module, at least based on the multiple digital images and the construction plan; input the stitched digital image into an object detection machine learning (ML) model to identify multiple actual locations of a plurality of actual holes, nails, and / or fasteners on the aircraft structure in the stitched digital image; generate an assembly status report of the aircraft structure, the assembly status report including differences between the plurality of actual locations and the plurality of expected locations; generate one or more event notifications for the aircraft structure, at least based on the assembly status report; wherein the one or more event notifications indicate one or more remedial actions to be taken, at least based on the differences; and output the one or more event notifications. In this example and / or other examples, one or more processors may be configured to execute instructions stored in memory to determine multiple regions of interest (ROIs) of a stitched digital image, and for each of the multiple ROIs, input the corresponding ROI into an object detection ML model to identify multiple actual locations of multiple actual holes, nails and / or fasteners on the aircraft structure within the corresponding ROI. The assembly status report for the aircraft structure may include the differences between multiple actual locations and multiple expected locations within the multiple ROIs.

[0137] This application involves the following provisions:

[0138] 1. A computing system, the computing system comprising:

[0139] One or more processors, said one or more processors being configured to execute instructions stored in memory to:

[0140] Receive a construction plan for an aircraft structure, the construction plan specifying multiple expected locations of a plurality of holes, pins and / or fasteners on the aircraft structure;

[0141] Receive multiple digital images of the aircraft structure from one or more cameras;

[0142] Based at least on the plurality of digital images and the construction plan, a stitched digital image of the aircraft structure is generated via a stitched image generation module;

[0143] Determine multiple regions of interest in the stitched digital image;

[0144] For each of the plurality of regions of interest, the corresponding region of interest is input into the object detection machine learning model to identify the multiple actual locations of multiple actual holes, nails and / or fasteners on the aircraft structure in the corresponding region of interest;

[0145] Generate an assembly status report for the aircraft structure, the assembly status report including the differences between the plurality of actual positions and the plurality of expected positions;

[0146] At least based on the assembly status report, one or more event notifications for the aircraft structure are generated, wherein the one or more event notifications indicate one or more remedial actions to be taken based at least on the discrepancies; and

[0147] Output one or more event notifications.

[0148] 2. The computing system according to Clause 1, wherein the one or more cameras are configured to project a plurality of landmark features onto the aircraft structure, wherein the plurality of landmark features are captured in the plurality of digital images of the aircraft structure, and wherein the stitched digital image generation module is configured to generate the stitched digital image of the aircraft structure based at least on aligning the plurality of landmark features between adjacent digital images in the plurality of digital images.

[0149] 3. The computing system according to Clause 1, wherein the stitched digital image generation module is configured to execute a feature extraction machine learning model, the feature extraction machine learning model being configured to extract aircraft-specific features from the plurality of digital images, wherein the feature extraction machine learning model is trained on at least a training digital image set of multiple different aircraft structures labeled with aircraft-specific features and hole-based layout features, and wherein the stitched digital image generation module is configured to generate the stitched digital image based at least on the aircraft-specific features output by the feature extraction machine learning model.

[0150] 4. The computing system according to Clause 1, wherein the object detection machine learning model is further configured to, for each of the plurality of regions of interest, identify the location of a foreign object debris on the aircraft structure in the corresponding region of interest, wherein the assembly status report further includes the identified location of the foreign object debris on the aircraft structure, and wherein the one or more remedial actions include sending one or more commands for removing the foreign object debris from the identified location.

[0151] 5. The computing system according to Clause 1, wherein the object detection machine learning model is further configured to, for each of the plurality of regions of interest, identify the location of a defect in the plurality of holes, nails, and / or fasteners in the corresponding region of interest, wherein the assembly status report further includes the location of the identified defects in the plurality of holes, nails, and / or fasteners on the aircraft structure, and wherein the one or more remedial actions include sending one or more commands for repairing the defects in the plurality of holes, nails, and / or fasteners at the identified locations on the aircraft structure.

[0152] 6. The computing system according to Clause 1, wherein the object detection machine learning model is further configured to generate one or more masked digital images of the region of interest of the aircraft structure, wherein the differences in the one or more masked digital images are highlighted with visual markers, and wherein the one or more processors are configured to execute instructions stored in memory to:

[0153] Output one or more masked digital images.

[0154] 7. The computing system according to Clause 1, wherein the difference includes identifying that no hole exists at a desired location on the aircraft structure, and wherein the one or more remedial actions include sending a command to drill a hole at the desired location on the aircraft structure.

[0155] 8. The computing system according to Clause 7, wherein the one or more remedial actions include sending a command to stop the filling operation performed at the intended location of the hole until the hole can be drilled at the intended location.

[0156] 9. The computing system according to Clause 1, wherein the one or more processors are configured to execute instructions stored in memory to:

[0157] The assembly status report of the aircraft structure is compared with a previous assembly status report of the aircraft structure to verify that previous discrepancies have been corrected since the previous assembly status report was generated; and

[0158] Based on the identification that a previous difference has not yet been corrected, one or more additional event notifications are generated that indicate one or more additional remedial actions to correct the previous difference.

[0159] 10. A computer-implemented method, the method comprising the following steps:

[0160] Receive a construction plan for an aircraft structure, the construction plan specifying multiple expected locations of a plurality of holes, pins and / or fasteners on the aircraft structure;

[0161] Receive multiple digital images of the aircraft structure from one or more cameras;

[0162] At least based on the plurality of digital images and the construction plan, a stitched digital image of the aircraft structure is generated via a stitched digital image generation module;

[0163] Determine multiple regions of interest in the stitched digital image;

[0164] For each of the plurality of regions of interest, the corresponding region of interest is input into the object detection machine learning model to identify the multiple actual locations of multiple actual holes, nails and / or fasteners on the aircraft structure in the corresponding region of interest;

[0165] Generate an assembly status report for the aircraft structure, the assembly status report including the differences between the plurality of actual positions and the plurality of expected positions;

[0166] At least based on the assembly status report, one or more event notifications for the aircraft structure are generated, wherein the one or more event notifications indicate one or more remedial actions to be taken based at least on the discrepancies; and

[0167] Output one or more event notifications.

[0168] 11. The computer-implemented method according to Clause 10, wherein the one or more cameras are configured to project a plurality of landmark features onto the aircraft structure, wherein the plurality of landmark features are captured in the plurality of digital images of the aircraft structure, and wherein the stitched digital image generation module is configured to generate the stitched digital image of the aircraft structure based at least on aligning the plurality of landmark features between adjacent digital images in the plurality of digital images.

[0169] 12. The computer-implemented method according to Clause 10, wherein the stitched digital image generation module is configured to execute a feature extraction machine learning model, the feature extraction machine learning model being configured to extract aircraft-specific features from the plurality of digital images, wherein the feature extraction machine learning model is trained on at least a training digital image set of multiple different aircraft structures labeled with aircraft-specific features and hole-based layout features, and wherein the stitched digital image generation module is configured to generate the stitched digital image based at least on the aircraft-specific features output by the feature extraction machine learning model.

[0170] 13. The computer-implemented method according to Clause 10, wherein the object detection machine learning model is further configured to, for each of the plurality of regions of interest, identify the location of a foreign object debris on the aircraft structure in the corresponding region of interest, wherein the assembly status report further includes the identified location of the foreign object debris on the aircraft structure, and wherein the one or more remedial actions include sending one or more commands for removing the foreign object debris from the identified location.

[0171] 14. The computer-implemented method according to Clause 10, wherein the object detection machine learning model is further configured to, for each of the plurality of regions of interest, identify the location of a defect in the plurality of holes, nails, and / or fasteners in the corresponding region of interest, wherein the assembly status report further includes the location of the identified defects in the plurality of holes, nails, and / or fasteners on the aircraft structure, and wherein the one or more remedial actions include sending one or more commands for repairing the defects in the plurality of holes, nails, and / or fasteners at the identified locations on the aircraft structure.

[0172] 15. The computer-implemented method according to Clause 10, wherein the object detection machine learning model is further configured to generate one or more masked digital images of the region of interest of the aircraft structure, wherein the differences in the one or more masked digital images are highlighted with visual markers, and wherein the computer-implemented method further comprises the step of: outputting the one or more masked digital images.

[0173] 16. The computer-implemented method according to Clause 10, wherein the difference includes identifying that no hole exists at a desired location on the aircraft structure, and wherein the one or more remedial actions include sending a command to drill a hole at the desired location on the aircraft structure.

[0174] 17. The computer-implemented method according to Clause 16, wherein the one or more remedial actions include sending a command to stop a filling operation performed at the intended location of the hole until the hole can be drilled at the intended location.

[0175] 18. The computer-implemented method according to Clause 10, the method further comprising the following steps:

[0176] The assembly status report of the aircraft structure is compared with a previous assembly status report of the aircraft structure to verify that previous discrepancies have been corrected since the previous assembly status report was generated; and

[0177] Based on the identification that a previous difference has not yet been corrected, one or more additional event notifications are generated that indicate one or more additional remedial actions to correct the previous difference.

[0178] 19. A computing system, the computing system comprising:

[0179] One or more processors, said one or more processors being configured to execute instructions stored in memory to:

[0180] Receive a construction plan for an aircraft structure, the construction plan specifying multiple expected locations of a plurality of holes, pins and / or fasteners on the aircraft structure;

[0181] Receive multiple digital images of the aircraft structure from one or more cameras;

[0182] At least based on the plurality of digital images and the construction plan, a stitched digital image of the aircraft structure is generated via a stitched digital image generation module;

[0183] The stitched digital image is input into an object detection machine learning model to identify multiple actual locations of multiple actual holes, nails and / or fasteners on the aircraft structure in the stitched digital image.

[0184] Generate an assembly status report for the aircraft structure, the assembly status report including the differences between the plurality of actual positions and the plurality of expected positions;

[0185] At least based on the assembly status report, one or more event notifications for the aircraft structure are generated, wherein the one or more event notifications indicate one or more remedial actions to be taken based at least on the discrepancies; and

[0186] Output one or more event notifications.

[0187] 20. The computing system according to Clause 19, wherein the one or more processors are configured to execute instructions stored in memory to:

[0188] Determine multiple regions of interest in the stitched digital image; and

[0189] For each of the plurality of regions of interest, the corresponding region of interest is input into the object detection machine learning model to identify multiple actual locations of multiple actual holes, nails and / or fasteners on the aircraft structure in the corresponding region of interest, wherein the assembly status report of the aircraft structure includes the differences between the multiple actual locations and the multiple expected locations in the plurality of regions of interest.

[0190] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples are not intended to be limiting, as many variations are possible. The particular routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described may be performed in the order shown and / or described, in another order, in parallel, or omitted. Similarly, the order of the above processing may be changed.

[0191] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations and other features, functions, actions and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computing system, the computing system comprising: One or more processors, said one or more processors being configured to execute instructions stored in memory to: Receive a construction plan for an aircraft structure, the construction plan specifying multiple expected locations of a plurality of holes, pins and / or fasteners on the aircraft structure; Receive multiple digital images of the aircraft structure from one or more cameras; Based at least on the plurality of digital images and the construction plan, a stitched digital image of the aircraft structure is generated via a stitched image generation module; Determine multiple regions of interest in the stitched digital image; For each of the plurality of regions of interest, the corresponding region of interest is input into the object detection machine learning model to identify the multiple actual locations of multiple actual holes, nails and / or fasteners on the aircraft structure in the corresponding region of interest; Generate an assembly status report for the aircraft structure, the assembly status report including the differences between the plurality of actual positions and the plurality of expected positions; At least based on the assembly status report, one or more event notifications for the aircraft structure are generated, wherein the one or more event notifications indicate one or more remedial actions to be taken based at least on the discrepancies; and Output one or more event notifications.

2. The computing system according to claim 1, wherein, The one or more cameras are configured to project a plurality of landmark features onto the aircraft structure, wherein the plurality of landmark features are captured in the plurality of digital images of the aircraft structure, and wherein the stitched digital image generation module is configured to generate the stitched digital image of the aircraft structure based at least on aligning the plurality of landmark features between adjacent digital images in the plurality of digital images.

3. The computing system according to claim 1, wherein, The stitched digital image generation module is configured to execute a feature extraction machine learning model, which is configured to extract aircraft-specific features from the plurality of digital images. The feature extraction machine learning model is trained on a training digital image set of multiple different aircraft structures labeled with aircraft-specific features and hole-based layout features. The stitched digital image generation module is configured to generate the stitched digital image based at least on the aircraft-specific features output by the feature extraction machine learning model.

4. The computing system according to claim 1, wherein, The object detection machine learning model is also configured to, for each of the plurality of regions of interest, identify the location of a foreign object debris on the aircraft structure within the corresponding region of interest, wherein the assembly status report also includes the identified location of the foreign object debris on the aircraft structure, and wherein the one or more remedial actions include sending one or more commands to remove the foreign object debris from the identified location.

5. The computing system according to claim 1, wherein, The object detection machine learning model is also configured to, for each of the plurality of regions of interest, identify the location of defects in the plurality of holes, nails, and / or fasteners in the corresponding region of interest, wherein the assembly status report also includes the location of the identified defects in the plurality of holes, nails, and / or fasteners on the aircraft structure, and wherein the one or more remedial actions include sending one or more commands for repairing the defects in the plurality of holes, nails, and / or fasteners at the identified locations on the aircraft structure.

6. The computing system according to claim 1, wherein, The object detection machine learning model is also configured to generate one or more masked digital images of the region of interest of the aircraft structure, wherein the differences in the one or more masked digital images are highlighted with visual markers, and wherein the one or more processors are configured to execute instructions stored in memory to: Output one or more masked digital images.

7. The computing system according to claim 1, wherein, The difference includes identifying the absence of a hole at a desired location on the aircraft structure, and wherein the one or more remedial actions include sending a command to drill a hole at the desired location on the aircraft structure.

8. The computing system according to claim 7, wherein, The one or more remedial actions include sending a command to stop the filling operation performed at the intended location of the hole until the hole can be drilled at the intended location.

9. A computer-implemented method, the method comprising the following steps: Receive a construction plan for an aircraft structure, the construction plan specifying multiple expected locations of a plurality of holes, pins and / or fasteners on the aircraft structure; Receive multiple digital images of the aircraft structure from one or more cameras; At least based on the plurality of digital images and the construction plan, a stitched digital image of the aircraft structure is generated via a stitched digital image generation module; Determine multiple regions of interest in the stitched digital image; For each of the plurality of regions of interest, the corresponding region of interest is input into the object detection machine learning model to identify the multiple actual locations of multiple actual holes, nails and / or fasteners on the aircraft structure in the corresponding region of interest; Generate an assembly status report for the aircraft structure, the assembly status report including the differences between the plurality of actual positions and the plurality of expected positions; At least based on the assembly status report, one or more event notifications for the aircraft structure are generated, wherein the one or more event notifications indicate one or more remedial actions to be taken at least based on the discrepancies; as well as Output one or more event notifications.

10. A computing system, the computing system comprising: One or more processors, said one or more processors being configured to execute instructions stored in memory to: Receive a construction plan for an aircraft structure, the construction plan specifying multiple expected locations of a plurality of holes, pins and / or fasteners on the aircraft structure; Receive multiple digital images of the aircraft structure from one or more cameras; At least based on the plurality of digital images and the construction plan, a stitched digital image of the aircraft structure is generated via a stitched digital image generation module; The stitched digital image is input into an object detection machine learning model to identify multiple actual locations of multiple actual holes, nails and / or fasteners on the aircraft structure in the stitched digital image. Generate an assembly status report for the aircraft structure, the assembly status report including the differences between the plurality of actual positions and the plurality of expected positions; At least based on the assembly status report, one or more event notifications for the aircraft structure are generated, wherein the one or more event notifications indicate one or more remedial actions to be taken at least based on the discrepancies; as well as Output one or more event notifications.

Images