Automated paint quality inspection for aircraft

Abstract

Automated paint quality inspection for aircraft. According to various embodiments, an automated aircraft paint application quality inspection technique is presented. The technique includes: retrieving from an electronic persistent storage device a control image depicting at least a portion of an aircraft paint application on an aircraft under quality inspection; capturing a process image depicting at least a portion of an aircraft paint application on the aircraft under quality inspection; detecting a plurality of features in the control image and the process image; registering the control image with the process image; detecting at least one difference between the control image and the process image; generating an output image, wherein the output image includes a depiction of the aircraft under quality inspection wherein the at least one difference is annotated; and causing the output image to be displayed.

Description

Technical Field

[0001] The present disclosure relates generally to aircraft coating applications and associated quality control. Background Art

[0002] Existing visual inspections of paintwork in final assembly of aircraft rely primarily on human vision and can therefore be subjective. In addition, the paint quality inspection process for aircraft can be subject to human interpretation. Since for many aircraft, decorative painting is at a critical point in the delivery process, the ability to identify and resolve paint quality issues in a timely manner can minimize possible delays due to rework. In addition, a consistent quality inspection process is needed that can be implemented not only at the manufacturer's facility, but also at an offload paint application site where expertise about possible problems and inspection standards may be less than at the manufacturer's site. Summary of the invention

[0003] According to various embodiments, an automated aircraft paint application quality inspection method is provided. The method includes the steps of retrieving a reference image depicting at least a portion of an aircraft paint application on an aircraft under quality inspection from an electronic persistent storage device; capturing a process image depicting at least a portion of an aircraft paint application on the aircraft under quality inspection; detecting a plurality of features in the reference image and the process image; registering the reference image with the process image; detecting at least one difference between the reference image and the process image; generating an output image, wherein the output image includes a depiction of the aircraft under quality inspection wherein the at least one difference is annotated; and causing the output image to be displayed.

[0004] Various optional features of the above embodiments include the following features. The method may include the steps of detecting at least one reflection and not considering the at least one reflection as a difference between the control image and the process image. The step of not considering the at least one reflection as a difference between the control image and the process image may include the steps of capturing at least one image of the aircraft under quality inspection from a first perspective and capturing at least one image of the aircraft under quality inspection from a second perspective. The step of not considering the at least one reflection as a difference between the control image and the process image may include the steps of placing at least one standardized marker on the aircraft under quality inspection before capturing at least one image of the aircraft under quality inspection from the first perspective and capturing at least one image of the aircraft under quality inspection from the second perspective. The method may include the steps of repairing anomalies corresponding to the at least one difference. The step of detecting a plurality of features may include the steps of detecting a plurality of features using a Speeded Up Robust Features (SURF) algorithm. Aligning the control image with the process image may be performed using a Random Sample Consensus (RANSAC) algorithm. The step of generating an output image may include the steps of combining at least a portion of an image captured from a first perspective with at least a portion of an image captured from a second perspective different from the first perspective. The control image may include a combination of a first control image taken from a first viewing angle and a second control image taken from a second viewing angle different from the first viewing angle, and the process image may include a combination of a first process image taken from a third viewing angle and a second process image taken from a fourth viewing angle different from the third viewing angle. The step of detecting at least one difference may include the step of performing a pixel-by-pixel comparison between the control image and the process image.

[0005] According to various embodiments, a system for automated aircraft paint application quality inspection is provided. The system includes: an electronic persistent storage device storing a reference image depicting at least a portion of an aircraft paint application on an aircraft under quality inspection; a plurality of cameras configured to capture a plurality of images of the aircraft under quality inspection; and at least one electronic processor communicatively connected to the electronic persistent storage device and the plurality of cameras, the at least one electronic processor executing instructions for performing operations, the operations including: retrieving the reference image from the electronic persistent storage device; capturing a process image depicting at least a portion of an aircraft paint application on the aircraft under quality inspection through the plurality of cameras; detecting a plurality of features in the reference image and the process image; registering the reference image with the process image; detecting at least one difference between the reference image and the process image; generating an output image, wherein the output image includes a depiction of the aircraft under quality inspection wherein the at least one difference is annotated; and causing the output image to be displayed.

[0006] Various optional features of the above embodiments include the following features. The operation may further include the following operation: detecting at least one reflection and not considering the at least one reflection as a difference between the control image and the process image. The operation of not considering at least one reflection as a difference between the control image and the process image may include the following operation: capturing at least one image of the aircraft under quality inspection from a first perspective by the plurality of cameras, and capturing at least one image of the aircraft under quality inspection from a second perspective by the plurality of cameras. The system may include at least one standardized marker configured to be placed on the aircraft under quality inspection before capturing at least one image of the aircraft under quality inspection from the first perspective and capturing at least one image of the aircraft under quality inspection from the second perspective. The operation may further include the following operation: guiding a user to repair an anomaly corresponding to the at least one difference. The operation of detecting a plurality of features may include the following operation: detecting a plurality of features using a Speeded Up Robust Features (SURF) algorithm. Aligning the control image with the process image may be performed using a Random Sample Consensus (RANSAC) algorithm. The operation of generating an output image may include the following operation: combining at least a portion of an image captured from a first perspective with at least a portion of an image captured from a second perspective different from the first perspective. The control image may include a combination of a first control image taken from a first viewing angle and a second control image taken from a second viewing angle different from the first viewing angle, and the process image may include a combination of a first process image taken from a third viewing angle and a second process image taken from a fourth viewing angle different from the third viewing angle. The operation of detecting at least one difference may include the operation of performing a pixel-by-pixel comparison between the control image and the process image. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The examples are better understood with reference to the following detailed description so that the various features of the examples may be more fully appreciated when considered in conjunction with the accompanying drawings, in which:

[0008] Figure 1 is an image of an aircraft livery on an aircraft according to some embodiments;

[0009] Figure 2 is a schematic diagram of an image capture device according to some embodiments;

[0010] Figure 3 depicts a control image and a process image of an aircraft according to some embodiments;

[0011] Figure 4 depicts feature detection in control and process images according to some embodiments;

[0012] Figure 5 depicts feature matching according to some embodiments;

[0013] Figure 6 depicts a characteristic displacement vector according to some embodiments;

[0014] Figure 7 depicts a graphical user interface according to some embodiments;

[0015] Figure 8 depicts example output images according to some implementations;

[0016] Fig. 9 is a flow chart of a method for inspecting aircraft coating application quality according to some embodiments;

[0017] Fig.10 is a flow chart of a reflection processing method according to some embodiments;

[0018] Fig.11 is a diagram of standardized notations according to some embodiments; and

[0019] Fig.12 is a schematic diagram of a system according to some embodiments. DETAILED DESCRIPTION

[0020] Reference will now be made in detail to the disclosed examples illustrated in the accompanying drawings. Whenever possible, the same reference numerals will be used throughout the accompanying drawings to represent the same or similar parts. In the following description, reference is made to the accompanying drawings that form a part of the present invention and are shown by illustrating specific examples. These examples are described in sufficient detail to enable those skilled in the art to practice them, and it will be understood that other examples may be utilized and may be changed without departing from the scope of the present disclosure. Therefore, the following description is exemplary only.

[0021] In general, some embodiments provide a computerized in-process quality inspection of a structural surface (e.g., an aircraft during final assembly). More specifically, some embodiments provide an automated quality inspection of an aircraft coating application (e.g., painting), wherein automated image processing aligns machine vision features of an image of the aircraft under inspection captured with corresponding features in a control (reference) image and uses the alignment to identify possible defects (e.g., misplaced painting features, missing painting features, discolored images, foreign debris, templates that should be removed, and other paint application expedients, etc.). The embodiment may then display an augmented reality depiction of the aircraft under inspection, in which these possible defects are emphasized or otherwise annotated. Some embodiments include wearable augmented reality head hardware for implementing such a display. The embodiment may take into account reflections from painted or unpainted surfaces, the relative quality of ambient lighting, and the range of possible paint colors. The embodiment may include an electronic persistent repository of control images for feature recognition and comparison.

[0022] Some embodiments use multiple mounted cameras that are moved or positioned at varying angles to allow automatic identification of reflections in captured images. These cameras allow some embodiments to distinguish between reflections and shadows from paint and surface features and ignore the former in quality analysis.

[0023] Example embodiments will now be described with reference to the accompanying drawings.

[0024] Figure 1 1 is an image of an aircraft livery 106 on an aircraft 100 according to some embodiments. Typically, an aircraft may have an image of multiple colors in multiple locations. The depicted aircraft 100 includes a portion of an aircraft livery 102 (a string of stars) and a portion of an aircraft livery 104 (a shadow on the tail). Various images, emblems, and alphanumeric text on a particular aircraft may be referred to herein as the "livery" of the aircraft. Typically, an aircraft livery may represent an emblem of an airline, an aircraft manufacturer, a government license, and various other required, optional, and / or trademarked emblems. For example, one or more stencils may be used to apply an aircraft livery.

[0025] Figure 2 is a schematic diagram of an image capture device 200 according to some embodiments. In general, embodiments may be implemented where an aircraft paint is applied to an aircraft. Thus, embodiments may be implemented at a main plant or paint hangar or where expertise in paint quality may be limited (e.g., a remote or outsourced paint facility). In addition, embodiments may include taking into account reflections, shadows, and other visible phenomena that are not representative of actual physical materials present on the aircraft surface (see Figure 3 ) hardware and processing techniques. These embodiments can exclude these phenomena from the automated processing otherwise applied to paint and other surface features and debris so that they are ignored for the purpose of automated quality inspection.

[0026] The image capture device includes a plurality of cameras 202, 204, 206 positioned to capture images of the aircraft under quality inspection from perspectives 208, 210, and 212, respectively. As disclosed herein, some embodiments capture images of the aircraft from multiple angles and / or positions to enable detection of reflections, shadows, and other transient visual phenomena (see Figure 3). Cameras 202, 204, 206 may be located on rails or other structures that allow them to change position to capture images of substantially the same area of ​​the aircraft from multiple angles. These movable cameras may be moved automatically (e.g., by motors configured to change position along multiple sets of rails). The rails may be vertical and / or horizontal. For embodiments that include movable cameras, camera 202 may be sufficient without cameras 204 and 206 because camera 202 can be moved to capture images from any of perspectives 208, 210, or 212. In addition, for embodiments that include movable cameras, these cameras may have relatively low definition because these movable cameras can be zoomed in so that their field of view is focused on the relevant aircraft portion. For embodiments that include stationary cameras, cameras 202, 204, 206 may have high definition because image portions can be extracted and processed separately. This document (e.g., below with reference to Fig.10 ) presents details of this process and the process used to exclude transient visual phenomena from consideration as possible paint application defects. Six groups of two or more cameras (or six movable cameras) may be used to capture each of the six sections of the aircraft from multiple perspectives. However, it should be noted that the six groups of cameras are exemplary; in other embodiments, the number of cameras may be determined on a case-by-case basis based on the size of both the hangar and the aircraft.

[0027] Figure 3 Depicted is a reference image 302 and a process image 304 of an aircraft according to some embodiments. For example, both reference image 302 and process image 304 may be used Figure 2 The control image 302 and the process image 304 may be captured by the image capture device 200, but for the control image 302 and the process image 304, such capture may occur at different stages of the overall aircraft paint quality inspection process disclosed herein. The control image 302 and other control images are used as reference images for automatic computer vision comparison with newly captured images of the aircraft taken during the quality inspection process (e.g., the process image 304). The control image 302 and similar control images may be stored in a persistent electronic storage device (e.g., as described below). Fig.11 The system 1100 is shown and described in electronic persistent storage 1112 . Note that both the control image 302 and the process image 304 display reflections 306 and shadows 308 .

[0028] Figure 4Feature detection in control image 302 and process image 304 is depicted in accordance with some embodiments. The feature detection used in accordance with some embodiments may be automatic feature detection using computer vision techniques. Suitable techniques include Speeded Up Robust Features (“SURF”), such as disclosed in U.S. Published Application No. 2009 / 0238460. These techniques identify visually distinct features in electronic images, such as feature 406 in control image 402. (Note that a feature corresponding to feature 406 appears in process image 404.) As described herein (e.g., with reference to Fig. 9 and Fig.10 ) disclosed in , these techniques can be applied independently to control images and process images.

[0029] Figure 5 Feature matching according to some embodiments is depicted. Generally, disclosed embodiments may use automated computer vision techniques to match features detected in a control image with features detected in a corresponding process image. Figure 5 A control image 502 and a process image 504 are depicted, along with leads 506 connecting three example features in the control image 502 to corresponding features in the process image 504. In general, features between the control image and the process image may be matched using random sample consensus ("RANSAC") according to various embodiments. RANSAC is particularly well suited for embodiments because it is robust in the presence of data provided by error-prone feature detectors. As described herein, embodiments may use RANSAC to register a control image to a process image. Figure 8 , Fig. 9 , Fig.10 and Fig.11 Further uses of RANSAC according to some embodiments are shown and described.

[0030] Figure 6 Depicted is a feature displacement vector 604 according to some embodiments. In registering the reference image with the process image based on matching detected features, embodiments may utilize RANSAC. RANSAC has a maximum local displacement value as a user-settable parameter. This value determines the maximum local displacement between features in the reference image and the process image for a feature to be considered corresponding. Beyond the maximum local displacement value, RANSAC determines the feature as not matching. Figure 6 As depicted in , by applying RANSAC, it can be determined that features in the combined image correspond because, although they do not exactly coincide, they are within a maximum local displacement value set by the user. Displacement vector 604 depicts the position difference.

[0031] Figure 7A graphical user interface 700 is depicted in accordance with some embodiments. A user may use the graphical user interface 700 to set user-selectable values ​​for various parameters used by embodiments. Some embodiments utilize a graphical user interface to set user-selectable SURF and RANSAC parameters and other values.

[0032] Thus, the graphical user interface 700 includes a field for the user to select one or more control (baseline) images 702, a field for the user to select one or more process images 704, and a field for the user to select a file name for an output image 706. The selected control image and process image are processed by the disclosed embodiment to generate an output image, which is saved according to the file name selected by the user. The graphical user interface 700 also has a set of radio buttons 708 for selecting how to display any detected differences between the control image and the process image that represent possible defects. Thus, the graphical user interface 700 includes a radio button 710 for emphasizing the detected differences using lines, a radio button 712 for emphasizing the detected differences using crosses, a radio button 714 for emphasizing the detected differences using displacement vectors, a radio button 716 for emphasizing the detected differences using residuals, or a radio button 718 for emphasizing the detected differences using segments. Selecting the radio button 710 for emphasizing the detected differences using lines causes the output image to display feature points and corresponding lines. Selecting the radio button 712 to use crosses to emphasize detected differences causes the output image to depict each feature point as a cross of a different color. Selecting the radio button 714 to use displacement to emphasize detected differences causes the output image to display the feature points and the corresponding displacement vectors. That is, the displacement setting causes the output image to depict how the location of the feature points differs between the control image and the process image. Selecting the radio button 716 to use residuals to emphasize detected differences causes the output image to display the intensity differences between the different images after registration between the images. The residual setting is particularly useful for emphasizing foreign objects in the process image. Selecting the radio button 718 setting to use segmentation to emphasize detected differences hides any foreign objects in the output image. The segmentation setting is particularly useful for emphasizing where the images are different.

[0033] The graphical user interface 700 includes a bar that the user can use to select SURF parameter values. Specifically, the graphical user interface 700 includes a SURF Hessian threshold parameter bar 720 that the user can set. The SURF Hessian threshold parameter is used to calculate feature points, and the default value is 100. If more feature points are desired for registering images, this value can be increased. However, a higher value can result in more corner points and more noise. The graphical user interface 700 also includes a SURF minimum distance parameter bar 722 that the user can set. When SURF compares two points, this value can be increased to require a closer match. The graphical user interface 700 also includes a SURF virtual button 724, which can cause the SURF parameter pop-up window to be displayed when the virtual button 724 is activated. The SURF parameter pop-up window includes additional bars for SURF parameter values ​​that the user can set.

[0034] The graphical user interface 700 also includes a bar that the user can use to select RANSAC parameter values. Therefore, the graphical user interface 700 includes a "maximum local displacement" bar 726 in which the maximum local displacement value can be entered. The maximum local displacement value controls the size of the area that the RANSAC algorithm will search when trying to find matching features. The maximum local displacement value selects the maximum number of pixels of feature points that are shifted between the images being registered and are still considered to be matched. The graphical user interface 700 also includes a "high residual threshold" bar 728, in which the user can enter a value of the intensity difference between the two compared images that can still be considered as matching. The higher value of the "high residual threshold" parameter leads to more matches, but may not fully capture shape information. The graphical user interface 700 also includes a "low threshold" bar 730. The "low threshold" parameter is combined with the high residual threshold to determine how much difference is allowed between the images, while still allowing a conclusion of matching to be drawn. The graphical user interface 700 also includes a RANSAC virtual button 732, which causes the RANSAC parameter pop-up window to be displayed when the RANSAC virtual button 732 is activated. The RANSAC parameter pop-up window includes an additional column for user-settable RANSAC parameter values. The graphical user interface 700 also includes a reconstruction size column 734 for entering a user-selectable reconstruction size value. The reconstruction size value determines the pixel radius from the detected difference below, that is, small objects and noise can be removed from the detected difference when generating the output image.

[0035] Figure 8Depicted is an example output image 800 according to some embodiments. Specifically, an embodiment may output an output image (e.g., output image 800) for display to a user, for example. The output image emphasizes or otherwise annotates a combined image including data from multiple angles and possibly from both a process image and a control image. As shown, the output image 800 includes an elliptical annotation 802 that emphasizes residual fragments of tape present on an aircraft undergoing quality inspection. Since the corresponding tape fragments do not appear in the corresponding control image, the embodiment detects and annotates these fragments in the output image. The user may take subsequent steps to repair possible defects, such as removing any foreign matter, applying missed paint, or removing and relocating misplaced paint applications.

[0036] Fig. 9 is a flow chart of a method 900 for inspecting the quality of aircraft paint application according to some embodiments. The method 900 may be used as described above with reference to Figures 2 to 8 The method 900 may be performed using the hardware and techniques shown and described below. Fig.11 The system shown and described is implemented.

[0037] At block 902, method 900 retrieves a persistent storage device (e.g., Fig.11 The one or more reference images can be retrieved from a persistent storage device 1112 of the computer system. Such retrieval can be performed, for example, via an electronic computer network. For example, the reference images can be retrieved using Figure 2 The control image may be initially acquired by the device 200 of the aircraft. Thus, the control image may include images taken from multiple perspectives of substantially the same portion of the aircraft. Multiple aircraft parts (each with multiple perspectives) may be represented in the control image. The represented aircraft parts and perspectives may cover all paint and other coating applications of the aircraft. The captured control images may be stored and indexed according to aircraft position and perspective.

[0038] At block 904, method 900 captures one or more process images of an aircraft undergoing quality inspection. For example, the process images may be captured using Figure 2 The process image may be captured by the device 200 of FIG. 10 . Thus, the process image may include images taken from multiple perspectives of substantially the same portion of the aircraft undergoing quality inspection. Multiple aircraft portions (each having multiple perspectives) may be represented in the process image. The process image may correspond to the reference image in the sense that each process image has a corresponding reference image of substantially the same aircraft portion taken from substantially the same perspective.

[0039] At block 906, method 900 detects features in the process image. It should be noted that the actions of this block may also be performed on the control image. Specifically, the actions of this block may be performed on the control image at any point before the actions of block 910. Method 900 may be performed using the above reference Figure 4 and Figure 7 An implementation of SURF is shown and described to detect features.

[0040] At block 908, method 900 detects reflections, shadows, and other transient visual phenomena in the process image. It should be noted that the actions of this block may also be performed on the control image. Specifically, the actions of this block may be performed on the control image at any point before the actions of block 912. Fig.10 Describe the actions of this box in detail.

[0041] At block 910, method 900 registers the process image with the reference image. The actions of this block may be performed using, for example, the method described above with reference to Figures 4 to 7 RANSAC implementation shown and described. It should be noted that the actions of box 908 may result in some control images being merged into a merged control image and / or some process images being merged into a merged process image. For example, these merged images may include a union of the detected features. In such a case, the actions of this box may be applied to align the merged control image with the merged process image.

[0042] At block 912, method 900 detects differences between the registered process image and the reference image of block 910. The detection may be performed by comparing the images pixel by pixel and noting any differences that exceed a threshold. The detection may utilize a mathematical affine transformation (i.e., a mathematical linear transformation that preserves points, lines, and planes) between the reference image and the process image for registration. As described below with reference to Fig.10 As described, the detection may ignore any pixels identified as representing reflections or other transient visual phenomena. That is, the difference detection process may ignore any pixels identified as reflections (or other transient visual phenomena) based on a pixel-by-pixel comparison. In this way, some embodiments are able to automatically detect possible aircraft paint defects while ignoring transient visual features that do not represent paint or other physical anomalies.

[0043] At block 914, method 900 generates an output image representing any differences detected at block 912. Specifically, the actions of this block may include rendering an emphasis or other annotation (e.g., a circular, oval, rectangular, or square mark around any detected differences). Figure 8 An example suitable output image is presented in .

[0044] At block 916, method 900 causes the output image to be displayed. The actions of this block may take any of a variety of forms. According to some embodiments, the output image is displayed on a computer monitor. According to some embodiments, the output image is displayed in a virtual reality headset. According to some embodiments, the output image may be output to an electronic persistent storage or a user, for example, via an electronic computer network.

[0045] At block 918, method 900 includes the user repairing any detected discrepancies. The actions of this block may include the user removing any foreign matter from the aircraft, applying any missed paint or other paint applications, removing any improper paint or other paint applications, or removing and replacing any misplaced paint or other paint applications. The actions of this block are dependent on and guided by the output images of blocks 914 and 916.

[0046] Fig.10 is a flow chart of a reflection processing method 1000 according to some embodiments. The method 1000 may be implemented as performing a Fig. 9 In addition, the method 1000 may be applied to the reference at any point prior to the action of block 912 of the method 900. Fig. 9 Since method 1000 is applicable to both control images and process images, the following description of method 1000 refers to a generic "image" or "image under operation", it being understood that such an image may be a control image or a process image. Method 1000 may utilize the above referenced Figures 2 to 8 The method 1000 may be implemented using the hardware and techniques shown and described below. Fig.11 The system shown and described is implemented.

[0047] At block 1002, method 1000 places a standardized marker on the aircraft. To perform the actions of this block, a user may generate a marker based on a known dictionary and dimensions, print the marker to scale, and attach the marker to the aircraft at a location where the marker is visible to at least one camera and does not obstruct personnel. For example, the marker may be one foot by one foot. A suitable marker is an ArUco marker, which is square and includes a wide black border that facilitates rapid detection in an image and an internal matrix that may encode information. The actions of this block and the next block allow computer vision techniques to be applied in method 1000.

[0048] At block 1004, method 1000 detects the marker located in the image according to block 1002. To apply method 1000 to block 908 of method 900, the image in which the marker is detected may be a process image captured according to block 904 of method 900. To apply method 1000 to the control image used in method 900, the image in which the marker is detected may be captured at any suitable time (e.g., at a time immediately after the control image is captured). Method 1000 may detect the marker in the image using a dictionary for generating the marker, detect the corner of the marker, and obtain the coordinates of the corner. These coordinates may be represented as storage addresses corresponding to pixel locations.

[0049] At block 1006, method 1000 detects blobs in the image. Any suitable blob detection technique (e.g., RANSAC) may be used. Implementations may limit blob detection to detecting blobs that are roughly elliptical in shape (or more generally, roughly conic sections in shape). This restriction enhances the efficiency of the process and takes advantage of the overall tubular shape of most aircraft. Typically, reflections on tubular surfaces resemble ellipses or other conic sections.

[0050] At box 1008, method 1000 processes the image in various ways to enhance the detected spots. Some embodiments use adaptive thresholding in a known manner to obtain a binary image. These embodiments may use known techniques to identify the outlines of each spot and convert the image under operation to a black and white version as a working image. Some embodiments may use known corrosion and expansion techniques to modify (massage) the image under operation. These embodiments operate to close the spot boundaries using known techniques. Some embodiments submit the image to contour detection. These embodiments use these techniques to identify the boundary contours of each spot. These and other processing steps may be applied by various embodiments.

[0051] At block 1010, the method 1000 fits ellipses to the detected and processed spots. The actions of this block may include fitting ellipses to the identified boundary contours of the spots. The actions of this block may include checking the fit of the ellipses. Spots that sufficiently fit the ellipses are tracked as candidates for reflections. Therefore, at this block, the method 1000 maintains generally elliptical contours because these ellipses mark spots corresponding to possible reflections.

[0052] At block 1012, method 1000 fits bounding boxes to each reflection candidate. For example, these boxes better define the location of the reflection candidates because their corners are well defined. In addition, bounding boxes can correct for imperfect contour determination.

[0053] At block 1014, method 1000 detects features in the image. Feature detection may be limited to detecting features on and within reflection candidates. Figures 4 to 7 As shown and described, suitable techniques include SURF.

[0054] At block 1016, method 1000 determines distances from the bounding box (e.g., the corners of the bounding box) to the normalized marker (e.g., the corners of the normalized marker). These distances are stored in memory for later comparison with similar distances calculated using images taken from different perspectives.

[0055] At block 1018, method 1000 determines whether to repeat the actions of blocks 1004, 1006, 1008, 1010, 1012, 1014, and 1016 after moving the camera (or using a different camera). The determination is based on whether method 1000 has captured multiple images of substantially the same portion of the aircraft from different perspectives. The reflection detection process uses at least two perspectives, so some embodiments may proceed to block 1020 unless there are at least two such images for each portion of the aircraft under quality inspection. Some embodiments utilize more than two such images for each portion of the aircraft under inspection. If block 1018 determines that additional images are needed, control passes to block 1020. Otherwise, control passes to block 1022.

[0056] At block 1020, method 1000 moves the camera (e.g., Figure 2 The camera 202 of the device 200 of the embodiment of the present invention may be used to capture the image of the aircraft from different perspectives. After block 1020, control is transferred to block 1004.

[0057] At block 1022, method 1000 matches spots detected in substantially the same portion of the aircraft among images captured from multiple perspectives. Features detected in accordance with block 1014 may be used for this purpose. Figures 4 to 7 As shown and described, a matching process such as RANSAC may be used to perform the matching.After this block, each blob together with any other blobs determined to match the respective blob are referred to as a "set of matched blobs".

[0058] At block 1024, the method 1000 determines spots in the image to be processed as reflections or other transient visual phenomena. Typically, after capturing one image from angle "a" and another image from angle "b", if a feature moves (e.g., shifts, changes contrast, or disappears), the feature is processed as a reflection because reflections change as the angle of the light source or the position of the surface changes. If the feature does not change position, the feature is processed normally with the rest of the image. Therefore, at this block, the method 1000 compares the distances determined for the various spots within each matching spot set at block 1016. In other words, if the distances from a particular ArUco marker to each spot in the matching spot set are different between each image viewing angle, the spot is then processed as a reflection or other transient visual phenomenon. If the distances from a particular ArUco marker to each spot in the matching spot set are the same between each image viewing angle, the spot is then processed as the rest of the image.

[0059] Fig.111 is a diagram of a standardized marker 1102 according to some embodiments. As shown, the standardized marker is an example of an ArUco marker that encodes the numbers "123". However, various embodiments may use a standardized marker having any of a variety of encoded data.

[0060] Fig.12 1 is a schematic diagram of a system 1200 for implementing the methods shown and described (e.g., methods 900 and 1000). The system 1200 can be based on electronic hardware including one or more electronic processors 1210 that can be connected to the Internet 1204 when communicating. The system 1200 includes a computer that can be connected to the Internet 1204 when communicating and is connected to the client computer 1202 and the camera 202 (see Figure 2 The network interface 1208 may include a physical network interface such as a network adapter. The system 1200 may be a dedicated computer suitable for reliability and high-bandwidth communication. Thus, for example, the system 1200 may be implemented in a cluster of hardware server computers. The processor 1210 may be a multi-core processor suitable for processing large amounts of information. The processor 1210 is connected to the persistent storage device 1212 when communicating and may execute the stored instructions to cooperate with the client computer 1202 to implement the disclosed herein as described with reference to Fig. 9 and Fig.10 The techniques specifically shown and described. Persistent storage 1212 may be a redundant array of inexpensive disk drives (RAID) configuration for increased reliability. Processor 1210 is also communicatively connected to a graphics coprocessor (GPU) 1214. The graphics coprocessor may accelerate the techniques disclosed herein by performing operations in parallel.

[0061] In addition, the present disclosure includes examples according to the following enumerated paragraphs:

[0062] A1. A method (900) for automated aircraft (100) paint (106) quality inspection, the method comprising the steps of: retrieving (902) from an electronic persistent storage device (1112) a control image (302, 402, 502) depicting at least a portion of an aircraft paint (102, 104) of the aircraft under quality inspection; capturing (904) a process image (304, 404, 504) depicting at least a portion of the aircraft paint (102, 104) on the aircraft under quality inspection; detecting (906) a plurality of features (406) in the control image and the process image; registering (910) the control image with the process image; detecting (912) at least one difference between the control image and the process image; generating (914) an output image (800), wherein the output image includes a depiction of the aircraft under quality inspection wherein the at least one difference is annotated; and causing (916) the output image to be displayed.

[0063] A2. The method according to A1 further comprising the step of detecting (908) at least one reflection (306) and not considering the at least one reflection (306) as a difference between the control image and the process image.

[0064] A3. A method according to A2, wherein the step of not considering at least one reflection as a difference between the control image and the process image includes the following steps: capturing at least one image of the aircraft under quality inspection from a first perspective (208, 210, 212) and capturing at least one image of the aircraft under quality inspection from a second perspective (208, 210, 212).

[0065] A4. A method according to A3, wherein the step of not considering at least one reflection as a difference between the control image and the process image includes the following steps: before capturing at least one image of the aircraft under quality inspection from a first perspective and capturing at least one image of the aircraft under quality inspection from a second perspective, placing at least one standardized mark (1102) on the aircraft under quality inspection.

[0066] A5. According to the method described in A1 to A4, the method also includes the following steps: repairing the anomaly corresponding to the at least one difference.

[0067] A6. A method according to A1 to A5, wherein the step of detecting multiple features includes the following steps: using a speeded up robust features (SURF) algorithm to detect multiple features.

[0068] A7. A method according to A1 to A6, wherein the step of aligning the control image with the process image is performed using a random sample consensus (RANSAC) algorithm.

[0069] A8. A method according to A1 to A7, wherein the step of generating an output image includes the following steps: combining at least a portion of an image taken from a first perspective (208, 210, 212) with at least a portion of an image taken from a second perspective (208, 210, 212) different from the first perspective.

[0070] A9. A method according to A1 to A8, wherein the control image includes a combination of a first control image taken from a first perspective (208, 210, 212) and a second control image taken from a second perspective (208, 210, 212) different from the first perspective, and wherein the process image includes a combination of a first process image taken from a third perspective (208, 210, 212) and a second process image taken from a fourth perspective (208, 210, 212) different from the third perspective.

[0071] A10. A method according to A1 to A9, wherein the step of detecting at least one difference includes the step of performing a pixel-by-pixel comparison between the control image and the process image.

[0072] B1. A system (1200) for automated aircraft (100) paint (106) quality inspection, the system comprising: an electronic persistent storage device (1212) storing a comparison image (302, 402, 502) depicting at least a portion (102, 104) of an aircraft paint of the aircraft under quality inspection; a plurality of cameras (202, 204, 206) configured to capture a plurality of images of the aircraft under quality inspection; and at least one electronic processor (1210) communicatively coupled to the electronic persistent storage device and the plurality of cameras, the at least one electronic processor (1210) being coupled to the electronic persistent storage device and the plurality of cameras. 10) executing instructions for performing operations, the operations comprising: retrieving (902) a control image from an electronic persistent storage device; capturing (904) a process image (304, 404, 504) depicting at least a portion of an aircraft livery (102, 104) on an aircraft under quality inspection via the plurality of cameras; detecting (906) a plurality of features (406) in the control image and the process image; registering (910) the control image with the process image; detecting (912) at least one difference between the control image and the process image; generating (914) an output image (800), wherein the output image includes a depiction of the aircraft under quality inspection wherein the at least one difference is annotated; and causing (916) the output image to be displayed.

[0073] B2. The system of B1, wherein the operations further comprise detecting (908) at least one reflection (306) and not considering the at least one reflection (306) as a difference between the control image and the process image.

[0074] B3. A system according to B2, wherein the operation of not considering at least one reflection as a difference between the control image and the process image includes the following operations: capturing at least one image of the aircraft under quality inspection from a first perspective (208, 210, 212) by the multiple cameras and capturing at least one image of the aircraft under quality inspection from a second perspective (208, 210, 212) by the multiple cameras.

[0075] B4. The system of B3 further comprising at least one standardized marker (1102) configured to be placed on the aircraft under quality inspection before capturing at least one image of the aircraft under quality inspection from a first perspective and capturing at least one image of the aircraft under quality inspection from a second perspective.

[0076] B5. A system according to B1 to B4, wherein the operation further comprises the following operation: guiding the user to repair the anomaly corresponding to the at least one difference.

[0077] B6. A system according to B1 to B5, wherein the operation of detecting multiple features includes the following operation: using a speeded up robust features (SURF) algorithm to detect multiple features.

[0078] B7. A system according to B1 to B6, wherein the operation of aligning the control image with the process image is performed using a random sample consensus (RANSAC) algorithm.

[0079] B8. A system according to B1 to B7, wherein the operation of generating an output image includes the following operation: combining at least a portion of an image captured from a first perspective (208, 210, 212) with at least a portion of an image captured from a second perspective (208, 210, 212) different from the first perspective.

[0080] B9. A system according to B1 to B8, wherein the control image includes a combination of a first control image taken from a first perspective (208, 210, 212) and a second control image taken from a second perspective (208, 210, 212) different from the first perspective, and wherein the process image includes a combination of a first process image taken from a third perspective (208, 210, 212) and a second process image taken from a fourth perspective (208, 210, 212) different from the third perspective.

[0081] B10. A system according to B1 to B9, wherein the operation of detecting at least one difference includes the following operation: performing a pixel-by-pixel comparison between the control image and the process image.

[0082] It is noted that some embodiments support any remote inspection of features of a three-dimensional structural surface, not limited to aircraft.

[0083] Some of the above examples may be performed in part using a computer application or program. Computer programs may exist in various forms (including active and inactive). For example, a computer program may exist as one or more software programs, software modules, or both, and the computer program may include source code, object code, executable code, or other format of program instructions, firmware programs, or hardware description language (HDL) files. Any of the above items may be implemented in compressed or uncompressed form on a computer-readable medium that may include a computer-readable storage device and medium. Exemplary computer-readable storage devices and media include conventional computer system RAM (random access memory), ROM (read-only memory), EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), and magnetic disks or optical disks or tapes.

[0084] Those skilled in the art will be able to make various modifications to the described examples without departing from the true spirit and scope. The terms and descriptions used herein are merely illustrative and are not intended to be limiting. In particular, although the method has been described by way of example, the steps of the method may be performed in a different order than that illustrated or may be performed simultaneously. Those skilled in the art will recognize that these and other variations are possible within the spirit and scope defined by the appended claims and their equivalents.

Claims

1. A method (900) for automatically inspecting the quality of an aircraft (100) painting (106), the method comprising the following steps: retrieving (902) from an electronic persistent storage device (1112) a comparison image (302, 402, 502) depicting at least a portion of an aircraft livery (102, 104) of an aircraft under quality inspection; placing at least one standardized marking (1102) on the aircraft under quality inspection; capturing (904) a process image (304, 404, 504) depicting at least the portion of the aircraft paint (102, 104) on the aircraft under quality inspection; capturing at least a first perspective image of at least the portion of the aircraft paint under quality inspection from a first perspective and capturing at least a second perspective image of at least the portion of the aircraft paint under quality inspection from a second perspective different from the first perspective; detecting (908) at least one reflection (306) by detecting different distances between the at least one standardized marker and the candidate spot in the first view image and the second view image, and not considering the at least one reflection (306) as a difference between the control image and the process image; detecting (906) a plurality of features (406) in the control image and the process image; registering the control image with the process image by matching detected features (910); detecting (912) at least one difference between the control image and the process image; generating (914) an output image (800), wherein the output image includes a depiction of the aircraft under quality inspection wherein the at least one difference is annotated; and The output image is caused (916) to be displayed.

2. The method according to claim 1, further comprising the following steps: An exception corresponding to the at least one difference is repaired.

3. The method according to claim 1 or 2, wherein: The step of detecting a plurality of features comprises the following steps: using a speeded up robust feature SURF algorithm to detect a plurality of features.

4. The method according to claim 1 or 2, wherein: The step of registering the control image with the process image is performed using a random sample consensus RANSAC algorithm.

5. The method according to claim 1 or 2, wherein: The step of generating an output image includes the step of combining at least a portion of an image captured from a first viewing angle (208, 210, 212) with at least a portion of an image captured from a second viewing angle (208, 210, 212) different from the first viewing angle.

6. The method according to claim 1 or 2, wherein: The control image comprises a combination of a first control image taken from a first viewing angle (208, 210, 212) and a second control image taken from a second viewing angle (208, 210, 212) different from the first viewing angle, and wherein the process image comprises a combination of a first process image taken from a third viewing angle (208, 210, 212) and a second process image taken from a fourth viewing angle (208, 210, 212) different from the third viewing angle.

7. The method according to claim 1 or 2, wherein: The step of detecting at least one difference comprises the step of performing a pixel-by-pixel comparison between the reference image and the process image.

8. A system (1200) for automatic aircraft (100) painting (106) quality inspection, the system comprising: an electronic persistent storage device (1212) storing a comparison image (302, 402, 502) depicting at least a portion (102, 104) of an aircraft livery of an aircraft under quality inspection; a plurality of cameras (202, 204, 206) arranged to capture a plurality of images of an aircraft under quality inspection; as well as at least one electronic processor (1210) communicatively coupled to the electronic persistent memory and the plurality of cameras, the at least one electronic processor executing instructions for performing operations comprising: retrieving (902) the control image from the electronic persistent storage device; placing at least one standardized marking (1102) on the aircraft under quality inspection; capturing (904) by the plurality of cameras a process image (304, 404, 504) depicting at least the portion of the aircraft paint (102, 104) on the aircraft under quality inspection; capturing at least a first perspective image of at least the portion of the aircraft paint under quality inspection from a first perspective and capturing at least a second perspective image of at least the portion of the aircraft paint under quality inspection from a second perspective different from the first perspective; detecting at least one reflection by detecting different distances between the at least one standardized marker and the candidate spot in the first view image and the second view image, and not considering the at least one reflection as a difference between the control image and the process image; detecting (906) a plurality of features (406) in the control image and the process image; registering the control image with the process image by matching detected features (910); detecting (912) at least one difference between the control image and the process image; generating (914) an output image (800), wherein the output image includes a depiction of the aircraft under quality inspection wherein the at least one difference is annotated; and The output image is caused (916) to be displayed.

9. The system according to claim 8, wherein: The operations also include the following operation: guiding the user to repair the anomaly corresponding to the at least one difference.

10. The system according to claim 8 or 9, wherein: The operation of detecting the plurality of features includes the following operation: detecting the plurality of features using a speeded up robust feature SURF algorithm.

11. The system according to claim 8 or 9, wherein: The operation of registering the control image with the process image is performed using a random sample consensus RANSAC algorithm.

12. The system according to claim 8 or 9, wherein: The operation of generating an output image includes the operation of combining at least a portion of an image captured from a first perspective (208, 210, 212) with at least a portion of an image captured from a second perspective (208, 210, 212) different from the first perspective.

13. The system according to claim 8 or 9, wherein: The control image comprises a combination of a first control image taken from a first viewing angle (208, 210, 212) and a second control image taken from a second viewing angle (208, 210, 212) different from the first viewing angle, and wherein the process image comprises a combination of a first process image taken from a third viewing angle (208, 210, 212) and a second process image taken from a fourth viewing angle (208, 210, 212) different from the third viewing angle.

14. The system according to claim 8 or 9, wherein: The operation of detecting at least one difference includes the operation of performing a pixel-by-pixel comparison between the control image and the process image.

Images