Aircraft surface defect visual inspection method based on unmanned aerial vehicle inspection

Abstract

The invention discloses an aircraft surface defect visual detection method based on unmanned aerial vehicle inspection, and relates to the technical field of aircraft surface defect detection, and the method comprises the steps: building a device employing a four-rotor unmanned aerial vehicle to carry a visual sensor; the technology of generating a high-dynamic image based on a multi-exposure image is adopted, image sequences of different exposure times on the surface of an aircraft are collected, and the high-dynamic image is generated in a polynomial weighted fusion mode; through an aircraft pose estimation method based on key points, unmanned aerial vehicle flight envelope and flight path planning and aircraft body surface target accurate positioning are carried out; an unsupervised body surface defect detection mode is established, image anomaly detection is realized by adopting an auto-encoder model, and the current image anomaly condition is judged by judging a pixel-level reconstruction error between an input image and an output image. The system is suitable for visual detection of aircraft surface defects in a ground environment, has the characteristics of automation, intellectualization and the like, and is of great significance in guaranteeing flight safety and improving army fighting capacity.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft surface defect detection, and in particular to an aircraft surface defect visual detection method based on unmanned aerial vehicle inspection. Background Art

[0002] Aircraft surface defects are highly concealed and have high hidden dangers, which seriously affect flight safety and reduce combat effectiveness, and can even easily cause flight accidents. In recent years, with the rapid development of the aviation industry, higher requirements have been put forward for ensuring flight safety. Therefore, regular inspection and maintenance of aircraft surface defects have become a mandatory requirement for aircraft maintenance. At present, the surface defects of the fuselage are mostly detected by non-destructive testing such as ultrasound and infrared, supplemented by manual visual inspection. Due to the limitations of large aircraft size, numerous components and human factors, there are shortcomings such as long detection time, high missed detection and false detection rate. With the rapid development of machine vision, the use of automated visual inspection instead of manual inspection has not only improved the detection efficiency, but also reduced the missed detection and false detection rate in the manual inspection process. Therefore, it is of great significance to study large-field-of-view, high-precision and intelligent related detection technologies that can meet the needs of online detection of aircraft surface defects to ensure aircraft flight safety.

[0003] With the rapid development of artificial intelligence, drone technology and robotics, large-space, flexible imaging of aircraft surfaces can be achieved by combining visual sensors with different carriers, and the automated and intelligent detection of aircraft surface defects can be achieved with the help of artificial intelligence recognition algorithms. Existing aircraft skin inspection based on crawling robots has not been actually applied on site due to its slow movement speed and low detection efficiency. Multiple groups of large robots have not yet been widely used due to their large size, blind spots, and high cost. Quadcopter drones are small, can hover, and cruise, and can be equipped with visual sensors and intelligent processing platforms to achieve flexible and efficient inspections of aircraft surfaces, but there are currently no mature products that can be promoted on a large scale. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method for visual detection of aircraft surface defects based on UAV inspection. Focusing on the UAV visual detection technology of aircraft surface defects, a quad-rotor UAV, an aircraft posture measurement camera and a high dynamic optical sensor are adopted. By determining the posture parameters of the aircraft to be inspected relative to the UAV system, a spatial patrol trajectory is established, and the gimbal is controlled to drive the high dynamic optical sensor to achieve full coverage of the aircraft fuselage to be inspected, and the aircraft surface defects are detected, thereby realizing aircraft surface defect detection.

[0005] The aircraft surface defect drone visual inspection technology uses a quad-rotor drone equipped with a visual sensor to inspect the aircraft surface, achieving automatic identification and early warning of target positioning and defects on the aircraft surface. It can greatly improve the efficiency and quality of ground maintenance of large aircraft, drones, etc., drive and promote the development of engineering technologies such as materials, manufacturing, and information, accelerate the diversified development of aviation science and technology, and play a certain leading role in improving innovation capabilities.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] A method for visually detecting aircraft surface defects based on drone inspection includes the following steps:

[0008] Step 11: Build a detection device to perform patrol inspection on the aircraft to be inspected. The detection device includes a quad-rotor drone, a built-in gimbal and a visual sensor; the visual sensor is used to image the aircraft to be inspected, obtain multi-exposure image data, and determine the posture parameters of the aircraft to be inspected relative to the quad-rotor drone by shooting the aircraft to be inspected, establish a spatial patrol trajectory, and achieve accurate matching and positioning of the aircraft target; the visual sensor performs multi-view imaging on the fuselage of the aircraft to be inspected to achieve rapid and high-precision detection and early warning of aircraft surface defects;

[0009] Step 12: Based on the method of merging multi-exposure image data to generate a high dynamic image, multiple single-exposure images are taken for the same scene, and a high-definition image of the entire scene is obtained through image alignment and weighted fusion;

[0010] Step 13: by adding a spatial positioning camera for the aircraft to be detected, a key point-based aircraft posture estimation algorithm is used to achieve preliminary positioning of the aircraft on the apron relative to the aircraft carrier platform; the key points are key points with obvious topological structures of the aircraft body and little change with viewing angle;

[0011] Step 14: import the 3D model data of the aircraft to be detected, and optimally design the flight envelope and track of the quadcopter, and plan the spatial station position of the quadcopter to be detected; align the surface image data of the aircraft to be detected collected each time with the 3D model data of the aircraft to be detected, so as to achieve accurate and rapid positioning of the surface target of the aircraft;

[0012] Step 15: Collect a large number of normal aircraft surface images, use the autoencoder model to detect image anomalies, and determine the abnormality of the current image by judging the pixel-level reconstruction error between the input image and the output image.

[0013] Further, the step 12 comprises:

[0014] The camera response function is expressed as a polynomial:

[0015] (1)

[0016] in, is the degree of the polynomial, For polynomial The coefficient of the second term, is the radiance function of the camera detector, Indicates the exposure time, Represents the response function of the transformation from image pixel value to scene radiance, Represents the image position; by estimating the polynomial degree and coefficient Uniquely determine the camera response function;

[0017] Arrange the images of the aircraft surface to be detected in order of their exposure time from short to long, satisfying ;

[0018] in, , Indicates the time of two exposures;

[0019] The camera response function is recovered by minimizing the following error function:

[0020] (2)

[0021] in, is the ratio of the radiometric values ​​of images with different exposures, , by minimizing the error function, the coefficients of the polynomial are solved .

[0022] Further, the step 13 comprises:

[0023] Aircraft posture estimation and rapid positioning of the surface target of the aircraft to be detected are divided into: preliminary estimation of aircraft posture, optimal flight path planning, precise alignment of local areas of the aircraft, and rapid positioning of aircraft targets;

[0024] The initial estimation of the aircraft posture uses the visual sensor in the quadcopter's built-in gimbal to image the aircraft to be detected from a long distance, detect the pre-set key points of the aircraft, assist with the key frame data established in the early stage, and solve the preliminary posture parameters of the aircraft to be detected on the apron relative to the quadcopter through the PNP method, thereby realizing the preliminary positioning of the aircraft on the apron relative to the carrier platform.

[0025] Further, the step 14 comprises:

[0026] Based on the three-dimensional model library of the aircraft to be detected, the spatial orientation distribution of the aircraft to be detected in the UAV coordinate system is planned, and then the UAV patrol trajectory and envelope curve are planned according to the three-dimensional information to be detected, where the envelope curve and the patrol trajectory are both outside a safe distance from the aircraft to be detected; the quadcopter is controlled to image the aircraft surface using its own visual sensor along the planned track.

[0027] Further, the step 15 comprises:

[0028] A large number of normal aircraft surface images are collected to form a data set; the autoencoder model is an anomaly detection network model, and the input of the anomaly detection network model during training is a normal aircraft body image, so that the output of the anomaly detection network model is the same as the input of the anomaly detection network model;

[0029] Pixel-level SSIM loss The calculation formula is:

[0030] (3)

[0031] in, , are the average values ​​of the generated image and the average values ​​of the input image respectively; , are the standard deviation of the generated image and the standard deviation of the input image respectively; is the covariance of the generated image and the input image; , To prevent the constant term in the formula from dividing by 0, , Represents the horizontal and vertical coordinates of image pixels.

[0032] Furthermore, in the anomaly detection network model inference stage, the body surface image of the aircraft to be detected is used as the input of the anomaly detection network model. After calculation, the anomaly detection network model outputs an image reconstructed by the anomaly detection network model; the pixel-level reconstruction error between the input image and the output image is judged, and the value of the reconstruction error is compared with a pre-set threshold. If the value of the reconstruction error is less than the set threshold, it is determined that there is no anomaly; otherwise, it is considered that there is an anomaly in the image, and the body position where the abnormal image is located is marked in time, and an early warning is issued.

[0033] The beneficial effects of the present invention compared with the prior art are:

[0034] The present invention adopts a mode of visual inspection of aircraft surface defects based on drone inspection. The design adopts a mode of patrol inspection of the aircraft to be inspected by a four-rotor drone carrying a visual sensor, assisted by power supply control and automatic recognition devices, which can perform automatic and intelligent detection of aircraft surface defects on the ground. It has the characteristics of simple structure, high efficiency, intelligence, and strong versatility. By combining the visual sensor with the drone, the drone can be fully utilized to fly and reach in a large space, which greatly expands the detection range of the visual sensor. Assisted by the intelligent recognition algorithm for aircraft surface defects, the efficiency and quality of aircraft surface defect detection can be effectively improved. At the same time, it has automation and intelligence, and promotes the leap from "manual inspection" to "intelligent inspection" for aircraft surface defects. According to the technology of generating high dynamic images based on multi-exposure images, by collecting the aircraft surface Image sequences with different exposure times are generated by polynomial weighted fusion, which can effectively solve the problem of not being able to ensure clear imaging of various parts due to complex outdoor environmental light, large and complex three-dimensional topological structure of the aircraft, and material, improve the dynamic range and clarity of the image, and improve the accuracy of aircraft surface defect detection; the aircraft posture estimation method based on key points is used to accurately locate the flight envelope and trajectory of the drone and the surface target of the fuselage, and the optimal design of the flight envelope and trajectory of the drone is carried out to plan the spatial measurement station position of the drone to be detected, which reduces the complexity of the drone flight envelope caused by manual operation of the aircraft, inaccurate positioning, and the risk of secondary damage to the aircraft; at the same time, it ensures that the visual system accurately locates the aircraft detection area, laying the foundation for accurate detection of aircraft surface defects. By collecting a large number of normal aircraft surface images, the unsupervised anomaly detection method of image anomaly detection is realized by using the autoencoder model, and the current image abnormality is determined by judging the pixel-level reconstruction error between the input image and the output image, so as to realize accurate detection of aircraft surface defects without relying on negative samples. It has the characteristics of simplicity and efficiency, and has certain versatility, and is also suitable for defect detection scenarios of large workpieces.

[0035] The present invention can be further promoted and used by relevant users at home and abroad, and has important application value and broad market prospects. At the same time, the relevant research results can be extended to aerospace, rail transportation and other fields to meet their needs for appearance defect detection of large equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of a method for visually detecting aircraft surface defects based on drone inspection of the present invention;

[0037] Figure 2 Schematic diagram of drone visual inspection of aircraft surface defects;

[0038] Figure 3 Schematic diagram of the unsupervised anomaly detection of body surface defects. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0040] The basic idea of ​​the present invention is to design a new mode of patrol inspection of aircraft surface defects using a quad-rotor drone carrying a visual sensor; to propose a method of generating high-dynamic images from images of the aircraft surface with different exposure levels; and to propose an unsupervised aircraft surface defect recognition method.

[0041] like Figure 1 As shown, a method for visually detecting aircraft surface defects based on drone inspection of the present invention comprises the following steps:

[0042] Step 11: Build a detection device to perform patrol inspection on the aircraft to be inspected. The detection device includes a quad-rotor drone, a built-in gimbal and a visual sensor. The visual sensor is used to image the aircraft to be inspected and obtain multi-exposure image data. By shooting the aircraft to be inspected, the posture parameters of the aircraft to be inspected relative to the quad-rotor drone can be determined, the spatial patrol trajectory can be established, and the precise matching and positioning of the body target can be achieved. Then the visual sensor performs multi-view imaging on the fuselage of the aircraft to be inspected to achieve rapid and high-precision detection and early warning of aircraft surface defects.

[0043] Step 12: Generate a high dynamic range image based on the multi-exposure image, including:

[0044] The image acquisition camera on the aircraft surface uses a high-dynamic camera, which can capture high-definition images of scenes in complex lighting environments. At the same time, in order to ensure the best imaging quality of each part of the aircraft body, and not be affected by factors such as outdoor light and body material, a high-dynamic image is generated by merging multiple exposure images to obtain high-dynamic imaging of the aircraft body surface.

[0045] The present invention uses a polynomial to represent the camera response function:

[0046] (1)

[0047] in, is the degree of the polynomial, For polynomial The coefficient of the second term, is the radiance function of the camera detector, Indicates the exposure time, Represents the response function of the transformation from image pixel value to scene radiance, Represents the image position; by estimating the polynomial degree and coefficient Uniquely determines the camera response function.

[0048] By estimating the degree of the polynomial and coefficient This polynomial can be uniquely determined.

[0049] Arrange the images of the aircraft surface to be detected in order of their exposure time from short to long, satisfying .

[0050] The camera response function is recovered by minimizing the following error function:

[0051] (2)

[0052] in, is the ratio of the radiometric values ​​of images with different exposures, , by minimizing the error function, the coefficients of the polynomial are solved .

[0053] Step 13: Aircraft pose estimation based on key points (points where the curvature of the aircraft surface changes significantly and changes less with the viewing angle are taken as aircraft key points, such as the nose, wing tip, and tail tip, etc.), including:

[0054] Aircraft posture estimation and rapid positioning of the surface target of the aircraft to be detected are divided into: preliminary estimation of aircraft posture, optimal flight path planning, precise alignment of local areas of the aircraft, and rapid positioning of aircraft targets.

[0055] The initial estimation of the aircraft posture uses the visual sensor in the built-in gimbal of the quadcopter to image the aircraft to be detected from a distance, detect the key points of the aircraft set in advance, and assist with the key frame data established in the early stage. The preliminary posture parameters of the aircraft to be detected relative to the drone at rest on the apron are solved by the PNP method;

[0056] Step 14: Plan the flight envelope and trajectory of the drone and accurately locate the surface targets of the aircraft, including:

[0057] Based on the three-dimensional model library of the aircraft to be detected, the spatial orientation distribution of the aircraft to be detected in the coordinate system of the drone can be planned, and then the drone patrol trajectory and envelope curve can be planned according to the three-dimensional information to be detected from the surface data of the aircraft to be detected, wherein the envelope curve and the patrol trajectory are both outside the safe distance from the aircraft to be detected, ensuring that the four-rotor drone will not collide with the aircraft to be detected and cause secondary damage. The present invention controls the four-rotor drone to image the surface of the aircraft using its own visual sensor under the planned track, which can ensure the accuracy of the aircraft surface area photographed by the current high-dynamic camera, and accurately image the local area of ​​the aircraft, so as to achieve rapid and accurate positioning of the target on the aircraft surface, and lay the foundation for subsequent recognition of the surface area of ​​the body.

[0058] Step 15: Perform unsupervised body surface defect detection, including:

[0059] The autoencoder model is used to detect anomalies in the image for the recognition of aircraft surface defects. The autoencoder model is an anomaly detection network model. A large number of normal aircraft surface images are collected in the early stage to form a data set. This work only needs to be collected once for a type of aircraft, and the data set is universal. The structure of the anomaly detection network model is as follows: Figure 3 As shown in the figure, the input of the anomaly detection network model during training is a normal body picture, so that the output of the anomaly detection network model is as similar as possible to the input of the anomaly detection network model, and indicators such as pixel-level SSIM loss are used.

[0060] In the model inference stage, the surface image of the aircraft to be detected is used as the input of the model. After the model is calculated, it outputs an image reconstructed by the model. The pixel-level reconstruction error between the input image and the output image is determined, and the value of the reconstruction error is compared with the pre-set threshold. If the value of the reconstruction error is less than the set threshold, it is determined that there is no abnormality. Otherwise, it is considered that there is an abnormality in the image, and the position of the body where the abnormal image is located is marked in time, and an early warning is issued.

[0061] Pixel-level SSIM loss The calculation formula is:

[0062] (3)

[0063] in, , are the average values ​​of the generated image and the average values ​​of the input image respectively; , are the standard deviation of the generated image and the standard deviation of the input image respectively; is the covariance of the generated image and the input image; , To prevent the constant term in the formula from dividing by 0, , Represents the horizontal and vertical coordinates of image pixels.

[0064] like Figure 2 As shown in the figure, the aircraft to be inspected is parked on the apron, and the carrier platform carries the UAV into the airport. The UAV first determines the posture parameters of the aircraft to be inspected, that is, completes the initial positioning of the aircraft to be inspected, and then plans to form the UAV patrol trajectory and flight envelope. The UAV collects data from various parts of the aircraft to be inspected and performs aircraft surface defect detection. The entire inspection work is completed within the inspection space envelope.

[0065] like Figure 3 As shown in the figure, the surface image of the aircraft to be inspected collected by the drone is input, and the image features are extracted and reconstructed through the convolutional network. Finally, the reconstructed image is obtained. By calculating the difference between the abnormal image and the reconstructed image, the abnormal area in the image is formed, and the detection and positioning of the aircraft surface defects are completed.

Claims

1. A method for visual inspection of aircraft surface defects based on drone inspection, characterized in that: The steps include: Step 11: Build a detection device to perform patrol inspection on the aircraft to be inspected. The detection device includes a quad-rotor drone, a built-in gimbal and a visual sensor; the visual sensor is used to image the aircraft to be inspected, obtain multi-exposure image data, and determine the posture parameters of the aircraft to be inspected relative to the quad-rotor drone by shooting the aircraft to be inspected, establish a spatial patrol trajectory, and achieve accurate matching and positioning of the aircraft target; the visual sensor performs multi-view imaging on the fuselage of the aircraft to be inspected to achieve rapid and high-precision detection and early warning of aircraft surface defects; Step 12: Based on the method of merging multi-exposure image data to generate a high dynamic image, multiple single-exposure images are taken for the same scene, and a high-definition image of the entire scene is obtained through image alignment and weighted fusion; Step 13: by adding a spatial positioning camera for the aircraft to be detected, a key point-based aircraft posture estimation algorithm is used to achieve preliminary positioning of the aircraft on the apron relative to the aircraft carrier platform; the key points are key points with obvious topological structures of the aircraft body and little change with viewing angle; Step 14: import the 3D model data of the aircraft to be detected, and optimally design the flight envelope and track of the quadcopter, and plan the spatial station position of the quadcopter to be detected; align the surface image data of the aircraft to be detected collected each time with the 3D model data of the aircraft to be detected, so as to achieve accurate and rapid positioning of the surface target of the aircraft; Step 15: Collect a large number of normal aircraft surface images, use the autoencoder model to detect image anomalies, and determine the abnormality of the current image by judging the pixel-level reconstruction error between the input image and the output image.

2. The method for visually detecting aircraft surface defects based on drone inspection according to claim 1 is characterized in that: The step 12 comprises: The camera response function is expressed as a polynomial: (1) in, is the degree of the polynomial, For polynomial The coefficient of the second term, is the radiance function of the camera detector, Indicates the exposure time, Represents the response function of the transformation from image pixel value to scene radiance, Represents the image position; by estimating the polynomial degree and coefficient Uniquely determine the camera response function; Arrange the images of the aircraft surface to be detected in order of their exposure time from short to long, satisfying ; in, , Indicates the time of two exposures; The camera response function is recovered by minimizing the following error function: (2) in, is the ratio of the radiometric values ​​of images with different exposures, , by minimizing the error function, the coefficients of the polynomial are solved .

3. The method for visually detecting aircraft surface defects based on drone inspection according to claim 1 is characterized in that: The step 13 comprises: Aircraft posture estimation and rapid positioning of the surface target of the aircraft to be detected are divided into: preliminary estimation of aircraft posture, optimal flight path planning, precise alignment of local areas of the aircraft, and rapid positioning of aircraft targets; The initial estimation of the aircraft posture uses the visual sensor in the quadcopter's built-in gimbal to image the aircraft to be detected from a long distance, detect the pre-set key points of the aircraft, assist with the key frame data established in the early stage, and solve the preliminary posture parameters of the aircraft to be detected on the apron relative to the quadcopter through the PNP method, thereby realizing the preliminary positioning of the aircraft on the apron relative to the carrier platform.

4. The method for visually detecting aircraft surface defects based on drone inspection according to claim 1 is characterized in that: The step 14 comprises: Based on the three-dimensional model library of the aircraft to be detected, the spatial orientation distribution of the aircraft to be detected in the UAV coordinate system is planned, and then the UAV patrol trajectory and envelope curve are planned according to the three-dimensional information to be detected, where the envelope curve and the patrol trajectory are both outside a safe distance from the aircraft to be detected; the quadcopter is controlled to image the aircraft surface using its own visual sensor along the planned track.

5. The method for visually detecting aircraft surface defects based on drone inspection according to claim 1 is characterized in that: The step 15 comprises: A large number of normal aircraft surface images are collected to form a data set; the autoencoder model is an anomaly detection network model, and the input of the anomaly detection network model during training is a normal aircraft body image, so that the output of the anomaly detection network model is the same as the input of the anomaly detection network model; Pixel-level SSIM loss The calculation formula is: (3) in, , are the average values ​​of the generated image and the average values ​​of the input image respectively; , are the standard deviation of the generated image and the standard deviation of the input image respectively; is the covariance of the generated image and the input image; , To prevent the constant term in the formula from dividing by 0, , Represents the horizontal and vertical coordinates of image pixels.

6. The method for visually detecting aircraft surface defects based on drone inspection according to claim 5 is characterized in that: In the anomaly detection network model inference stage, the body surface image of the aircraft to be detected is used as the input of the anomaly detection network model. After calculation, the anomaly detection network model outputs an image reconstructed by the anomaly detection network model; the pixel-level reconstruction error between the input image and the output image is judged, and the value of the reconstruction error is compared with a pre-set threshold. If the value of the reconstruction error is less than the set threshold, it is determined that there is no anomaly; otherwise, it is considered that there is an anomaly in the image, and the body position where the abnormal image is located is marked in time, and an early warning is issued.

Images