A visual detection method for aircraft surface defects based on drone inspection

Abstract

The present invention discloses a method for visually detecting aircraft surface defects based on drone inspections, which relates to the technical field of aircraft surface defect detection, including: building a device using a quad-rotor drone carrying a visual sensor; using a high-dynamic image generation technology based on multi-exposure images, by collecting a sequence of images of the aircraft surface with different exposure times, and using a polynomial weighted fusion method to generate high-dynamic images; using a key point-based aircraft posture estimation method, the drone flight envelope and trajectory planning and the accurate positioning of targets on the body surface; establishing an unsupervised body surface defect detection method, using an autoencoder model to realize image anomaly detection, and determining the current image anomaly by judging the pixel-level reconstruction error between the input image and the output image. The present invention is suitable for visually detecting aircraft surface defects in a ground environment, has the characteristics of automation and intelligence, and is of great significance to ensuring flight safety and improving the combat effectiveness of troops.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft surface defect detection, and in particular to a method for visually detecting aircraft surface defects based on unmanned aerial vehicle inspection. Background Art

[0002] Aircraft surface defects are highly concealed and present a high risk of potential danger. They seriously impact flight safety, reduce combat effectiveness, and can even easily cause accidents. In recent years, the rapid development of the aviation industry has placed higher demands on flight safety. Therefore, regular inspection and repair of aircraft surface defects has become a mandatory requirement for aircraft maintenance. Currently, surface defects are primarily detected using manual visual inspection, supplemented by non-destructive testing such as ultrasonic and infrared testing. However, due to limitations such as the large size of aircraft, the large number of components, and human factors, these methods suffer from long inspection times, high rates of missed and false detections. With the rapid development of machine vision, automated visual inspection has replaced manual inspection, improving inspection efficiency while reducing the missed and false detection rates associated with manual inspection. Therefore, research on large-field-of-view, high-precision, and intelligent inspection technologies that can meet the needs of online detection of aircraft surface defects is crucial for ensuring aircraft flight safety.

[0003] With the rapid development of artificial intelligence, drone technology, and robotics, the integration of visual sensors with various carriers enables large-scale, flexible imaging of aircraft surfaces. This, combined with artificial intelligence recognition algorithms, enables automated, intelligent detection of aircraft surface defects. Existing aircraft skin inspections based on crawling robots are slow and inefficient, making them impractical for field use. Systems based on multiple large robots are not widely applicable due to their bulk, blind spots, and high cost. Quadcopter drones, with their compact size, hovering, and cruising capabilities, can be equipped with visual sensors and intelligent processing platforms to enable flexible and efficient inspections of aircraft surfaces. However, mature products suitable for large-scale deployment are not yet available. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a method for visual detection of aircraft surface defects based on drone inspection. It focuses on the drone visual detection technology of aircraft surface defects, adopts a four-rotor drone, an aircraft posture measurement camera and a high-dynamic optical sensor, and establishes a spatial patrol trajectory by determining the posture parameters of the aircraft to be inspected relative to the drone system. The gimbal is controlled to drive the high-dynamic optical sensor to achieve full coverage of the aircraft fuselage to be inspected, and detect aircraft surface defects, thereby realizing aircraft surface defect detection.

[0005] The drone visual inspection technology for aircraft surface defects uses a four-rotor drone equipped with a visual sensor to inspect the aircraft surface, realizing the automatic identification and early warning of target positioning and defects on the aircraft surface. It can greatly improve the efficiency and quality of ground maintenance of large aircraft, drones, etc., drive and promote the development of engineering technologies such as materials, manufacturing, and information, accelerate the diversified development of aviation science and technology, and play a certain leading role in improving innovation capabilities.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for visually detecting aircraft surface defects based on drone inspection includes the following steps:

[0008] Step 11: Build a detection device to perform patrol inspections on the aircraft to be inspected. The detection device includes a quadrotor drone, a built-in gimbal, and a visual sensor. The visual sensor is used to image the aircraft to be inspected, obtain multi-exposure image data, and determine the posture parameters of the aircraft to be inspected relative to the quadrotor drone by photographing the aircraft to be inspected. A spatial patrol trajectory is established, and accurate matching and positioning of the aircraft target is achieved. The visual sensor performs multi-perspective imaging of the aircraft's fuselage to achieve rapid and high-precision detection and early warning of aircraft surface defects.

[0009] Step 12: Based on the method of merging multi-exposure image data to generate a high dynamic range image, multiple single-exposure images of the same scene are captured, and a high-definition image of the entire scene is obtained through image alignment and weighted fusion;

[0010] Step 13: By adding a spatial positioning camera for the aircraft to be detected, a key point-based aircraft pose estimation algorithm is used to achieve preliminary positioning of the aircraft relative to the carrier platform on the apron; the key points are key points where the aircraft body topology structure is obvious and changes little with viewing angle;

[0011] Step 14: Import the 3D model data of the aircraft to be inspected, optimize the flight envelope and trajectory of the quadcopter, and plan the spatial measurement station position of the quadcopter to be inspected; align the surface image data of the aircraft to be inspected each time with the 3D model data of the aircraft to be inspected to achieve accurate and rapid positioning of the surface target of the aircraft;

[0012] Step 15: Collect a large number of images of normal aircraft surfaces, use the autoencoder model to detect image anomalies, and determine the abnormality of the current image by judging the pixel-level reconstruction error between the input image and the output image.

[0013] Furthermore, the step 12 includes:

[0014] The camera response function is expressed as a polynomial:

[0015] (1)

[0016] in, is the degree of the polynomial, is a polynomial The coefficient of the second term, is the radiosity function of the camera detector, Indicates the exposure time, Represents the response function of the transformation from image pixel value to scene radiance, Represents the image position; by estimating the polynomial degree and coefficients Uniquely determine the camera response function;

[0017] Arrange the surface images of the aircraft to be detected in order of their exposure time from short to long, satisfying ;

[0018] in, , Indicates the double exposure time;

[0019] The camera response function is recovered by minimizing the following error function:

[0020] (2)

[0021] in, is the ratio of the radiometric values ​​of images with different exposures, , by minimizing the error function, the coefficients of the polynomial are obtained .

[0022] Furthermore, the step 13 includes:

[0023] Aircraft pose estimation and rapid surface target positioning of the aircraft to be detected are divided into: preliminary aircraft pose estimation, optimal flight path planning, precise alignment of aircraft local areas, and rapid positioning of aircraft targets;

[0024] The initial estimation of the aircraft's posture is done by using the visual sensor in the quadcopter's built-in gimbal to image the aircraft to be inspected from a distance, detecting the pre-set key points of the aircraft, and using the key frame data established in the early stage to solve the preliminary posture parameters of the aircraft to be inspected stationary on the apron relative to the quadcopter through the PNP method, thereby achieving the preliminary positioning of the aircraft on the apron relative to the carrier platform.

[0025] Furthermore, the step 14 includes:

[0026] Based on the three-dimensional model library of the aircraft to be detected, the spatial orientation distribution of the aircraft to be detected in the drone coordinate system is planned, and then the drone's patrol trajectory and envelope curve are planned according to the three-dimensional information to be detected. The envelope curve and the patrol trajectory are both outside a safe distance from the aircraft to be detected; the quadcopter is controlled to image the aircraft surface using its own visual sensor along the planned trajectory.

[0027] Furthermore, the step 15 includes:

[0028] A large number of normal aircraft surface images are collected to form a data set. The autoencoder model is an anomaly detection network model. During training, the input of the anomaly detection network model is a normal aircraft body image, so that the output of the anomaly detection network model is the same as the input of the anomaly detection network model.

[0029] Pixel-level SSIM loss The calculation formula is:

[0030] (3)

[0031] in, 、 are the average value of the generated image and the average value of the input image respectively; 、 are the standard deviation of the generated image and the standard deviation of the input image respectively; is the covariance of the generated image and the input image; 、 To prevent the constant term in the formula from dividing by 0, 、 Represents the horizontal and vertical coordinates of image pixels.

[0032] Furthermore, in the anomaly detection network model inference stage, the surface image of the aircraft to be detected is used as the input of the anomaly detection network model. After calculation, the anomaly detection network model outputs an image reconstructed by the anomaly detection network model; the pixel-level reconstruction error between the input image and the output image is judged, and the value of the reconstruction error is compared with a pre-set threshold. If the value of the reconstruction error is less than the set threshold, it is determined that there is no anomaly; otherwise, it is considered that there is an anomaly in the image, and the body position where the abnormal image is located is marked in time, and an early warning is issued.

[0033] The beneficial effects of the present invention compared with the prior art are:

[0034] The present invention adopts a mode of visual inspection of aircraft surface defects based on drone inspection, and designs a mode of patrol inspection of the aircraft to be inspected by a four-rotor drone carrying a visual sensor, supplemented by power supply control and automatic recognition devices, which can perform automatic and intelligent detection of aircraft surface defects on the ground, and has the characteristics of simple structure, high efficiency, intelligence, and strong versatility; by combining the visual sensor with the drone, the drone can give full play to its ability to fly and reach in a large space, greatly expanding the detection range of the visual sensor, and supplemented by the intelligent recognition algorithm for aircraft surface defects, it can effectively improve the efficiency and quality of aircraft surface defect detection, and at the same time has automation and intelligence, promoting the leap from "manual inspection" to "intelligent inspection" of aircraft surface defects; according to the technology of generating high dynamic image based on multi-exposure images, by collecting aircraft surface Using a polynomial weighted fusion method to generate high-dynamic range images from image sequences with varying exposure times effectively addresses the challenges of ensuring clear imaging of all parts due to complex field lighting, the large and complex three-dimensional topology of the aircraft, and its material properties. This approach improves image dynamic range and clarity, enhancing the accuracy of aircraft surface defect detection. A keypoint-based aircraft pose estimation method is used to plan the UAV's flight envelope and trajectory, accurately locate targets on the aircraft surface, and optimize the flight envelope and trajectory to determine the spatial location of the UAV's inspection stations. This reduces the complexity of the UAV's flight envelope, inaccurate positioning, and the risk of secondary damage to the aircraft caused by manual operation. Furthermore, the method ensures that the visual system accurately locates the aircraft's inspection area, laying the foundation for accurate aircraft surface defect detection. By collecting a large number of normal aircraft surface images, an unsupervised anomaly detection method using an autoencoder model is implemented. This method determines image anomalies by determining the pixel-level reconstruction error between the input and output images, enabling accurate detection of aircraft surface defects without relying on negative samples. This method is simple, efficient, and versatile, making it suitable for defect detection of large workpieces.

[0035] This invention can be further promoted and used by relevant users at home and abroad, and has important application value and broad market prospects. At the same time, the relevant research results can be expanded to fields such as aerospace and rail transportation to meet their needs for appearance defect detection of large equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for visually detecting aircraft surface defects based on drone inspections according to the present invention;

[0037] Figure 2 Schematic diagram of drone visual inspection for aircraft surface defects;

[0038] Figure 3 Schematic diagram of the unsupervised anomaly detection of aircraft surface defects. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0040] The basic idea of ​​this invention is to design a new mode of patrol inspection of aircraft surface defects using a quadrotor drone carrying a visual sensor; to propose a method of generating high-dynamic images from images of the aircraft surface with different exposure levels; and to propose an unsupervised aircraft surface defect recognition method.

[0041] like Figure 1 As shown, a method for visually detecting aircraft surface defects based on drone inspection of the present invention includes the following steps:

[0042] Step 11: Build an inspection device to conduct a patrol inspection of the aircraft under inspection. The inspection device includes a quadrotor drone, a built-in gimbal, and a visual sensor. The visual sensor is used to image the aircraft under inspection and obtain multi-exposure image data. By photographing the aircraft under inspection, the aircraft's position parameters relative to the quadrotor can be determined, a spatial patrol trajectory can be established, and precise matching and positioning of the aircraft target can be achieved. The visual sensor then performs multi-view imaging of the aircraft's fuselage, enabling rapid and high-precision detection and early warning of aircraft surface defects.

[0043] Step 12: Generate a high dynamic range image based on the multi-exposure image, including:

[0044] The aircraft surface image acquisition camera uses a high-dynamic range (HDR) camera, capable of capturing high-definition images in complex lighting environments. To ensure optimal image quality for all parts of the aircraft, regardless of factors such as outdoor lighting and the aircraft's material, a high-dynamic range (HDR) image of the aircraft's surface is generated by merging multiple exposures.

[0045] The present invention uses a polynomial to represent the camera response function:

[0046] (1)

[0047] in, is the degree of the polynomial, is a polynomial The coefficient of the second term, is the radiosity function of the camera detector, Indicates the exposure time, Represents the response function of the transformation from image pixel value to scene radiance, Represents the image position; by estimating the polynomial degree and coefficients Uniquely determines the camera response function.

[0048] By estimating the polynomial degree and coefficients This polynomial can be uniquely determined.

[0049] Arrange the surface images of the aircraft to be detected in order of their exposure time from short to long, satisfying .

[0050] The camera response function is recovered by minimizing the following error function:

[0051] (2)

[0052] in, is the ratio of the radiometric values ​​of images with different exposures, , by minimizing the error function, the coefficients of the polynomial are obtained .

[0053] Step 13: Aircraft pose estimation based on key points (points where the curvature of the aircraft surface changes significantly and changes little with the viewing angle are considered aircraft key points, such as the nose, wing tip, and tail tip), including:

[0054] Aircraft pose estimation and rapid positioning of the surface target of the aircraft to be detected are divided into: preliminary estimation of aircraft pose, optimal flight path planning, precise alignment of local areas of the aircraft, and rapid positioning of aircraft targets.

[0055] The initial estimation of the aircraft's posture uses the visual sensor in the quadcopter's built-in gimbal to remotely image the aircraft to be inspected, detect pre-set key points on the aircraft, and use the PNP method to solve the initial posture parameters of the aircraft to be inspected relative to the drone while it is stationary on the apron.

[0056] Step 14: Plan the UAV's flight envelope and trajectory, and accurately locate targets on the aircraft's surface, including:

[0057] Based on the three-dimensional model library of the aircraft to be detected, the spatial orientation distribution of the aircraft to be detected in the drone coordinate system can be planned. Then, the drone's patrol trajectory and envelope curve can be planned based on the three-dimensional information to be detected from the surface data of the aircraft to be detected. The envelope curve and the patrol trajectory are both at a safe distance from the aircraft to be detected, ensuring that the quadcopter drone will not collide with the aircraft to be detected and cause secondary damage. The present invention controls the quadcopter drone to image the aircraft surface using its own visual sensor under the planned trajectory. This can ensure the accuracy of the aircraft surface area captured by the current high-dynamic camera, and accurately image the local area of ​​the aircraft, achieving rapid and accurate positioning of targets on the aircraft surface, laying the foundation for subsequent surface area recognition of the aircraft body.

[0058] Step 15: Perform unsupervised surface defect detection on the aircraft body, including:

[0059] The autoencoder model is used to detect anomalies in the image for the aircraft surface defect recognition. The autoencoder model is an anomaly detection network model. A large number of normal aircraft surface images are collected in the early stage to form a data set. This work only needs to be collected once for a type of aircraft, and the data set is universal. The anomaly detection network model structure is as follows: Figure 3 As shown in the figure, the input of the anomaly detection network model during training is a normal body picture, so that the output of the anomaly detection network model is as similar as possible to the input of the anomaly detection network model, and indicators such as pixel-level SSIM loss are used.

[0060] During the model inference phase, an image of the aircraft's surface is used as input. After computation, the model outputs a reconstructed image. The system determines the pixel-level reconstruction error between the input and output images and compares the error with a pre-set threshold. If the error is less than the threshold, the system determines that no anomaly exists. Otherwise, the image is considered to contain an anomaly, and the location of the aircraft where the anomaly is located is promptly marked, triggering an alert.

[0061] Pixel-level SSIM loss The calculation formula is:

[0062] (3)

[0063] in, 、 are the average value of the generated image and the average value of the input image respectively; 、 are the standard deviation of the generated image and the standard deviation of the input image respectively; is the covariance of the generated image and the input image; 、 To prevent the constant term in the formula from dividing by 0, 、 Represents the horizontal and vertical coordinates of image pixels.

[0064] like Figure 2 As shown in the figure, the aircraft to be inspected is parked on the apron, and the carrier platform brings in the drone. The drone first determines the posture parameters of the aircraft to be inspected, that is, completes the initial positioning of the aircraft to be inspected, and then plans the drone's patrol trajectory and flight envelope. The drone collects data from various parts of the aircraft to be inspected and performs surface defect detection on the aircraft. The entire inspection work is completed within the inspection space envelope.

[0065] like Figure 3 As shown in the figure, the surface image of the aircraft to be inspected collected by the drone is input, and the image features are extracted and reconstructed through the convolutional network. Finally, the reconstructed image is obtained. By calculating the difference between the abnormal image and the reconstructed image, the abnormal area in the image is formed, and the detection and positioning of aircraft surface defects are completed.

Claims

1. A method for visual inspection of aircraft surface defects based on drone inspection, characterized in that: The steps include: Step 11: Build a detection device to perform patrol inspections on the aircraft to be inspected. The detection device includes a quadrotor drone, a built-in gimbal, and a visual sensor. The visual sensor is used to image the aircraft to be inspected, obtain multi-exposure image data, and determine the posture parameters of the aircraft to be inspected relative to the quadrotor drone by photographing the aircraft to be inspected. A spatial patrol trajectory is established, and accurate matching and positioning of the aircraft target is achieved. The visual sensor performs multi-perspective imaging of the aircraft's fuselage to achieve rapid and high-precision detection and early warning of aircraft surface defects. Step 12: Based on the method of merging multi-exposure image data to generate a high dynamic range image, multiple single-exposure images of the same scene are captured, and a high-definition image of the entire scene is obtained through image alignment and weighted fusion; Step 13: By adding a spatial positioning camera for the aircraft to be detected, a key point-based aircraft pose estimation algorithm is used to achieve preliminary positioning of the aircraft relative to the carrier platform on the apron; the key points are key points where the aircraft body topology structure is obvious and changes little with viewing angle; Step 14: Import the 3D model data of the aircraft to be inspected, optimize the flight envelope and trajectory of the quadcopter, and plan the spatial measurement station position of the quadcopter to be inspected; align the surface image data of the aircraft to be inspected each time with the 3D model data of the aircraft to be inspected to achieve accurate and rapid positioning of the surface target of the aircraft; Step 15: Collect a large number of images of normal aircraft surfaces, use the autoencoder model to detect image anomalies, and determine the abnormality of the current image by judging the pixel-level reconstruction error between the input image and the output image.

2. The method for visually detecting aircraft surface defects based on drone inspection according to claim 1, characterized in that: The step 12 includes: The camera response function is expressed as a polynomial: (1) in, is the degree of the polynomial, is a polynomial The coefficient of the second term, is the radiosity function of the camera detector, Indicates the exposure time, Represents the response function of the transformation from image pixel value to scene radiance, Represents the image position; by estimating the polynomial degree and coefficients Uniquely determine the camera response function; Arrange the surface images of the aircraft to be detected in order of their exposure time from short to long, satisfying ; in, , Indicates the double exposure time; The camera response function is recovered by minimizing the following error function: (2) in, is the ratio of the radiometric values ​​of images with different exposures, , by minimizing the error function, the coefficients of the polynomial are obtained .

3. The method for visually detecting aircraft surface defects based on drone inspection according to claim 1, characterized in that: The step 13 comprises: Aircraft pose estimation and rapid surface target positioning of the aircraft to be detected are divided into: preliminary aircraft pose estimation, optimal flight path planning, precise alignment of aircraft local areas, and rapid positioning of aircraft targets; The initial estimation of the aircraft's posture is done by using the visual sensor in the quadcopter's built-in gimbal to image the aircraft to be inspected from a distance, detecting the pre-set key points of the aircraft, and using the key frame data established in the early stage to solve the preliminary posture parameters of the aircraft to be inspected stationary on the apron relative to the quadcopter through the PNP method, thereby achieving the preliminary positioning of the aircraft on the apron relative to the carrier platform.

4. The method for visually detecting aircraft surface defects based on drone inspection according to claim 1, characterized in that: The step 14 comprises: Based on the three-dimensional model library of the aircraft to be detected, the spatial orientation distribution of the aircraft to be detected in the drone coordinate system is planned, and then the drone's patrol trajectory and envelope curve are planned according to the three-dimensional information to be detected. The envelope curve and the patrol trajectory are both outside a safe distance from the aircraft to be detected; the quadcopter is controlled to image the aircraft surface using its own visual sensor along the planned trajectory.

5. The method for visually detecting aircraft surface defects based on drone inspection according to claim 1, characterized in that: The step 15 comprises: A large number of normal aircraft surface images are collected to form a data set. The autoencoder model is an anomaly detection network model. During training, the input of the anomaly detection network model is a normal aircraft body image, so that the output of the anomaly detection network model is the same as the input of the anomaly detection network model. Pixel-level SSIM loss The calculation formula is: (3) in, 、 are the average value of the generated image and the average value of the input image respectively; 、 are the standard deviation of the generated image and the standard deviation of the input image respectively; is the covariance of the generated image and the input image; 、 To prevent the constant term in the formula from dividing by 0, 、 Represents the horizontal and vertical coordinates of image pixels.

6. The method for visually detecting aircraft surface defects based on drone inspection according to claim 5, characterized in that: In the anomaly detection network model inference stage, the surface image of the aircraft to be detected is used as the input of the anomaly detection network model. After calculation, the anomaly detection network model outputs an image reconstructed by the anomaly detection network model; the pixel-level reconstruction error between the input image and the output image is judged, and the value of the reconstruction error is compared with a pre-set threshold. If the value of the reconstruction error is less than the set threshold, it is determined that there is no anomaly; otherwise, it is considered that there is an anomaly in the image, and the body position where the abnormal image is located is marked in time, and an early warning is issued.

Images