An aircraft appearance defect detection method and system based on deep learning

Abstract

This invention proposes a deep learning-based method and system for detecting aircraft exterior defects. By introducing deformable convolution and global attention mechanisms into the feature extraction module, it can dynamically adapt to defect features of different shapes on the aircraft exterior and pay more attention to feature information of defect regions. It also enhances the feature representation of small targets by fusing contextual information from different receptive fields of extracted features at different scales within the extracted features. Furthermore, it uses the EMA-Slide function to assign weights to samples near unclear classification boundaries, improving its focus on difficult-to-identify samples.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method and system for detecting aircraft appearance defects based on deep learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The external surface structure and components of an aircraft are crucial for its safe, stable, and efficient operation. However, with increasing service life and the influence of flight environment factors or human factors, the external surface structure and components of an aircraft are prone to damage and defects such as cracks and corrosion. These damages are directly related to the structural integrity of the aircraft; severe damage can not only threaten flight safety but also potentially lead to catastrophic consequences. Therefore, regularly conducting external structural inspections of aircraft is an important means of maintaining aircraft safety and an essential step in ensuring stable aircraft operation.

[0004] Currently, most aircraft exterior defect inspections are still conducted manually. However, due to the structural limitations of the aircraft itself, this method not only has a wide inspection range, low efficiency, poor reliability, and risks associated with high-altitude operations, but also relies heavily on human experience, requiring a high level of professional competence from inspectors. With the rapid development of artificial intelligence, deep learning is becoming increasingly popular in various industrial scenarios due to its real-time, non-contact, high-precision, and intelligent characteristics. However, its application in the field of aircraft exterior defect inspection remains insufficient.

[0005] Currently, the main problems in aircraft exterior defect detection are as follows:

[0006] 1. Due to the large variation in size and irregular shape of defects in aircraft appearance, it is difficult to extract features;

[0007] 2. The problem of small and difficult-to-detect aircraft exterior defects during long-distance inspection;

[0008] 3. The dataset of aircraft appearance defects has an imbalance problem. Summary of the Invention

[0009] To overcome the shortcomings of the existing technologies, this invention provides a deep learning-based method and system for detecting aircraft exterior defects. By introducing deformable convolution and global attention mechanisms into the feature extraction module, the feature extraction capability of the model is enhanced. Contextual information from different receptive fields of features at different scales is obtained and fused, which enhances the feature representation of small targets. The EMA-Slide function is introduced to improve attention to difficult-to-identify samples.

[0010] To achieve the above objectives, a first aspect of the present invention provides a deep learning-based method for detecting aircraft appearance defects, comprising:

[0011] Acquire an image of the aircraft exterior to be inspected;

[0012] Using a trained detection model, detection results are obtained based on the acquired aircraft exterior images;

[0013] The trained detection model includes a feature extraction network, a neck network, and a detection head; the feature extraction network incorporates deformable convolution and a global attention mechanism.

[0014] The feature extraction network is used to extract different scale features of different defect shapes in aircraft exterior images;

[0015] Using the neck network, contextual information of different receptive fields for features at different scales is obtained, and contextual information of different receptive fields for features at the same scale is aggregated to obtain a feature-enhanced representation;

[0016] Using the detection head, the defect detection results of the aircraft appearance are obtained based on the feature enhancement representation;

[0017] During the training of the detection model, the EMA-Slide function is used to focus on training samples with unclear classification boundaries to obtain corresponding sample weight coefficients. The loss function of the detection model is then improved based on the obtained sample weight coefficients.

[0018] A second aspect of the present invention provides a deep learning-based aircraft appearance defect detection system, comprising:

[0019] Acquisition module: Acquires images of the aircraft's exterior to be inspected;

[0020] Detection module: Using a trained detection model, the module obtains detection results based on the acquired aircraft exterior images.

[0021] The trained detection model includes a feature extraction network, a neck network, and a detection head; the feature extraction network incorporates deformable convolution and a global attention mechanism.

[0022] The feature extraction network is used to extract different scale features of different defect shapes in aircraft exterior images;

[0023] Using the neck network, contextual information of different receptive fields for features at different scales is obtained, and contextual information of different receptive fields for features at the same scale is aggregated to obtain a feature-enhanced representation;

[0024] Using the detection head, the defect detection results of the aircraft appearance are obtained based on the feature enhancement representation;

[0025] During the training of the detection model, the EMA-Slide function is used to focus on training samples with unclear classification boundaries to obtain corresponding sample weight coefficients. The loss function of the detection model is then improved based on the obtained sample weight coefficients.

[0026] A third aspect of the present invention provides a computer device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, a deep learning-based method for detecting aircraft appearance defects is performed.

[0027] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs a deep learning-based method for detecting aircraft appearance defects.

[0028] The above one or more technical solutions have the following beneficial effects:

[0029] In this invention, by introducing deformable convolution and global attention mechanisms into the feature extraction module, it is possible to dynamically adapt to the defect features of different shapes on the aircraft appearance and pay more attention to the feature information of the defect region; by obtaining the contextual information of different receptive fields of different scale features of the extracted features and fusing them, the feature representation of small targets can be enhanced.

[0030] In this invention, the EMA-Slide function is used to assign weights to samples near the unclear classification boundary, thereby increasing attention to difficult samples that are hard to identify.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0033] Figure 1 This is a diagram of the ASD-YOLO network structure in Embodiment 1 of the present invention;

[0034] Figure 2 This is a structural diagram of the DCNC3 module in Embodiment 1 of the present invention;

[0035] Figure 3 This is a diagram of the GAM network structure in Embodiment 1 of the present invention;

[0036] Figure 4 This is a structural diagram of the CEM module in Embodiment 1 of the present invention.

[0037] Figures 5(a)-5(d) The images show example diagrams of ASD-YOLO detection results at different locations in Embodiment 1 of the present invention. Detailed Implementation

[0038] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0040] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0041] Example 1

[0042] This embodiment discloses a deep learning-based method for detecting aircraft appearance defects, including:

[0043] Acquire an image of the aircraft exterior to be inspected;

[0044] Using a trained detection model, detection results are obtained based on the acquired aircraft exterior images;

[0045] The trained detection model includes a feature extraction network, a neck network, and a detection head; the feature extraction network incorporates deformable convolution and a global attention mechanism.

[0046] The feature extraction network is used to extract different scale features of different defect shapes in aircraft exterior images;

[0047] Using the neck network, contextual information of different receptive fields for features at different scales is obtained, and contextual information of different receptive fields for features at the same scale is aggregated to obtain a feature-enhanced representation;

[0048] Using the detection head, the defect detection results of the aircraft appearance are obtained based on the feature enhancement representation;

[0049] During the training of the detection model, the EMA-Slide function is used to focus on training samples with unclear classification boundaries to obtain corresponding sample weight coefficients. The loss function of the detection model is then improved based on the obtained sample weight coefficients.

[0050] In this embodiment, the detection model, namely the ASD-YOLO network, is used to detect defects in the aircraft's exterior. Figure 1 As shown, the ASD-YOLO network uses Yolov5 as its backbone and mainly consists of three parts: backbone, neck, and head. The input RGB image first obtains multi-scale feature information through the backbone, then the features are further fused and enhanced in the neck, and finally the obtained three-scale feature maps are used for multi-scale target detection in the head.

[0051] To address the challenge of extracting features from aircraft exterior defects due to their large scale variations and irregular shapes, an improved deformable convolutional module (DCNC3) and a global attention module (GAM) are integrated into the backbone feature extraction network. The improved DCNC3 module dynamically adapts to defect features of different shapes, while the GAM mechanism focuses more on the feature information of the defect region. Together, they enhance the model's feature extraction capability.

[0052] Specifically, to enable the network to adapt to geometric deformations, this embodiment replaces the traditional C3 module in the backbone network with a deformable convolutional module (DCNC3). This aims to improve the network's accuracy in detecting objects with greater shape changes or diverse poses. The deformable convolutional module (DCNC3) can adaptively learn the shape of different convolutional kernels based on the input data. Essentially, it learns the offset from the previous feature map through an additional convolutional layer, and then adds the 2D offset to the regular grid sampling positions in the standard convolution, allowing the sampling grid to deform freely. At the same time, feature gain coefficients are added to control the effective receptive field.

[0053] The DCNC3 module's calculation process can be described as follows: First, a standard 3x3 convolution is performed on the input feature layer, with 3N channels, where N is the number of channels in the original input feature map. 2N channels are used for N 2D (x and y directions) offsets to obtain the offset domain, and the offset domain feature map has the same spatial resolution as the input feature map. The outputs of the remaining N channels are mapped to values ​​in the range [0,1] through a sigmoid layer to obtain an adjustable gain coefficient. After determining the deformed sampling point, the final value of that point on the output feature map is obtained by multiplying it by the corresponding weight and feature gain coefficient.

[0054] The backbone network of the existing Yolov5 network includes a first convolutional block, a second convolutional block, a first C3 module, a third convolutional block, a second C3 module, a fourth convolutional block, a third C3 module, a fifth convolutional block, a fourth C3 module, and an SPPF module connected in sequence.

[0055] Specifically, the second, third, and fourth C3 modules in the backbone network of the original Yolov5 network are improved and replaced with DCNC3 modules. The DCNC3 module replaces the BottleNeck block in the C3 module with a residual structure formed by sequentially connected ConvBNSiLU and DCNBNSiLU.

[0056] To enable the network to focus more on the features of the damaged target while suppressing irrelevant background information, this example introduces a global attention mechanism (GAM) to improve the feature extraction capability of the backbone network. Compared to previous attention methods that ignore the importance of spatial and channel information in enhancing cross-dimensional interactions, GAM transcends the spatial and channel dimensions, capturing important features across all three dimensions. It preserves both channel and spatial information while amplifying the cross-dimensional interactions of the global spatial channels, significantly improving network performance. The GAM structure is as follows: Figure 3 As shown.

[0057] Specifically, a global attention mechanism (GAM) is introduced before the SPPF module input in the backbone network of the original Yolov5 network to enable greater attention to the feature information of the defect region.

[0058] The original Yolov5 network struggles to detect minute damage on aircraft surfaces. As the network deepens and the receptive field increases, some information from the original feature maps is lost or merged during multiple downsampling processes, and the spatial resolution of the feature maps gradually decreases, which is particularly disadvantageous for small targets. To improve the network's ability to detect small targets, a CEM module is used to aggregate contextual information from feature maps at different scales to enhance feature representation. Simultaneously considering both local and global information further enhances the feature representation capabilities for small targets.

[0059] The existing YOLOv5 network's neck network includes, in sequence, a sixth convolutional block, a first upsampling layer, a first cross-layer stitching layer, a fifth C3 module, a seventh convolutional block, a second upsampling layer, a second cross-layer stitching layer, a sixth C3 module, and an eighth convolutional block; and, in sequence, a first CEM module, a third cross-layer stitching layer, a seventh C3 module, and a ninth convolutional block; and, in sequence, a second CEM module, a fourth cross-layer stitching layer, an eighth C3 module, and a tenth convolutional block; the output of the sixth C3 module is connected in sequence to the eleventh convolutional block and the fourth cross-layer stitching layer; and the output of the eighth C3 module is connected in sequence to the twelfth convolutional block and the third cross-layer stitching layer.

[0060] After the outputs of the second and third DCNC3 modules, connect them to the third CEM module and the fourth CEM module respectively; connect the outputs of the third CEM module and the fourth CEM module to the first cross-layer splicing layer and the second cross-layer splicing layer respectively.

[0061] like Figure 4 As shown, it uses dilated convolutions with different dilation rates to obtain contextual information for different receptive fields. Specifically, the CEM module includes parallel 3×3 convolutional layers with rate=1, rate=3, and rate=5. After each of the 3×3 convolutional layers with rate=1, rate=3, and rate=5, a 1×1 convolutional layer is connected, and then the outputs of the three 1×1 convolutional layers are summed.

[0062] Given that aircraft damage images involve multiple damage types, with significant differences in sample size across categories and varying levels of sample difficulty within each category, this imbalance greatly impacts the performance of the training model. Therefore, this embodiment proposes a sliding weight function EMA-Slide based on the exponential moving average concept. The EMA-Slide function yields loss weight coefficients for different samples, which are then applied to the binary cross-entropy (BCE) loss used to calculate confidence and classification. This effectively focuses on difficult samples, thus mitigating the sample imbalance problem.

[0063] The specific calculation process is as follows:

[0064]

[0065] L ESL =L BCE *ES(x,μ)

[0066] Among them, L BCE Represents the cross-entropy loss function; L represents the predicted probability of each sample; y represents the true label of the sample, which is 1 if it belongs to a certain category, and 0 otherwise; ESL Let represent the new EMA-SlideLoss loss function; ES(x,μ) represents the sample loss weight coefficients obtained through the EMA-Slide function, calculated as follows:

[0067] EMA-Slide introduces an automatically updated IoU mean μ based on an exponential moving average as a threshold to determine the difficulty of a sample, thereby assigning different loss weights to different samples. The rule is that samples greater than μ are considered positive samples, and samples less than μ are considered difficult to identify. Generally, samples far below μ are considered unlikely to have a significant impact through optimization, while samples near μ may have ambiguous boundaries and can be improved through further identification and optimization. To more fully utilize these samples to optimize the network's learning ability, this embodiment designs the EMA-Slide function to smoothly assign larger weights to samples near μ, increasing the relative loss of easily confused difficult samples, making the network pay more attention to these samples, and simultaneously making the network more focused on the recent IoU data trend under the current training, allowing the model to be adjusted based on the latest optimization results in each iteration. The EMA-Slide function formula is as follows:

[0068]

[0069] In the formula, x represents the IoU value of the current predicted sample; μ is the mean IoU of all samples, which is automatically updated with a new value using an exponential shift method each time it is called. The update calculation formula is as follows:

[0070]

[0071] In the formula, μ n This is the current mean IoU of all samples, passed as a parameter each time the EMA-Slide function is called, and the updated value is automatically calculated; μ n-1 β0 is the mean IoU after the last update; β0 is the initial exponential moving weight decay rate; β is the exponential moving weight decay coefficient, controlling the weight of the current mean IoU; n represents the number of loss calculation updates; τ is a time constant used to control the decay rate. The aircraft appearance detection method proposed in this embodiment was compared with the original algorithm and other algorithms on the public dataset in Roboflow. The quantitative comparison results of different algorithms for five types of aircraft appearance defects are shown in Table 1. The detection visualization of the proposed method is shown in Table 1. Figures 5(a)-5(d) As shown.

[0072] Table 1 Comparison of detection results of different algorithms

[0073]

[0074] The detection results show that the trained model can detect five types of aircraft appearance defects, including cracks, missing screws, and paint peeling. Compared with current mainstream methods, the method proposed in this embodiment has certain advantages, and its detection effect is more efficient and accurate. Moreover, the dataset used in this embodiment has an image perspective that is close to that of an elevated viewpoint. Therefore, the method can be well applied and deployed in UAV-based aircraft appearance inspection. Real-time verification was conducted on the DJI M30 UAV platform, confirming the applicability and generalization of this method for UAV inspection scenarios of aircraft appearance defects.

[0075] In summary, the deep learning-based aircraft exterior inspection method proposed in this embodiment revolutionizes traditional manual inspection methods, enabling a more efficient aircraft exterior defect detection mode and adapting to complex and ever-changing exterior scenarios, thus possessing a certain degree of versatility. By improving the intelligent inspection and control methods for exterior inspection, it will replace and liberate maintenance personnel from related tasks, promote the use of new technologies to assist in maintenance inspection management, thereby improving maintenance support safety, efficiency, and reliability, and driving the intelligent development of the aircraft maintenance support field.

[0076] Example 2

[0077] The purpose of this embodiment is to provide a deep learning-based aircraft appearance defect detection system, including:

[0078] Acquisition module: Acquires images of the aircraft's exterior to be inspected;

[0079] Detection module: Using a trained detection model, the module obtains detection results based on the acquired aircraft exterior images.

[0080] The trained detection model includes a feature extraction network, a neck network, and a detection head; the feature extraction network incorporates deformable convolution and a global attention mechanism.

[0081] The feature extraction network is used to extract different scale features of different defect shapes in aircraft exterior images;

[0082] Using the neck network, contextual information of different receptive fields of features at different scales is obtained, and the contextual information of different receptive fields of features at different scales is aggregated to obtain feature-enhanced representations;

[0083] Using the detection head, defect detection results are obtained from the aircraft exterior image based on the feature enhancement representation;

[0084] During the training of the detection model, the EMA-Slide function is used to focus on training samples with unclear classification boundaries to obtain corresponding sample weight coefficients. The loss function of the detection model is then improved based on the obtained sample weight coefficients.

[0085] Example 3

[0086] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0087] Example 4

[0088] The purpose of this embodiment is to provide a computer-readable storage medium.

[0089] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0090] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0091] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0092] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A deep learning-based method for detecting aircraft exterior defects, characterized in that, include: Acquire an image of the aircraft exterior to be inspected; Using a trained detection model, detection results are obtained based on the acquired aircraft exterior images; The trained detection model includes a feature extraction network, a neck network, and a detection head. The feature extraction network incorporates deformable convolution and a global attention mechanism. The detection model uses a Yolov5 network as its backbone, replacing the second, third, and fourth C3 modules in the feature extraction network with deformable convolution. Specifically, the deformable convolution replaces the BottleNeck block in the C3 module with a residual structure composed of sequentially connected ConvBNSiLU and DCNBNSiLU. The additional convolutional layers in the deformable convolution learn offsets from the feature map, and the learned offsets are added to the regular grid sampling positions in the standard convolution, allowing the network to automatically deform and adapt to defect features of different shapes. The feature extraction network is used to extract different scale features of different defect shapes in the aircraft appearance image; a global attention mechanism is introduced before the SPPF module in the feature extraction network to pay more attention to the feature information of the defect area during feature extraction. For the features of different scales extracted from the neck network, the CEM module is used to obtain the context information of different receptive fields of different scale features. The context information of different receptive fields of the same scale feature is aggregated to obtain the feature enhancement representation. For features of different scales extracted by the feature extraction network, the CEM module is used to obtain contextual information of different receptive fields of features of different scales. The contextual information of different receptive fields of features of the same scale is aggregated to obtain an enhanced representation. The enhanced representation is further fused with the features after upsampling in the neck network. The CEM module uses dilated convolutions with different dilation rates to obtain contextual information of different receptive fields. Using the detection head, defect detection results are obtained from the aircraft exterior image based on the feature enhancement representation; During the training of the detection model, the EMA-Slide function is used to focus on training samples with unclear classification boundaries to obtain corresponding sample weight coefficients. The loss function of the detection model is improved based on the obtained sample weight coefficients. The average IoU of all bounding boxes is updated and calculated using an exponential moving method and used as a threshold. The EMA-Slide function is used to assign greater weights to training samples near the threshold.

2. A deep learning-based aircraft appearance defect detection system, implementing the deep learning-based aircraft appearance defect detection method as described in claim 1, characterized in that, include: Acquisition module: Acquires images of the aircraft's exterior to be inspected; Detection module: Using a trained detection model, the module obtains detection results based on the acquired aircraft exterior images. The trained detection model includes a feature extraction network, a neck network, and a detection head; the feature extraction network incorporates deformable convolution and a global attention mechanism. The feature extraction network is used to extract different scale features of different defect shapes in aircraft exterior images; Using the neck network, contextual information of different receptive fields for features at different scales is obtained, and contextual information of different receptive fields for features at the same scale is aggregated to obtain a feature-enhanced representation; Using the detection head, defect detection results are obtained from the aircraft exterior image based on the feature enhancement representation; During the training process of the detection model, the EMA-Slide function is used to focus on training samples with unclear classification boundaries to obtain corresponding sample weight coefficients. The loss function of the detection model is then improved based on the obtained sample weight coefficients.

3. A computer device, characterized in that, include: The system includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the deep learning-based aircraft appearance defect detection method as described in claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is executed by a processor as described in claim 1, a method for detecting aircraft appearance defects based on deep learning.

Images