Aviation transparent part defect collection and detection method based on mixed illumination

Abstract

The invention discloses an aviation transparent part defect collection and detection method based on mixed illumination, and the method comprises the steps: sequentially employing forward illumination and backlight illumination for the same defect region of an aviation transparent part, i.e., when one illumination is carried out, turning off the other illumination, and collecting an image pair; marking and enhancing data of the obtained images of different defect areas, constructing an aviation transparent part defect detection bimodal image data set which covers defect types including contusion, scratches, cracks and point defects, and strictly aligning two modal images in the data set in space; a defect detection model is constructed, the aviation transparent part defect detection bimodal image data set is used for training, and defect detection is carried out based on an image pair; according to the invention, forward illumination and backlight illumination are adopted to shoot the same defect position, so that more accurate defect detection can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial defect detection, and in particular relates to a method for collecting and detecting defects of aviation transparent parts based on mixed illumination. Background Art

[0002] Aviation transparent parts refer to transparent materials used in aircraft canopies, windshields, portholes and other parts. Due to the objective complexity of the production process and long-term flight wear, aviation transparent parts will inevitably have defects on their surfaces, such as cracks and scratches, which directly affect the pilot's field of vision and may even pose a structural safety hazard. How to accurately and efficiently perform automated detection of surface defects in aviation transparent parts is an urgent problem to be solved in production quality control and aircraft maintenance.

[0003] At present, the surface quality inspection method of aviation transparent parts still mainly relies on manual inspection, which depends on the experience of testers and has problems such as low efficiency, easy missed inspection, lack of consistency, etc. Deep learning has certain advantages in recognition and detection. However, deep learning requires a large amount of sample data to complete training. However, for aviation transparent parts, a single lighting shooting condition is generally used, which cannot fully express defect information and is prone to missed inspection and misjudgment. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an aviation transparent part defect collection and detection method based on mixed lighting, which uses forward lighting and backlighting to shoot the same defect position, thereby achieving more accurate defect detection.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for collecting defects of aviation transparent parts based on mixed illumination is characterized in that, for the same defect area of ​​the aviation transparent part, forward illumination and backlighting are used in sequence, that is, when one illumination is used, the other illumination is turned off, and an image pair is collected. The image pair data of different defect areas obtained are annotated and enhanced to construct a dual-modal image dataset for defect detection of aviation transparent parts, which covers defect types including bruises, scratches, cracks and point defects. The two modal images in the dataset are strictly aligned in space.

[0007] In one embodiment, the forward lighting is bright field forward lighting, using a ring light source to achieve uniform lighting to reduce reflections; the backlighting uses a surface light source luminous panel as a light source.

[0008] In one embodiment, the aviation transparent member is fixedly arranged, an electronic digital microscope is used as the image acquisition device, and a carrying device is used to carry the image acquisition device and the annular light source to meet the stability required for acquisition; wherein:

[0009] The carrying device is a universal adjustment arm, and the defect collection is completed by manually adjusting the collection position; or,

[0010] The carrying device is a bidirectional sliding guide rail, which traverses the inspected sample according to the set moving step length. The surface of the surface light source luminous plate is frosted to reduce reflection, and it also serves as a sample carrying platform. During the detection process, after the image acquisition device moves to a sampling position, the forward lighting and backlighting are enabled in turn to capture images of the two modes respectively.

[0011] In one embodiment, the acquired image data of different defect areas are labeled, and the implementation method is as follows:

[0012] First, the Segment Anything (ViT-Large) Model is used to pre-label the selected defects and obtain the polygonal boundary of the defect. Then a script is written to read the JSON file and calculate the minimum enclosing rectangle of the polygon to obtain a more accurate OBB format labeling box.

[0013] Furthermore, in the process of labeling, the defect category is judged according to the performance of the defect under two lighting conditions, and the defects that are obvious only under a single lighting method are labeled and merged into a labeling file after labeling; at the same time, in view of the fact that there are differences in the defect contours under the two lighting methods, the union of the defect areas of the two images is selected, that is, the outer contours are taken after the two pictures are superimposed.

[0014] In one embodiment, the acquired image data of different defect areas are enhanced, and the implementation method includes:

[0015] Using a lossless enhancement method and / or an arbitrary angle rotation enhancement method to perform pre-rotation enhancement;

[0016] Use Mosaic data enhancement method to perform data enhancement during training;

[0017] Among them, since the OBB standard format does not allow the coordinates of the annotation points to be located outside the image, it is necessary to normalize the overflow labels. The correction method for the overflow labels is as follows:

[0018] 1), if the four points of the rectangular box are not in the field of view, delete the target (end), otherwise go to 2);

[0019] 2) If all four points of the rectangular frame are in the field of view, no adjustment is required (end), otherwise go to 3);

[0020] 3) Select a point that is not in the field of view, obtain the long and short sides of the rectangle where the point is located, translate the short side along the direction of the long side (the calculation of the translation amount is to solve a linear programming problem) so that the point falls on the boundary of the image, and go to 2).

[0021] Another aspect of the present invention further provides a defect detection method based on the aviation transparent part defect collection method based on mixed illumination, comprising the following steps:

[0022] Step 1, constructing a defect detection model, wherein the defect detection model grayscales the forward illumination image and the backlight illumination image of an image pair respectively as the red channel and the green channel of the fused image, and takes the blue channel as the mean of the two to obtain a composite image, and performs defect detection based on the composite image;

[0023] Step 2, using the aviation transparent part defect detection dual-modal image dataset to train the defect detection model;

[0024] Step 3: Input the collected forward illumination image and backlight illumination image into the trained defect detection model to obtain its defect detection results.

[0025] Another aspect of the present invention provides another defect detection method for an aviation transparent component defect collection method based on mixed illumination, comprising the following steps:

[0026] Step 1, construct an attention-based dual-modal fusion defect detection model; the model includes a feature extraction backbone, a feature fusion module (AMFF), a spatial pyramid pooling module (FSPPF) and a detection layer;

[0027] The feature extraction trunk has two parallel trunks, each taking the forward illumination image and the backlight illumination image of an image pair as input, and extracting the multi-scale features of the forward illumination mode and the backlight illumination mode; the feature fusion module is inserted between the two feature extraction trunks to obtain fusion features of different receptive fields; the spatial pyramid pooling module takes the outputs of the two feature extraction trunks as input to obtain the final output fusion feature map; the detection layer takes the outputs of the feature fusion module and the spatial pyramid pooling module as input to obtain the detection result;

[0028] Step 2, using the aviation transparent part defect detection dual-modal image dataset to train the defect detection model;

[0029] Step 3: Input the collected forward illumination image and backlight illumination image into the trained defect detection model to obtain its defect detection results.

[0030] In one embodiment, the feature extraction backbone is a CSPDarknet-like feature extraction backbone in a YOLOv8 network; there are two feature fusion modules, which are respectively inserted into the P3 layer and the P4 layer of two CSPDarknet-like feature extraction backbones; the P5 layers of the two CSPDarknet-like feature extraction backbones are connected to the spatial pyramid pooling module.

[0031] In one embodiment, the feature fusion module is composed of a Concat module and a CBAM module; the feature fusion process is as follows:

[0032] Firstly, the Concat module is used to concatenate the feature maps of the two modalities, and the preliminary fusion features are obtained based on channel superposition. Secondly, the preliminary fusion features are sent to the channel attention module CAM, and the channel weights between the two modalities and within the modalities are adaptively learned through the channel attention mechanism. The weight distribution of the two-way features is adaptively adjusted by calculating the contribution rate of the features of different modalities to the detection. Finally, the spatial attention module SAM is applied to improve the attention to the defect target area, improve the expression ability of the fusion features, and obtain the final improved feature map.

[0033] The spatial pyramid pooling module is composed of a Concat module, an SPPF module and a CBAM module, and the process is as follows:

[0034] Firstly, the Concat module is used to concatenate the outputs of the P5 layer of the two feature extraction backbones. Secondly, the SPPF module is used to perform pooling operations of different scales on the concatenated results to enhance the detection capability of targets of different scales. Finally, the CBAM module is used to optimize the feature map to obtain the final output fusion feature map.

[0035] Compared with the prior art, the present invention uses forward lighting and backlighting to collect defect images. The present invention collects and constructs data sets based on real aviation transparent parts samples, and designs an attention-based dual-modal fusion defect detection model. Experiments show that the dual-modal detection method improves mAP by 5.6% to 98.4% compared with the single-modal detection method that only uses forward lighting images, and the recognition accuracy and stability are greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The following are the lighting methods used in industrial machine vision defect detection, where a is forward lighting, b is diffuse forward lighting, and c is backlighting.

[0037] Figure 2 These are defect area images collected using three lighting methods according to an embodiment of the present invention.

[0038] Figure 3It is a schematic diagram of the lighting structure of the present invention.

[0039] Figure 4 This is a schematic diagram of four types of defects collected in the present invention, where a is a bruise, b is a scratch, c is a crack, and d is a point defect.

[0040] Figure 5 The figure is a schematic diagram of the process of image acquisition by the automatic control means of the present invention.

[0041] Figure 6 This is a schematic diagram of a 24-fold rotation enhancement sample in an embodiment of the present invention.

[0042] Figure 7 It is a schematic diagram of the RGB channel fusion method of the present invention.

[0043] Figure 8 Schematic diagram of the network architecture of the attention-based dual-modal fusion defect detection model of the present invention.

[0044] Fig. 9 Schematic diagram of the feature fusion module principle of the present invention.

[0045] Fig.10 is the detection result of the RGB channel fusion model, where a is the precision-confidence curve, b is the precision-recall curve, and c is the recall-confidence curve.

[0046] Fig.11 Detection results of the attention-based dual-modal fusion defect detection model of the present invention, a is the precision-confidence curve, b is the precision-recall curve, and c is the recall-confidence curve. DETAILED DESCRIPTION

[0047] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0048] Since the aviation transparent parts dataset required for defect detection in the present invention is key data in a specific field and there is currently no public defect sample data, data collection work is performed on some real aircraft cockpit canopies that are obtained by the present invention and used as a source of subsequent experimental data.

[0049] In industrial machine vision defect detection, appropriate lighting structure design is crucial for the acquisition of defect images. Among them, forward lighting is the most widely used type of lighting. By adjusting the incident angle of light to control whether the light is directly reflected on the camera, forward lighting can be divided into bright field forward lighting and dark field forward lighting. Figure 1 In addition, for highly reflective objects with smooth surfaces, diffuse forward lighting can effectively reduce the strong reflections produced by direct lighting, which is usually based on dome structures or multi-angle lights to achieve uniform lighting, such as Figure 1 As shown in b.

[0050] The counterpart to forward lighting is backlighting, which places the light source behind the object, such as Figure 1 Backlighting is often used for defect detection of transparent materials. It can highlight internal defects but is not sensitive to surface defects.

[0051] In view of the optical properties of aviation transparent samples, the present invention selected three common lighting methods for experiments to determine the lighting structure that meets the detection requirements. The present invention collected some images of defective areas under different lighting conditions. Some of the results are as follows: Figure 2 shown.

[0052] It can be observed that bright field forward lighting can easily capture surface scratch defects, but is easily affected by reflections and uneven light; backlighting can well reflect the depth information of defects, thereby distinguishing surface scratches and bruises, and highlighting deeper point defects that should not be ignored. The disadvantage is that the detailed features of the surface and part of the boundary contour information will be lost. The present invention believes that the information provided by the images of these two types of lighting is complementary to the judgment of the defect situation, and the combination of the two can make more accurate positioning and evaluation of defects.

[0053] In addition, the experiment found that the dark field forward lighting captured similar defect features as the backlighting, which is similar to an inversion operation. Considering that the effect of dark field lighting is sensitive to factors such as lighting angle and ambient light interference, it is difficult to implement in actual industrial inspection environments. The present invention retains a more stable backlighting to extract the depth information of the defect.

[0054] Finally, the present invention chooses to use a dual-mode lighting structure, referring to Figure 3 As shown, forward lighting and backlighting are used to photograph and detect defective areas, respectively. Thus, the present invention realizes the acquisition of aviation transparent parts defects based on mixed illumination, that is, for the same defective area of ​​aviation transparent parts, forward lighting and backlighting are used in turn to acquire an image pair, and then the image pair data of different defective areas are annotated and enhanced to construct a dual-modal image dataset for defect detection of aviation transparent parts, which covers defect types including bruises, scratches, cracks and point defects, and the two modal images in the dataset are strictly aligned in space.

[0055] In terms of the implementation of forward lighting, the present invention selects bright field forward lighting and uses a ring light source to achieve relatively uniform lighting to reduce reflections; and the backlighting uses a surface light source luminous panel as a light source.

[0056] In order to meet the needs of defect collection, the present invention builds a prototype defect collection platform, which is mainly composed of an electronic digital microscope, a universal adjustment arm, a surface light source luminous plate and a ring light source.

[0057] The diameter of defects on the glass canopy is mostly distributed in the range of 80μm to 500μm, which cannot be directly photographed using a conventional CCD camera. Therefore, an electronic digital microscope is selected as the image acquisition device, which has an adjustable focal length and LED light. The focal length is set to 20mm and the magnification is 400×.

[0058] When shooting images of two different lighting methods at the same position, the relative position of the acquisition lens and the sample must be fixed. Taking into account the curved surface properties of the sample, a universal adjustment arm is used as a carrying device to carry the image acquisition device and the ring light source while fixing the aviation transparent parts. The arm length of the device is 60cm and the maximum load-bearing capacity is 1.0kg. It can fully meet the stability required for acquisition and complete the defect acquisition by manually adjusting the acquisition position.

[0059] The forward lighting method uses 8 ring LED lights to provide lighting, which are located around the collection lens. The backlighting method uses a 30*42cm backlight panel (i.e., a surface light source panel) with a brightness of 500nits and a color temperature of 6500K.

[0060] Based on the existing real samples of aviation transparent parts, two lighting methods were used for 235 defect coverage areas, and a total of 470 original images were collected. After image analysis and experience, the collected defects were classified into four categories: contusion, scratches, cracks, and spots, which basically cover all kinds of defects that may appear on the upper cover of the aircraft cockpit glass. Typical specific features include Figure 4 shown.

[0061] Furthermore, the present invention can also introduce automatic control technology, that is, using a bidirectional sliding guide as a bearing device, referring to Figure 5 , traverse the inspected sample according to the set moving step length, the surface of the surface light source light-emitting board is frosted to reduce reflection, and it also serves as a sample carrier. During the detection process, after the image acquisition device moves to a sampling position, the control module enables forward lighting and backlighting in turn to capture images of the two modes respectively.

[0062] In the annotation method, considering that most of the detected targets are relatively slender, have the characteristics of large aspect ratio, and have a large span in the image, in order to improve the detection effect, the present invention adopts the Oriented Bounding Box (OBB) for annotation and designs the model based on it.

[0063] The present invention specifically uses X-AnyLabeling for data labeling, which allows the use of some existing segmentation models to improve the efficiency of data labeling. Specifically, the Segment Anything (ViT-Large) model can be used to pre-label the selected defect to obtain the polygonal boundary of the defect. Then, a script is written to read the json file and calculate its minimum enclosing rectangle, thereby obtaining a more accurate OBB format labeling box.

[0064] It is worth noting that in the process of annotation, it is necessary to comprehensively consider the two images under two lighting conditions. First, the defect category is judged according to the performance of the defect under the two lighting conditions. For example, the depth and degree of damage of the defect are judged according to the image under backlight illumination, so as to distinguish scratch-type defects from contusion-type defects, and exclude small defects that can be ignored. The second is to perform a merging operation, that is, to mark all defects that are only obvious under a single lighting mode and merge them into a marking file. For example, scratch defects are usually only easy to observe under forward lighting, while some smaller point defects that are not obvious under forward lighting are very prominent under back lighting. The third is the marking contour processing. In the case where there are differences in the defect contours under the two lighting conditions, the union of the two defect areas will be selected, that is, the outer contours are taken after the two pictures are superimposed. The present invention checks each annotation file, which includes necessary manual adjustments to ensure the quality of the annotation.

[0065] In the field of industrial defect detection, defect samples are often scarce. In order to solve the problem of the limited number of defect images available for collection of aviation transparent parts, the present invention adopts some data enhancement methods. Since aviation transparent parts defect samples are not directional, in order to enable the model to fully learn the characteristics of slender defects at any angle, so as to improve the detection ability of slender defect samples at any angle, the present invention pre-rotates and enhances the defect samples. The method includes lossless enhancement method and / or arbitrary angle rotation enhancement. In this embodiment, an 8-fold enhancement is performed using a lossless enhancement method, and the implementation method includes mirroring and rotating every 90 degrees. An arbitrary angle rotation enhancement method is used to perform a 32-fold enhancement. The implementation method includes mirroring and rotating every 30 degrees, and filling RGB (114, 114, 0) in the blank space. The results are as follows. Figure 6 shown.

[0066] Since the OBB standard format does not allow the coordinates of the annotation points to be outside the image, it is also necessary to normalize the overflow labels. The correction method for overflow labels is:

[0067] 1), if the four points of the rectangular box are not in the field of view, delete the target (end), otherwise go to 2);

[0068] 2) If all four points of the rectangular frame are in the field of view, no adjustment is required (end), otherwise go to 3);

[0069] 3) Select a point that is not in the field of view, obtain the long and short sides of the rectangle where the point is located, translate the short side along the direction of the long side so that the point falls on the image boundary, and go to 2).

[0070] Furthermore, during the training process, you can also use the Mosaic data enhancement method for data enhancement. Mosaic is a data enhancement technique proposed in YOLOv4 that can significantly enhance the object detection model by expanding the diversity of training data. It works as follows:

[0071] 1),Image selection: Randomly select four images from the dataset.

[0072] 2),Image combination: Select a random point in the new image plane to divide the plane into four quadrants, adjust the size of the four images and put them into the four quadrants to synthesize a single mosaic-like image.

[0073] 3),Bounding box adjustment: The bounding box of each object in the original image is adjusted according to its new position and size in the mosaic image.

[0074] 4),Enhancement: Contrast, exposure, noise addition, etc. can be applied to the synthetic images to enhance the training data.

[0075] Combining different images into a single larger image helps create more diverse scenes and improves the model's generalization ability. This improves the model's performance, especially in detecting smaller objects due to the variations in scale and context introduced in the training dataset.

[0076] The aviation transparent parts defect dataset finally obtained by the above method is named ag_dual_obb, which contains 1752 defect locations and a total of 4784 defect samples. Each defect location includes two types of forward illumination and backlight illumination, and the two images are strictly aligned in space.

[0077] The distribution of the data set finally used in this invention is as follows:

[0078] Index Type Name Count 0 contusion 1104 1 scratches 1728 2 crack 232 3 spot defect 1720

[0079] There are 1576 images in the training set and 176 images in the test set, with a distribution ratio of 9:1

[0080] The dataset obtained by fusion of RGB channels is named ag_composite_obb. In addition, the difference image is used as the blue channel, and the resulting dataset is named ag_composite_diff_obb.

[0081] In order to simply verify the effectiveness of fusion detection for this task, the present invention first designs a front-end fusion (data-level fusion) method based on RGB channel fusion. RGB channel fusion method: The forward illumination image and the backlight illumination image are grayed out respectively as the red channel and green channel of the fused image, and the blue channel is taken from the mean of the two to obtain a composite image, such as Figure 7 shown.

[0082] The design idea of ​​the RGB channel fusion method is based on the fact that for the detection task of the present invention, graying hardly weakens the image information; and mean image fusion itself can be used as a simple image fusion method. In addition, the color of the result after channel fusion reflects the depth information, and also provides a certain reference basis for subsequent data annotation work.

[0083] Therefore, according to the above-mentioned aviation transparent parts defect data set, by constructing a defect detection model for training, the trained model can be used to realize defect detection. In this defect detection model, a synthetic image is obtained mainly according to the above-mentioned RGB channel fusion method, and defect detection is performed based on the synthetic image.

[0084] In order to better utilize the multimodal information under different lighting modes, this paper further proposes a new attention-based dual-modal fusion defect detection model. The overall network architecture is as follows: Figure 8 As shown in the figure, it mainly consists of two parallel independent feature extraction trunks, a modal feature fusion module MFCB (Multimodal Fusion with Convolutional Block Attention Module), a spatial pyramid pooling module FSPPF (Fusion-based Spatial PyramidPooling Fast with CBAM) and a detection layer.

[0085] The two feature extraction backbones take the forward illumination image and the backlight illumination image of an image pair as input respectively, and extract the multi-scale features of the forward illumination modality and the backlight illumination modality. In the present invention, the CSPDarknet-like feature extraction backbone in the YOLOv8 network is specifically used.

[0086] The feature fusion module is inserted between the two feature extraction trunks to obtain fusion features of different receptive fields. In the present invention, there are two feature fusion modules, which are respectively inserted into the P3 layer and the P4 layer of the two feature extraction trunks to obtain fusion features of different receptive fields.

[0087] The spatial pyramid pooling module takes the output of the two feature extraction trunks as input to obtain the final output fusion feature map. In the present invention, the P5 layers of the two feature extraction trunks are connected to the spatial pyramid pooling module.

[0088] The detection layer uses the output of the feature fusion module and the spatial pyramid pooling module as input to obtain the detection result. In the present invention, the features of three scales are sent to the detection layer, and the defect area marked by the rotating box is obtained using the OBB (oriented bounding box) detection head.

[0089] The modal feature fusion attention module MFCB consists of the Concat module and the CBAM module. The feature fusion process is divided into three steps. First, the Concat module is applied to the feature maps of the two modalities for splicing operations, and the preliminary fusion features are obtained based on channel superposition; the second step is to send the preliminary fusion features to the channel attention module CAM, and the channel weights between the two modalities and within the modalities are adaptively learned through the channel attention mechanism. The weight distribution of the two features is adaptively adjusted by calculating the contribution rate of the features of different modalities to the detection; the third step is to apply the spatial attention module SAM to increase the attention to the defect target area and improve the expression ability of the fusion features, so as to obtain the final improved feature map, such as Fig. 9 shown.

[0090] The spatial pyramid pooling module is mainly composed of the Concat module, the SPPF module and the CBAM module. First, the Concat module is used to concatenate the output results of the two feature extraction backbone P5 layers; secondly, the SPPF module is used to perform pooling operations of different scales on the concatenated results to enhance the detection ability of targets of different scales; finally, the CBAM module is used to optimize the feature map to obtain the final output fusion feature map. The output of this module and the output of the MFCB module of the P3 and P5 layers are used as the input of the Neck layer.

[0091] The present invention uses mean average precision (mAP) to evaluate the accuracy of model detection, and the higher the mAP, the better. It includes the mainstream mAP50, which measures the average precision when the intersection and union (IoU) threshold is 0.5, and the more stringent mAP50-95, which measures the average precision when the IoU threshold ranges from 0.5 to 0.95. These indicators are used to evaluate the accuracy of object detection models by calculating the precision and recall of the predicted bounding box compared to the true value.

[0092]

[0093] The experimental environment of the present invention is configured as follows: In terms of software, Python 3.10.14 and PyTorch 2.2.2 are used, and CUDA 11.8 is used for GPU acceleration. At the same time, Ultralytics 8.2.2 is used as the basic framework to support the implementation of the model. In terms of hardware, the experiment is run on a computer equipped with an AMD R7-7800X3D processor (8 cores and 16 threads), 48GB of memory, and a GeForce RTX 4080 graphics card (16GB of video memory).

[0094] The following hyperparameters were used in the training phase: 300 total rounds, 16 batch size, and 640 × 640 input image size. For data augmentation, the Mosaic-4 method was used. The optimizer selected was stochastic gradient descent (SGD), with a learning rate of 0.01, a momentum of 0.937, and a weight decay coefficient of 0.0005.

[0095] The experimental results are shown in the following table:

[0096]

[0097] Note: The two columns of data are mAP and mAP50-95 respectively

[0098] Benchmark Experiment Results Analysis

[0099] First, we train the image using only forward illumination and backlight illumination. The results show that the mAP50 reaches 92.8% when only forward illumination is used, but only 79% when only backlight illumination is used. However, by observing the data of the four defects, we can find that the recognition performance of backlight illumination in contusion and spot is significantly higher than that of forward illumination, while the mAP of scratches is only 23%, which greatly reduces the overall mAP.

[0100] Analysis of RGB channel fusion experimental results

[0101] The training was performed using the ag_composite_obb dataset based on RGB channel fusion. The results showed that a simple channel fusion of the forward-lit and backlit images achieved quite good results. The overall mAP level reached 97.9%, which is close to the natural error of manual annotation. The mAP data of various defects were all higher than the highest level using only a single lighting method, with only a slight decrease in mAP50-95 in the crack project. This shows the effectiveness of dual-modal fusion detection for aviation transparent parts defect detection tasks.

[0102] In addition, experiments on the ag_composite_diff_obb dataset show that in the selection of channel fusion methods, the third channel uses the mean of two illumination images, and its detection effect is slightly better than the method using image difference.

[0103] In addition, the experiment found that using Dynamic Snake Convolution in the neck layer did not improve the dataset of the present invention; adding CBAM attention mechanism could slightly improve the mAP50-95 value. (Yolov9 also did not have a significant improvement effect)

[0104] Experimental results analysis of dual-modal fusion defect detection model based on attention

[0105] refer to Fig.10 and Fig.11 The fusion model of the present invention achieved quite good results on the ag_dual_obb dual-modal dataset. The detection rates of contusion, crack, and spot categories all exceeded 99%. The detection rate of scratches was 95.4%, which was basically consistent with the training results of the channel fusion model on ag_composite_obb. Compared with the RGB channel fusion model, the total mAP50 increased from 97.9% to 98.4%, and the curve of the dual-modal fusion defect detection model of the present invention was smoother.

[0106] In summary, the present invention proposes for the first time a dual-modal aviation transparent parts defect dataset ag_dual_obb, which contains two types of images of forward illumination and backlight illumination of 1752 defect locations, and a total of 4784 defect samples. The samples are divided into four categories and annotated using the OBB format.

[0107] At the same time, the present invention proposes an attention-based dual-modal fusion defect detection model, which can detect targets on simultaneously input forward-lighting and back-lighting image pairs. It effectively combines the complementary advantages of forward-lighting and back-lighting features and can accurately detect defective targets.

[0108] The detection method of the present invention has a significant detection effect on the ag_dual_obb dataset. Compared with the results of the benchmark algorithm YOLOv8m detecting the forward-illuminated image and the back-illuminated image separately, the detection accuracy of the proposed algorithm is improved by 5.6% and 19.4% respectively; the experimental comparison results with multiple classic dual-modal target detection algorithms verify that the method of the present invention has good detection performance, and has good real-time and robustness.

Claims

1. A method for collecting defects of aviation transparent parts based on mixed illumination, characterized in that: For the same defect area of ​​aviation transparent parts, forward lighting and backlighting are used in turn to acquire an image pair. The image pair data of different defect areas are annotated and enhanced to construct a dual-modal image dataset for defect detection of aviation transparent parts. The defect types covered include bruises, scratches, cracks and point defects. The two modal images in the dataset are strictly aligned in space.

2. The method for collecting defects of aviation transparent parts based on mixed illumination according to claim 1 is characterized in that: The forward lighting is bright field forward lighting, using a ring light source to achieve uniform lighting to reduce reflections; the backlighting uses a surface light source luminous panel as a light source.

3. The method for collecting defects of aviation transparent parts based on mixed illumination according to claim 2 is characterized in that: The aviation transparent part is fixedly arranged, an electronic digital microscope is used as the image acquisition device, and a carrying device is used to carry the image acquisition device and the annular light source to meet the stability required for acquisition; wherein: The carrying device is a universal adjustment arm, and the defect collection is completed by manually adjusting the collection position; or, The carrying device is a bidirectional sliding guide rail, which traverses the inspected sample according to the set moving step length. The surface of the surface light source luminous plate is frosted to reduce reflection, and it also serves as a sample carrying platform. During the detection process, after the image acquisition device moves to a sampling position, the forward lighting and backlighting are enabled in turn to capture images of the two modes respectively.

4. The method for collecting defects of aviation transparent parts based on mixed illumination according to claim 1, characterized in that: The acquired images of different defect areas are labeled. The implementation method is as follows: First, the Segment Anything (ViT-Large) Model is used to pre-label the selected defects and obtain the polygonal boundary of the defect. Then a script is written to read the JSON file and calculate the minimum enclosing rectangle of the polygon to obtain an accurate OBB format labeling box.

5. The method for collecting defects of aviation transparent parts based on mixed illumination according to claim 1 or 4, characterized in that: During the annotation process, the defect category is judged according to the performance of the defect under two lighting conditions, and the defects that are obvious only under a single lighting method are annotated and merged into a annotation file after annotation; at the same time, in case there are differences in the defect contours under the two lighting methods, the union of the defect areas of the two images is selected, that is, the outer contours are taken after the two images are superimposed.

6. The method for collecting defects of aviation transparent parts based on mixed illumination according to claim 1 or 4, characterized in that: The acquired images of different defect areas are enhanced, and the implementation method includes: Using a lossless enhancement method and / or an arbitrary angle rotation enhancement method to perform pre-rotation enhancement; Use Mosaic data enhancement method to perform data enhancement during training; During the enhancement process, the overflow labels are normalized as follows: 1), if the four points of the rectangular box are not in the field of view, delete the target, otherwise go to 2); 2) If all four points of the rectangular frame are in the field of view, no adjustment is required, otherwise go to 3); 3) Select a point that is not in the field of view, obtain the long and short sides of the rectangle where the point is located, translate the short side along the direction of the long side so that the point falls on the image boundary, and go to 2).

7. A defect detection method based on the aviation transparent part defect collection method based on mixed illumination according to claim 1, characterized in that: The steps include: Step 1, constructing a defect detection model, wherein the defect detection model grayscales the forward illumination image and the backlight illumination image of an image pair respectively as the red channel and the green channel of the fused image, and takes the blue channel as the mean of the two to obtain a composite image, and performs defect detection based on the composite image; Step 2, using the aviation transparent part defect detection dual-modal image dataset to train the defect detection model; Step 3: Input the collected forward illumination image and backlight illumination image into the trained defect detection model to obtain its defect detection results.

8. A defect detection method based on the aviation transparent part defect collection method based on mixed illumination according to claim 1, characterized in that: The steps include: Step 1, construct an attention-based dual-modal fusion defect detection model; the model includes a feature extraction backbone, a feature fusion module, a spatial pyramid pooling module and a detection layer; The feature extraction trunk has two parallel trunks, which respectively take the forward illumination image and the backlight illumination image of an image pair as input to extract multi-scale features of the forward illumination mode and the backlight illumination mode; the feature fusion module is inserted between the two feature extraction trunks to obtain fusion features of different receptive fields; The spatial pyramid pooling module takes the outputs of the two feature extraction trunks as input to obtain a fusion feature map of the final output; the detection layer takes the outputs of the feature fusion module and the spatial pyramid pooling module as input to obtain a detection result; Step 2, using the aviation transparent part defect detection dual-modal image dataset to train the defect detection model; Step 3: Input the collected forward illumination image and backlight illumination image into the trained defect detection model to obtain its defect detection results.

9. The defect detection method according to claim 8, characterized in that: The feature extraction backbone is a CSPDarknet-like feature extraction backbone in a YOLOv8 network; there are two feature fusion modules, which are respectively inserted into the P3 layer and the P4 layer of two CSPDarknet-like feature extraction backbones; and the P5 layers of the two CSPDarknet-like feature extraction backbones are connected to the spatial pyramid pooling module.

10. The defect detection method according to claim 8, characterized in that: The feature fusion module is composed of a Concat module and a CBAM module; the feature fusion process is as follows: Firstly, the Concat module is used to concatenate the feature maps of the two modalities, and the preliminary fusion features are obtained based on channel superposition. Secondly, the preliminary fusion features are sent to the channel attention module CAM, and the channel weights between the two modalities and within the modalities are adaptively learned through the channel attention mechanism. The weight distribution of the two-way features is adaptively adjusted by calculating the contribution rate of the features of different modalities to the detection. Finally, the spatial attention module SAM is applied to improve the attention to the defect target area, improve the expression ability of the fusion features, and obtain the final improved feature map. The spatial pyramid pooling module is composed of a Concat module, an SPPF module and a CBAM module, and the process is as follows: First, the Concat module is used to concatenate the outputs of the two feature extraction backbone P5 layers; Secondly, the SPPF module is used to perform pooling operations of different scales on the splicing results to enhance the detection ability of targets of different scales; finally, the CBAM module is used to optimize the feature map to obtain the final output fusion feature map.

Images