A machine vision-based system and method for detecting surface defects of an aircraft structure

Abstract

The present application relates to the technical field of machine vision, in particular to an aviation structure surface defect detection system and method based on machine vision, which comprises three parts of training, reasoning and visualization, the training part carries out model training and optimization according to the defect data set to be detected, the reasoning part adopts two targeted designs of sliding window and feature fusion according to the morphology of defects and the image features of the detected aviation structure, to improve the detection accuracy, and the visualization part adds a projection device to mark the actual position of defects on the detected aviation structure. The present application replaces manual visual inspection with machine vision, adopts the technologies of sliding window, deep learning and traditional method fusion to solve the detection rate of small targets in high-resolution images in the detection process, carries out defect detection on the surface of aviation structure based on machine vision, realizes the collection of defect sample data, and ensures the accuracy of defect detection.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a machine vision-based system and method for detecting surface defects in aerospace structural components. Background Technology

[0002] Currently, the main method for detecting surface defects in aerospace structural components is manual visual inspection, which relies on experience and knowledge to determine the category of defects. However, manual inspection has disadvantages such as strong subjectivity, easy fatigue, and strong correlation with personal experience, resulting in low inspection efficiency, easy omissions, time-consuming and labor-intensive, and difficulty in guaranteeing inspection accuracy.

[0003] Machine vision has found practical applications in industries such as welding defect detection in electronics and packaging defect detection in pharmaceuticals, which are characterized by batch processing and standardization. Surface defect detection based on machine vision has been widely used in fields such as LCD panels and circuit boards. Applying machine vision to the surface defect detection of aerospace structural components presents challenges. However, achieving surface defect detection in aerospace structural components requires a large amount of defect image data for model training. The aerospace manufacturing field is characterized by customization; even tiny surface defects can cause serious product quality problems. Furthermore, structural components often have complex structures such as rivets and mounting brackets, and the changing ambient light leads to inconsistent camera imaging, making machine vision technology difficult to use for detection and compromising accuracy. Therefore, collecting defect sample data is difficult, and the size of the defects is extremely small compared to aerospace structural components, making existing deep learning image recognition methods insufficient for their detection requirements. Summary of the Invention

[0004] (1) Technical problems to be solved

[0005] This invention provides a machine vision-based surface defect detection system and method for aerospace structural components, which solves the technical problems of difficulty in collecting defect sample data and inability to guarantee the accuracy of defect detection.

[0006] (2) Technical solution

[0007] In a first aspect, the present invention provides a machine vision-based surface defect detection system for aerospace structural components, comprising: a training module, an inference module, and a visualization module; the training module acquires surface images of aerospace structural components using a high-resolution camera, establishes a dataset of defect images to be detected for image annotation, and uses a sliding window to traverse the defect dataset, with the traversed images serving as input to a convolutional neural network model for training; the inference module acquires surface images of aerospace structural components using a high-resolution camera, traverses the current frame image using a sliding window, and outputs the detection result after model inference using the traversed images as input; the visualization module projects the detected defect coordinates onto the inspected aerospace structural component using a projection device, facilitating manual processing.

[0008] Secondly, the present invention provides a machine vision-based method for detecting surface defects in aerospace structural components, comprising the following steps:

[0009] S1, the training phase, involves training and optimizing the model based on the defect dataset to be detected;

[0010] S2, the reasoning stage, based on the morphology of the defect and the image features of the inspected aerospace structural component, adopts sliding window and feature fusion to improve the detection accuracy;

[0011] S3, the visualization stage, marks the actual location of defects on the inspected aerospace structural components.

[0012] Furthermore, in S1, the training phase includes the following steps:

[0013] S10 uses a high-definition line scan camera to collect image data of the inspected aerospace structural components. It collects surface images of the aerospace structural components under different lighting conditions and different acquisition angles, establishes a dataset, and labels the dataset. The labeled images are divided into training set and test set according to the proportion.

[0014] S11, perform data augmentation operations on the collected dataset, including but not limited to image flipping, cropping, image brightness enhancement or reduction, and masking of random regions of the image, to expand the training dataset;

[0015] S12, the collected defect dataset and standard images are subtracted, and the subtracted images are superimposed on the defect images. The fused images are cropped according to the labeled defect locations, and the high-resolution large images are cropped into low-resolution small images. The low-resolution small images are batched into the network model for training and optimization. The learning rate and training batch of the network are adjusted, and different loss functions are used for repeated training until the model performs best on the test set. The weights and parameters of the model at this time are saved.

[0016] Furthermore, the sliding window in the training phase only slices the labeled area, dividing the high-resolution image according to the region where the defect is located, thereby increasing the area ratio of the target defect relative to the input image.

[0017] Furthermore, the training phase uses feature fusion. After the collected defect images are differencing with standard images, traditional image features of the defects are obtained and superimposed onto the original image dataset. This fully utilizes template images, improves the model's expressive power, and increases the model's detection rate of minor defects.

[0018] Furthermore, in S2, the reasoning phase includes the following steps:

[0019] S20: Image acquisition is performed on the inspected aerospace structural component to obtain high-resolution images;

[0020] S21, the image is sliced ​​using a sliding window method with overlapping ratio to ensure that the edge parts of the small image are not missed after slicing. Each high-resolution image is sliced ​​into multiple low-resolution images and fed into the trained network for inference.

[0021] S22: The detection results obtained from low-resolution inference are mapped onto the original high-resolution image. The corresponding detection regions with duplicates in the large image are then deduplicated using non-maximum suppression to obtain the final detection results.

[0022] Furthermore, the inference stage uses a sliding window to segment the high-resolution image, with the segmentation method being from left to right and from top to bottom, and adjacent sliding windows having overlapping parts to ensure that defects are not missed.

[0023] Furthermore, in S3, the visualization phase includes the following steps:

[0024] S30, calibrate the projector with the aircraft structural component being inspected to determine the positional relationship between the projector and the surface of the aircraft structural component being inspected;

[0025] S31, calibrate the projector and the line scan camera to determine the conversion relationship between the image captured by the line scan camera and the image projected by the projector;

[0026] S32: Obtain the defect coordinates during the defect detection stage, transform them to obtain the image coordinates to be projected by the projector, and perform projection visualization of the actual defect.

[0027] Furthermore, in the visualization stage, the projector receives the two-dimensional image defect coordinates acquired by the line scan camera, performs image relationship transformation, and then projects them onto the surface of the aerospace structural component.

[0028] Furthermore, a sliding window and feature fusion are used in both the training and inference phases. In the training phase, the sliding window only slices the labeled regions, while in the inference phase, the sliding window slices all regions.

[0029] (3) Beneficial effects

[0030] In summary, this invention improves the efficiency of quality inspection and reduces the possibility of missed detections by replacing manual visual inspection with machine vision. At the same time, in order to solve the problem of the detection rate of small targets in high-resolution images during the inspection process, this invention adopts the sliding window technology and the technology of integrating deep learning with traditional methods. By using machine vision to detect defects on the surface of aerospace structural parts, it realizes the collection of defect sample data and ensures the accuracy of defect detection. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of a machine vision-based surface defect detection system for aerospace structural components according to an embodiment of the present invention.

[0033] Figure 2 A schematic diagram of the training phase of a machine vision-based method for detecting surface defects in aerospace structural components according to an embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram of the inference stage of a machine vision-based method for detecting surface defects in aerospace structural components according to an embodiment of the present invention.

[0035] Figure 4 This is a schematic diagram of the visualization stage of a machine vision-based method for detecting surface defects in aerospace structural components according to an embodiment of the present invention.

[0036] In the diagram: 1. Composite support; 2. Line scan camera; 3. All-in-one machine; 5. Line scan light source; 6. Turntable. Detailed Implementation

[0037] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of the present invention by way of example, but should not be used to limit the scope of the present invention. That is, the present invention is not limited to the described embodiments, and any modifications, substitutions and improvements to the parts, components and connection methods are covered without departing from the spirit of the present invention.

[0038] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0039] Please refer to Figure 1 In a first aspect, the present invention provides a machine vision-based surface defect detection system for aerospace structural components, comprising: a training module, an inference module, and a visualization module; the training module acquires surface images of aerospace structural components using a high-resolution camera, establishes a dataset of defect images to be detected for image annotation, and uses a sliding window to traverse the defect dataset, with the traversed images serving as input to a convolutional neural network model for training; the inference module acquires surface images of aerospace structural components using a high-resolution camera, traverses the current frame image using a sliding window, and outputs the detection result after model inference using the traversed images as input; the visualization module projects the detected defect coordinates onto the inspected aerospace structural component using a projection device, facilitating manual processing.

[0040] The aircraft structural component to be inspected is placed on a turntable 6, which rotates at a constant speed. A line scan camera 2 and a line scan light source 5 are mounted on a composite bracket 1 to ensure that the image positions illuminated by the camera and light source are consistent, guaranteeing image imaging quality. A three-color alarm device can be added, integrated with the all-in-one machine 3. A projection system is fixed on the other side. The specific implementation method for aircraft structural component defect detection is as follows: Rotation is started on the all-in-one machine 3. After the turntable 6 rotates at a constant speed, the line scan camera 2 starts image acquisition. Appropriate line frequency and exposure time are calculated to match the rotation speed of the turntable 6. One frame of image is acquired for exactly one revolution of the aircraft structural component being inspected. The acquired image is used as input for defect detection in the all-in-one machine 3, and the detection results are displayed in real-time on the front-end interface. When the turntable 6 rotates a certain angle, the projection system projects the defect coordinates onto the surface of the aircraft structural component being inspected, and the defect is manually cleaned. When the defect is cleaned, the alarm light is green; when it is not cleaned, the alarm light is red.

[0041] Secondly, the present invention provides a machine vision-based method for detecting surface defects in aerospace structural components, comprising the following steps:

[0042] S1, the training phase, involves training and optimizing the model based on the defect dataset to be detected;

[0043] S2, the reasoning stage, based on the morphology of the defect and the image features of the inspected aerospace structural component, adopts sliding window and feature fusion to improve the detection accuracy;

[0044] S3, the visualization stage, marks the actual location of defects on the inspected aerospace structural components.

[0045] During the training phase, high-resolution cameras are used to acquire surface images of aerospace structural components. The acquired images are first labeled (the coordinates of the defect locations are assigned), and the images with foreign objects are subtracted from standard images and then converted to grayscale (color images are converted to grayscale). The grayscale features are used as a fourth channel in addition to RGB. The labeled high-resolution images are then segmented into multiple low-resolution images, and the labeled images are retained as the training set. The low-resolution images, which incorporate traditional foreign object features, are then fed into the neural network for training. After training, the model's weights and hyperparameters are saved.

[0046] In the inference phase, high-resolution cameras are used to acquire surface images of aerospace structural components. The acquired images are preprocessed and scanned using a sliding window high-resolution image. The low-resolution images obtained from the scan are fed into the trained network for inference to obtain the detection results in the low-resolution images. The results of the low-resolution images obtained from the high-resolution images are merged and restored to the high-resolution image. Non-maximum suppression is performed to remove duplicates, and the final detection results are obtained.

[0047] In the visualization stage, the coordinates of the defect image are obtained. After image transformation, the projected coordinates of the defect under the projector are obtained, and the projection visualization is performed on the actual inspected aerospace structural component.

[0048] Please refer to Figure 2 In some embodiments, in S1, the training phase includes the steps of:

[0049] S10: Use a high-definition line scan camera 2 to collect image data of the inspected aerospace structural components. Under different lighting conditions and different acquisition angles, surface images of the aerospace structural components are acquired to establish a dataset. At the same time, the dataset is labeled, and the labeled images are divided into training set and test set according to the proportion.

[0050] S11, perform data augmentation operations on the collected dataset, including but not limited to image flipping, cropping, image brightness enhancement or reduction, and masking of random regions of the image, to expand the training dataset;

[0051] S12, the collected defect dataset and standard images are subtracted, and the subtracted images are superimposed on the defect images. The fused images are cropped according to the labeled defect locations, and the high-resolution large images are cropped into low-resolution small images. The low-resolution small images are batched into the network model for training and optimization. The learning rate and training batch of the network are adjusted, and different loss functions are used for repeated training until the model performs best on the test set. The weights and parameters of the model at this time are saved.

[0052] In some embodiments, the sliding window in the training phase only slices the labeled area, dividing the high-resolution image according to the region where the defect is located, thereby increasing the area ratio of the target defect relative to the input image.

[0053] In some embodiments, the training phase uses feature fusion. After the defective image is collected and the standard image is differentiated, the traditional image features of the defect are obtained and superimposed on the original image dataset. This fully utilizes the template image, improves the model's expressive power, and increases the model's detection rate of minor defects.

[0054] Please refer to Figure 3 In some embodiments, in S2, the inference phase includes the steps of:

[0055] S20: Image acquisition is performed on the inspected aerospace structural component to obtain high-resolution images;

[0056] S21, the image is sliced ​​using a sliding window method with overlapping ratio to ensure that the edge parts of the small image are not missed after slicing. Each high-resolution image is sliced ​​into multiple low-resolution images and fed into the trained network for inference.

[0057] S22: The detection results obtained from low-resolution inference are mapped onto the original high-resolution image. The corresponding detection regions with duplicates in the large image are then deduplicated using non-maximum suppression to obtain the final detection results.

[0058] In some embodiments, the inference stage uses a sliding window to segment the high-resolution image, with the segmentation method being from left to right and from top to bottom, and adjacent sliding windows having overlapping portions to ensure that defects are not missed.

[0059] Please refer to Figure 4 In some embodiments, in S3, the visualization phase includes the steps of:

[0060] S30, calibrate the projector with the aircraft structural component being inspected to determine the positional relationship between the projector and the surface of the aircraft structural component being inspected;

[0061] S31, calibrate the projector and the line scan camera 2 to determine the conversion relationship between the image captured by the line scan camera 2 and the image projected by the projector;

[0062] S32: Obtain the defect coordinates during the defect detection stage, transform them to obtain the image coordinates to be projected by the projector, and perform projection visualization of the actual defect.

[0063] In some embodiments, during the visualization stage, the projector receives the two-dimensional image defect coordinates acquired by the line scan camera 2, and after image relationship transformation, projects them onto the surface of the aerospace structural component.

[0064] In some embodiments, a sliding window and feature fusion are used in both the training and inference phases. In the training phase, the sliding window only slices the labeled regions, while in the inference phase, the sliding window slices all regions.

[0065] Using the above technical solution, an experiment was conducted on a test piece, which was a barrel-shaped structure, 1 meter high and 1 meter in diameter. The system used to inspect its outer surface took a total of 2 minutes, while manual visual inspection took over 20 minutes, and the rate of missed inspections was significantly higher than with the system of this invention.

[0066] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For the embodiments of the method, relevant parts can be referred to the description of the device embodiments (as appropriate). The present invention is not limited to the specific steps and structures described above and shown in the figures. Furthermore, for the sake of brevity, detailed descriptions of known methods and techniques are omitted here.

[0067] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art without departing from the scope of the invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting surface defects in aerospace structural components based on machine vision, characterized in that, Including the following steps: S1, the training phase, involves training and optimizing the model based on the defect dataset to be detected; S2, the reasoning stage, based on the morphology of the defect and the image features of the inspected aerospace structural component, adopts sliding window and feature fusion to improve the detection accuracy; S3, the visualization stage, marks the actual location of defects on the inspected aerospace structural components; In S1, the training phase includes the following steps: S10 uses a high-definition line scan camera to collect image data of the inspected aerospace structural components. It collects surface images of the aerospace structural components under different lighting conditions and different acquisition angles, establishes a dataset, and labels the dataset. The labeled images are divided into training set and test set according to the proportion. S11, perform data augmentation operations on the collected dataset, including but not limited to image flipping, cropping, image brightness enhancement or reduction, and masking of random regions of the image, to expand the training dataset; S12, the collected defect dataset and standard images are subtracted, and the subtracted images are superimposed on the defect images. The fused images are cropped according to the labeled defect locations, and the high-resolution large images are cropped into low-resolution small images. The low-resolution small images are batched into the network model for training and optimization. The learning rate and training batch of the network are adjusted, and different loss functions are used for repeated training until the model performs best on the test set. The weights and parameters of the model at this time are saved. In S2, the reasoning phase includes the following steps: S20: Image acquisition is performed on the inspected aerospace structural component to obtain high-resolution images; S21, the image is sliced ​​using a sliding window method with overlapping ratio to ensure that the edge parts of the small image are not missed after slicing. Each high-resolution image is sliced ​​into multiple low-resolution images and fed into the trained network for inference. S22: The detection results obtained from low-resolution inference are mapped onto the original high-resolution image. The corresponding detection regions with duplicates in the large image are then deduplicated using non-maximum suppression to obtain the final detection results.

2. The machine vision-based surface defect detection method for aerospace structural components according to claim 1, characterized in that, The sliding window in the training phase only slices the labeled area, dividing the high-resolution image according to the region where the defect is located, thereby increasing the area ratio of the target defect relative to the input image.

3. The machine vision-based surface defect detection method for aerospace structural components according to claim 1, characterized in that, The training phase uses feature fusion. After the defective images are collected and differentiated from the standard images, the traditional image features of the defects are obtained and superimposed on the original image dataset. This fully utilizes the template images, improves the model's expressive power, and increases the model's detection rate of minor defects.

4. The machine vision-based surface defect detection method for aerospace structural components according to claim 1, characterized in that, The inference stage uses a sliding window to segment the high-resolution image. The segmentation method is from left to right and from top to bottom, with overlapping parts between adjacent sliding windows to ensure that defects are not missed.

5. The machine vision-based surface defect detection method for aerospace structural components according to claim 1, characterized in that, In S3, the visualization phase includes the following steps: S30, calibrate the projector with the aircraft structural component being inspected to determine the positional relationship between the projector and the surface of the aircraft structural component being inspected; S31, calibrate the projector and the line scan camera to determine the conversion relationship between the image captured by the line scan camera and the image projected by the projector; S32: Obtain the defect coordinates during the defect detection stage, transform them to obtain the image coordinates to be projected by the projector, and perform projection visualization of the actual defect.

6. The machine vision-based surface defect detection method for aerospace structural components according to claim 5, characterized in that, In the visualization stage, the projector receives the two-dimensional image defect coordinates acquired by the line scan camera, transforms the image relationship, and projects them onto the surface of the aerospace structural component.

7. The machine vision-based surface defect detection method for aerospace structural components according to claim 1, characterized in that, During both the training and inference phases, a sliding window and feature fusion are employed. During the training phase, the sliding window only slices the labeled regions, while during the inference phase, it slices all regions.

8. A machine vision-based surface defect detection system for aerospace structural components, characterized in that, This method is used to implement the machine vision-based surface defect detection method for aerospace structural components as described in any one of claims 1 to 7.

Images