A method for detecting micro-defects of engine blades

Abstract

The application belongs to the technical field of machine vision, and discloses a kind of engine blade micro defect detection methods, comprising the following steps: low-resolution regional image is sent to super-resolution reconstruction neural network model, and the reconstructed high-resolution regional image is obtained;High-resolution regional image is sent into fine-grained classification network model in parallel, and the index of the regional image with defects in the current batch is obtained;According to the index, the corresponding regional image is extracted from the high-resolution regional image, and the extracted regional image is sent into the defect positioning model in parallel, to obtain the defect coordinates and defect category in each regional image;The defect coordinates on the complete blade image are calculated;Regional image is spliced to obtain a complete blade image;The defect coordinates, defect category and complete blade image are output.The detection method can replace artificial, and accurately detect the micro defects on the engine blade.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine vision, and particularly relates to an engine blade micro-defect detection method. BACKGROUND

[0002] The engine blade of an airplane is an important component of the engine, and the health degree of the engine blade will directly affect the safety of the airplane in the flight process. In the flight process of the airplane, the engine will generate high temperature and high pressure, and therefore the blade will be damaged to a certain extent, so it is necessary to detect the defects of the blade to determine whether the blade needs to be replaced or repaired.

[0003] At present, the blade defect detection is achieved by manual detection to determine the position of the defects, or by machine vision-based defect detection. The manual detection has strong subjectivity, and the accuracy of the manual detection will gradually decrease under long-time work. The machine vision-based defect detection is mostly for relatively obvious defects. For relatively slight defects with small proportion, the detection accuracy is still relatively low at present. SUMMARY

[0004] The present application aims to at least partially solve the above technical problems. To this end, the present application aims to provide an engine blade micro-defect detection method.

[0005] The technical solution adopted by the present application is as follows:

[0006] An engine blade micro-defect detection method comprises the following steps:

[0007] S1, training a super-resolution reconstruction network model to obtain a trained super-resolution reconstruction network model;

[0008] S2, training a fine-grained classification network model to obtain a trained fine-grained classification network model;

[0009] S3, training a defect positioning model to obtain a trained defect positioning model;

[0010] S4, sequentially collecting low-resolution regional images of the engine blade by a low-resolution camera, and the adjacent regional images have overlapping parts;

[0011] S5, inputting the collected multiple low-resolution regional images into the super-resolution reconstruction neural network model in a batch to obtain a batch of reconstructed high-resolution regional images;

[0012] S6, parallelly inputting the batch of reconstructed high-resolution regional images into the fine-grained classification network model to obtain the index of the regional image with defects in the current batch;

[0013] S7, extracting corresponding region images from the high-resolution region images of the current batch according to the index, and sending the extracted region images into the defect positioning model in parallel to obtain the defect coordinates and defect categories in each region image;

[0014] S8, calculating the defect coordinates on the complete leaf image according to the defect coordinates in the region image, the super-resolution reconstruction ratio, the index of the region image, and the number of overlapping pixels of the adjacent region images in the horizontal and vertical directions;

[0015] S9, splicing the region images to obtain a complete leaf image;

[0016] S10, outputting the defect coordinates, defect categories, and complete leaf image.

[0017] Preferably, step S1 comprises:

[0018] S11, using a high-resolution camera to sequentially collect region images image_h according to the divided regions, collecting images through a degradation method image_l=D(image_h) to form a low-resolution blurred image set image_l, and dividing image_h and image_l into a training set and a validation set;

[0019] S12, expanding the training set through data augmentation;

[0020] S13, training a super-resolution reconstruction network model based on a generative adversarial network using the expanded training set, the generative adversarial network comprising a generator g and a discriminator d, using all the low-resolution blurred image set image_l as the input of the generator g to obtain a predicted sr_ratio times resolution clear image set image_h_pred=g(image_l);

[0021] S14, calculating the discriminator d loss loss_d=-E x∈image_h (log(d(x)))-E y∈image_h_pred (log(1-d(y))), and calculating the generator g loss loss_g=-E y∈image_h_pred (log(d(y)));

[0022] S15, updating the generator g and the discriminator d through back propagation of the loss function until the loss function of the model on the validation set is lower than a preset threshold to stop updating, and obtaining a trained super-resolution reconstruction network model.

[0023] Preferably, step S2 comprises:

[0024] S21, predicting output of all image_l super-resolution reconstruction results using a generator g in a generative adversarial network, to obtain a reconstructed image set image_sr;

[0025] S22, labeling each image in image_sr, and labeling defective images as label=1 and non-defective images as label=0;

[0026] S23, dividing the labeled images into a training set and a validation set;

[0027] S24, expanding the training set through data augmentation;

[0028] S25, training a fine-grained classification network model using the expanded training set, and calculating a loss function of the fine-grained classification network model;

[0029] S26, updating network parameters through the calculated loss function, until the classification accuracy of the model on the validation set reaches a preset threshold, then stopping network updating, and obtaining a trained fine-grained classification network model.

[0030] Preferably, step S25 comprises: calculating the loss of positive samples or defective samples loss_neg=-(1-softmax(cls(x_pos))) γ log(cls(x_pos)),

[0031] Where x_pos is a positive sample in image_sr with label=1, cls is a fine-grained classification model, and γ is an adjustment parameter.

[0032] Calculating the loss of negative samples or non-defective samples loss_neg=-cls(x_neg) γ log(1-cls(x_neg)),

[0033] Where x_neg is a negative sample in image_sr with label=0.

[0034] Preferably, step S3 comprises:

[0035] S31, annotating defects in image_sr, marking defects using the smallest circumscribed rectangle and recording defect types, annotation including rectangular frame coordinate annotation (x, y, w, h) and target class annotation c, wherein (x, y) is the rectangular frame center point coordinate, (w, h) is the rectangular frame width and height, and c is the class index;

[0036] S32, dividing the annotated data set into a training set and a validation set;

[0037] S33, expanding the training set through data augmentation;

[0038] S34, send the labeled training set pictures into the defect positioning model to obtain network output and perform loss function calculation;

[0039] S35, update the network parameters through the back propagation algorithm until the detection accuracy of the model on the verification set reaches the preset threshold, and then stop updating to obtain the trained defect positioning model.

[0040] Preferably, step S4 comprises:

[0041] S41, issue a photographing instruction to the image acquisition module;

[0042] S42, the image acquisition module controls the low-resolution camera to take pictures;

[0043] S43, after the photo is returned, the image acquisition module and the mechanical control unit control the movement of the object table by a fixed step;

[0044] S44, after the movement is completed, the mechanical control unit notifies the image acquisition module, and then returns to step S42 until all areas are photographed.

[0045] Preferably, in step S8, the formula is:

[0046] x_orig = col_index * (dw - x_overlap) + x_det / sr_ratio,

[0047] y_orig = row_index * (dh - y_overlap) + y_det / sr_ratio,

[0048] w_orig = w_det / sr_ratio,

[0049] h_orig = h_det / sr_ratio,

[0050] coordinate recovery is realized;

[0051] where (x_orig, y_orig, w_orig, h_orig) are the bounding box coordinates of the detected defects on the complete leaf image, (x_orig, y_orig) are the center point coordinates of the bounding box, (w_orig, h_orig) are the width and height of the bounding box; row_index and col_index are the indices of the photographed area images, dw is the pixel width of each leaf area image, dh is the pixel height of each leaf area image, y_overlap and x_overlap are the number of overlapping pixels in the vertical direction and the horizontal direction when the area image is collected, respectively.

[0052] Preferably, step S8 further comprises: removing the repeatedly labeled bounding boxes of the overlapping area by a non-maximum suppression algorithm to obtain the labeling coordinates and category information of all defects in the complete leaf image.

[0053] Preferably, step S9 comprises:

[0054] S91, region image pre-processing is performed, Gaussian filtering is performed on each row of two region images adjacent to left and right to remove noise, SIFT key points are detected, the left image is defined as a reference image, and the right image is defined as a converted image;

[0055] S92, image registration is performed, key points of common parts of two region images adjacent to left and right are matched, and a coordinate transformation relationship between the two images is calculated;

[0056] S93, the converted image is transformed by the obtained coordinate transformation relationship, so that the converted image and the reference image are in the same coordinate system;

[0057] S94, the overlapping part of the converted image and the reference image is fused to obtain a fused image, and the left and right adjacent images are spliced;

[0058] S95, each row of region images spliced left and right are spliced up and down to obtain a complete leaf image.

[0059] The beneficial effects of the present application are:

[0060] The engine blade micro-defect detection method provided by the present application improves the defect feature proportion by region shooting, and obtains a high-quality image through super-resolution reconstruction, thereby improving the detection accuracy of the blade micro-defect to a certain extent. Secondly, since the region image processing is parallel batch processing, the recognition speed is ensured. Since the super-resolution reconstruction technology is used, the shooting effect close to that of a high-resolution camera can be reconstructed by a low-resolution camera, so the camera hardware cost is also saved. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a schematic diagram of the engine blade shooting region of the present application.

[0062] Figure 2 is a flowchart of the engine blade micro-defect detection method of the present application. DETAILED DESCRIPTION

[0063] The technical solutions of the present application will be described clearly and completely in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0064] As Figure 1 and Figure 2 shown, the engine blade micro defect detection method of the embodiment comprises the following steps:

[0065] S1, training a super-resolution reconstruction network model based on a generative adversarial network to obtain a trained super-resolution reconstruction network model. Step S1 comprises:

[0066] S11, image acquisition using a high-resolution camera, the acquisition mode is to sequentially acquire a regional image set image_h according to the division region, the acquired image is degraded by a degradation method image_l=D(image_h) to form a low-resolution blurred image set image_l, and image_h and image_l are divided into training set and validation set; wherein 80% of the data in image_h is the training set, and 20% of the data is the validation set, 80% of the data in image_l is the training set, and 20% of the data is the validation set;

[0067] S12, expand the training set through rotation, translation, inversion and other data augmentation to improve the generalization ability of the model;

[0068] S13, training a super-resolution reconstruction network model based on a generative adversarial network using the expanded training set, the generative adversarial network comprising a generator g and a discriminator d, using all low-resolution blurred image sets image_l as input of the generator g to obtain a predicted sr_ratio times resolution clear image set image_h_pred=g(image_l);

[0069] S14, calculating the discriminator d loss loss_d=-E x∈image_h (log(d(x)))-E y∈image_h_pred (log(1-d(y))) and the generator g loss loss_g=-E y∈image_h_pred (log(d(y)));

[0070] S15, update the generator g and the discriminator d through the loss function back propagation until the loss function of the model on the validation set is lower than the preset threshold to stop updating, and obtain the trained super-resolution reconstruction network model.

[0071] S2, training a fine-grained classification network model based on an attention mechanism to obtain a trained fine-grained classification network model. Step S2 comprises:

[0072] S21. Using the trained super-resolution reconstruction network model, i.e. the generator g in the generative adversarial network, predicts and outputs the super-resolution reconstruction results of all images_l to obtain the reconstructed image set image_sr.

[0073] S22. Label each image in image_sr, with defective images labeled as label=1 and defect-free images labeled as label=0;

[0074] S23. Divide the labeled images into a dataset, with 80% as the training set and 20% as the validation set;

[0075] S24. Expand the training set by data augmentation methods such as rotation, translation, and random scaling.

[0076] S25. Use the expanded training set to train the fine-grained classification network model and calculate the loss function of the fine-grained classification network model; using focal loss as the loss function is beneficial for solving problems such as difficult samples and imbalanced samples; calculate the loss of positive samples or defective samples: loss_neg=-(1-softmax(cls(x_pos))) γ log(cls(x_pos)), where x_pos are the positive samples with label=1 in image_sr, cls is the fine-grained classification model, and γ is the adjustment parameter; calculate the loss for negative samples or defect-free samples: loss_neg = -cls(x_neg). γ log(1-cls(x_neg)), where x_neg is the negative sample with label=0 in image_sr;

[0077] S26. Update the network parameters using the calculated loss function until the model's classification accuracy on the validation set reaches a preset threshold, then stop updating the network to obtain a trained fine-grained classification network model.

[0078] S3. Train a defect localization model based on YOLOv7 to obtain a trained defect localization model. Step S3 includes:

[0079] S31. Label the defects in image_sr, mark the defects using the minimum bounding rectangle and record the defect type. Use the YOLO labeling format. The labeling includes the rectangle coordinates (x,y,w,h) and the target category label c, where (x,y) are the coordinates of the center point of the rectangle, (w,h) are the width and height of the rectangle, and c is the category index.

[0080] S32. Divide the labeled dataset into a training set and a validation set, with 80% as the training set and 20% as the validation set;

[0081] S33, expand the training set by rotation, translation, random cropping and other data augmentation, and the label coordinates also change in this process;

[0082] S34, send the labeled training set picture into the defect positioning model to obtain network output and calculate the loss function; the loss function mainly includes three parts: coordinate loss, confidence loss and category loss; the coordinate loss and the confidence loss both adopt mean square error loss, and the confidence loss and the category loss both adopt cross entropy loss function;

[0083] S35, update the network parameters by the back propagation algorithm until the detection accuracy of the model on the validation set reaches the preset threshold, then stop updating, and obtain the trained defect positioning model.

[0084] S4, the image acquisition module controls the low-resolution camera to take pictures through TCP communication, and the low-resolution camera sequentially collects the regional images of the engine blade in areas according to a fixed sequence, and the adjacent regional images have overlapping parts. Step S4 includes:

[0085] S41, the upstream management system issues a photographing instruction to the image acquisition module through a web interface;

[0086] S42, the image acquisition module controls the low-resolution camera to take pictures through TCP communication;

[0087] S43, after the picture is returned, the image acquisition module and the mechanical control unit control the movement of the object table by a fixed step through TCP communication;

[0088] S44, after the movement is completed, the mechanical control unit notifies the image acquisition module through TCP communication, and then returns to step S42 until all the areas are photographed.

[0089] As shown in Figure 1 The rectangular frame A1, the rectangular frame A2, the rectangular frame A3 and the rectangular frame A4 are four shooting areas of the low-resolution camera, wherein the overlapping areas in the horizontal and vertical directions, the number of rows and the number of columns of the regional images are indicated.

[0090] S5, a plurality of low-resolution regional images collected are grouped into a batch and sent into the trained super-resolution reconstruction neural network model at the same time, to obtain a batch of reconstructed high-resolution regional images.

[0091] S6, a batch of reconstructed high-resolution regional images are sent into the trained fine-grained classification network model in parallel, to obtain the index of the regional images with defects in the current batch.

[0092] S7, extracting corresponding region images from the current batch of high-resolution region images according to the index output by the fine-grained classification network model, and feeding the extracted region images into the trained defect positioning model in parallel to obtain the defect coordinates and defect categories in each region image.

[0093] S8, calculating the defect coordinates on the complete leaf image according to the defect coordinates in the region image, the super-resolution reconstruction ratio, the index of the region image, and the number of overlapping pixels of the adjacent region images in the horizontal and vertical directions.

[0094] In step S8, the coordinates are recovered by the formula: x_orig = col_index * (dw - x_overlap) + x_det / sr_ratio, y_orig = row_index * (dh - y_overlap) + y_det / sr_ratio, w_orig = w_det / sr_ratio, h_orig = h_det / sr_ratio; wherein (x_orig, y_orig, w_orig, h_orig) are the bounding box coordinates of the detected defect on the complete leaf image, (x_orig, y_orig) are the center point coordinates of the bounding box, (w_orig, h_orig) are the width and height of the bounding box; row_index and col_index are the indices of the captured region images, for example, the entire part is divided into k_r * k_c rectangular regions for shooting, k_r represents the number of rows of regions, k_c represents the number of columns of regions, then col_index = 0, 1, 2,..., k_c-1, row_index = 0, 1, 2,..., k_r-1. dw is the pixel width of each leaf region image, dh is the pixel height of each leaf region image, y_overlap and x_overlap are the number of overlapping pixels in the vertical and horizontal directions when the region image is collected.

[0095] Since the region images have overlapping parts, and the defects may exist in the overlapping parts, the non-maximum suppression algorithm is used to remove the repeated annotation boxes in the overlapping regions to obtain the annotation coordinates and category information of all defects on the complete leaf image.

[0096] S9, the region images are spliced to obtain a complete leaf image by an image splicing algorithm. Step S9 includes:

[0097] S91, performing region image preprocessing, performing Gaussian filtering for each row of two adjacent region images on the left and right to remove noise, detecting SIFT key points, defining the left image as a reference image and the right image as a conversion image;

[0098] S92, image registration, by matching the key points of the common parts of the two adjacent regional images on the left and right, the coordinate transformation relationship between the two images is calculated;

[0099] S93, by the coordinate transformation relationship obtained, the converted image is transformed, so that it is in the same coordinate system with the reference image;

[0100] S94, the converted image and the overlapping part of the reference image are fused to obtain the fused image, and the left and right adjacent images are spliced;

[0101] S95, according to the principle method of steps S91-S95, each row of regional images after left and right splicing is spliced up and down to obtain a complete blade image.

[0102] S10, output the defect coordinates, defect categories and complete blade image, the output format is json format, and the blade image is transmitted in the encryption mode of base64.

[0103] The flow of the engine blade micro-defect detection method is to obtain low-resolution images by sequentially traversing and photographing each region on the surface of the blade, send the low-resolution images into the super-resolution reconstruction network model to obtain high-resolution clear images, then send the reconstructed images into the fine-grained classification network model to determine whether there is a defect, send the regional image with defects in the current blade into the defect positioning model to obtain the defect coordinates in the current regional image, and finally restore the coordinates to obtain the defect position on the original image, so that the micro-defects on the engine blade can be accurately detected.

[0104] The present application is not limited to the above-mentioned optional embodiments, anyone can derive other various forms of products under the inspiration of the present application, but regardless of any changes in shape or structure, any technical solutions falling within the scope of the claims of the present application fall within the protection scope of the present application.

Claims

1. An engine blade minute defect detection method characterized by, The method comprises the following steps: S1, training a super-resolution reconstruction network model to obtain a trained super-resolution reconstruction network model; S2, training a fine-grained classification network model to obtain a trained fine-grained classification network model; S3, training a defect positioning model to obtain a trained defect positioning model; S4, sequentially collecting low-resolution regional images of the engine blade by using a low-resolution camera, and adjacent regional images have overlapping parts; S5, inputting the collected multiple low-resolution regional images into the super-resolution reconstruction neural network model in batches to obtain a batch of reconstructed high-resolution regional images; S6, inputting the batch of reconstructed high-resolution regional images into the fine-grained classification network model in parallel to obtain the index of the regional image with defects in the current batch; S7, extracting the corresponding regional image from the high-resolution regional image in the current batch according to the index, and inputting the extracted regional image into the defect positioning model in parallel to obtain the defect coordinates and the defect category in each regional image; S8, calculating the defect coordinates on the complete blade image according to the defect coordinates in the regional image, the super-resolution reconstruction ratio, the index of the regional image, and the number of overlapping pixels of the adjacent regional images in the horizontal and vertical directions; S9, splicing the regional images to obtain a complete blade image; S10, outputting the defect coordinates, the defect category, and the complete blade image.

2. The method of claim 1, wherein Step S1 comprises: S11, using a high-resolution camera to sequentially collect regional images according to the divided regions to obtain a regional image set image_h, collecting images by a degradation method image_l=D(image_h) to form a low-resolution blurred image set image_l, and dividing image_h and image_l into a training set and a validation set; S12, expanding the training set through data augmentation; S13, training a super-resolution reconstruction network model based on a generative adversarial network using the expanded training set, the generative adversarial network comprising a generator g and a discriminator d, using all images in the low-resolution blurred image set image_l as the input of the generator g to obtain a predicted sr_ratio times resolution clear image set image_h_pred=g(image_l); S14, calculate discriminator d loss loss d = -E x∈image_h (log(d(x)))-E y∈image_h_pred (log(1-d(y))), calculate generator g loss loss g = -E y∈image_h_pred (log(d(y))) S15, updating the generator g and the discriminator d through a loss function back propagation until the loss function of the model on the validation set is lower than a preset threshold to stop updating, and obtaining a trained super-resolution reconstruction network model.

3. The method of claim 2, wherein Step S2 comprises: S21, using the generator g in the generative adversarial network to predict the super-resolution reconstruction result of all image_l to obtain a reconstructed image set image_sr; S22, labeling each image in image_sr, labeling a defective image as label=1 and a non-defective image as label=0; S23, dividing the labeled images into a training set and a validation set; S24, expanding the training set through data augmentation; S25, training the fine-grained classification network model by using the expanded training set, and calculating a loss function of the fine-grained classification network model; S26, updating network parameters through the calculated loss function, stopping network updating until the classification accuracy of the model on the verification set reaches a preset threshold, and obtaining the trained fine-grained classification network model.

4. The method of claim 3, wherein Step S25 includes: calculating the loss of the positive sample or the defective sample loss_neg = -(1 - softmax(cls(x_pos))) γ log(cls(x_pos)), Wherein x_pos is a positive sample with label = 1 in image_sr, cls is a fine-grained classification model, and γ is an adjustment parameter. Compute loss for negative samples or defect-free samples loss_neg = -cls(x_neg) γ log(1 - cls(x_neg)), Wherein x_neg is a negative sample with label = 0 in image_sr.

5. The method of claim 3, wherein Step S3 includes: S31, labeling defects in image_sr, marking defects using a minimum circumscribed rectangle and recording defect types, the labeling including rectangular frame coordinate labeling (x, y, w, h) and target category labeling c, wherein (x, y) is the rectangular frame center point coordinate, (w, h) is the rectangular frame width and height, and c is the category index; S32, dividing the labeled data set into a training set and a verification set; S33, expanding the training set through data augmentation; S34, sending the labeled training set picture into the defect positioning model to obtain network output and perform loss function calculation; S35, updating network parameters through a back propagation algorithm until the detection accuracy of the model on the verification set reaches a preset threshold, stopping updating, and obtaining the trained defect positioning model.

6. The method of claim 1, wherein Step S4 includes: S41, issuing a photographing instruction to the image acquisition module; S42, the image acquisition module controls the low-resolution camera to take a picture; S43, after the picture is returned, the image acquisition module and the mechanical control unit control the movement of the object table by a fixed step; S44, after the movement is completed, the mechanical control unit notifies the image acquisition module, and then returns to step S42 until all areas are photographed.

7. The method of claim 1, wherein In step S8, coordinate recovery is achieved through the formula: x_orig = col_index * (dw - x_overlap) + x_det / sr_ratio, y_orig = row_index * (dh - y_overlap) + y_det / sr_ratio, w_orig = w_det / sr_ratio, h_orig = h_det / sr_ratio, Wherein (x_orig, y_orig, w_orig, h_orig) is the boundary box coordinate of the detected defect on the complete leaf image, (x_orig, y_orig) is the boundary box center point coordinate, (w_orig, h_orig) is the boundary box width and height; row_index and col_index are the indexes of the photographed area image, dw is the pixel width of each leaf area image, dh is the pixel height of each leaf area image, y_overlap and x_overlap are the number of overlapping pixels in the vertical direction and the horizontal direction when the area image is collected, respectively. ​ 8. The method of claim 1, wherein The step S8 further comprises: removing the repeatedly labeled bounding boxes of the overlapping area by a non-maximum suppression algorithm to obtain the labeling coordinates and category information of all defects in the complete leaf image.

9. The method of claim 1, wherein The step S9 comprises: S91, performing region image preprocessing, performing Gaussian filtering to remove noise on each row of two region images adjacent to left and right, detecting SIFT key points, defining the left image as a reference image and the right image as a converted image; S92, image registration, calculating the coordinate transformation relationship between the two images by matching the key points of the common part of the two region images adjacent to left and right; S93, transforming the converted image according to the obtained coordinate transformation relationship so that it is in the same coordinate system as the reference image; S94, fusing the overlapping part of the converted image and the reference image to obtain a fused image, and completing the splicing of the left and right adjacent images; S95, splicing each row of region images after left and right splicing to obtain a complete leaf image.

10. The method of claim 1, wherein: In the step S10, the output format of the defect coordinates, the defect category and the complete leaf image is a json format, and the leaf image is transmitted in an encryption manner of base64.

Images