Onboard intelligent detection method for aircraft engine blade damage

Abstract

The invention discloses an onboard intelligent detection method for aircraft engine blade damage, and the method comprises the steps: collecting a plurality of damage images of aircraft engine blades, and marking the damaged parts; constructing a blade damage data set based on the damage image, dividing the blade damage data set into a training set and a test set according to a predetermined proportion, and inputting the training set into an intelligent detection model for training; after the training is completed, obtaining a PT model, and converting the PT model into an onx model and an RKNN model; a detection unit interface is designed through QT software, opencv and a related third-party library are added, and the RKNN model is input into the QT software; and copying the executable file which is compiled by the QT and the library file which needs to be used to the RK3588S development board which is burnt with the Ubuntu system. The camera is connected to the development board with the packaged system and program, the detection unit is started and started, and the start is clicked, so that the real-time detection and identification of the blade damage of the aero-engine can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision detection for aero - engines, and particularly to an on - board intelligent detection method for blade damage of aero - engines. Background Art

[0002] As a means of aero - engine condition monitoring, borescope inspection is widely used in the field of aero - engine detection and plays a very important role in ensuring aviation flight safety. However, although borescope monitoring is nearly mature, there are still the following aspects that can be improved:

[0003] In terms of image acquisition, conventional borescope inspection is limited to image and video acquisition and lacks the ability to process image and video data in real - time. When conducting on - site engine inspections, inspectors often only take photos or record videos of the damaged parts first and then, after the event, determine the type of damage by analyzing the images or videos and write an evaluation analysis report, which is used as a basis for later maintenance and monitoring of engine damage. This results in the lack of timeliness in borescope inspection, being unable to process aero - engine damage image data in real - time and with low detection efficiency.

[0004] In terms of detection accuracy, conventional borescope inspection mainly relies on manual visual inspection. The detection results are limited by the professional skills and experience of inspectors. Inspectors need to have proficient borescope inspection experience to accurately find the damage defects of engine blades and determine the defect categories based on experience and manuals for the collected damage pictures. However, there are still relatively few experienced borescope experts, making off - site inspections difficult and increasing the time cost of detection.

[0005] In terms of detection judgment, conventional borescope inspection judgment mainly relies on borescope inspectors, and factors such as the mental state of inspectors, evaluation criteria, and quality standards of inspectors all have certain impacts on the detection results, which may lead to misdiagnosis and missed diagnosis of the detection results and it is difficult to ensure the accuracy of the detection results.

[0006] The above information disclosed in the background art section is only used to enhance the understanding of the background of the present invention and may therefore include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0007] The present invention provides an on - board intelligent detection method for blade damage of aero - engines, which realizes real - time detection and identification of blade damage of aero - engines with high identification accuracy.

[0008] The on - board intelligent detection method for blade damage of aero - engines includes:

[0009] Step S1, collecting damage images of multiple aero - engine blades and marking the damaged parts;

[0010] Step S2: Based on the damage images, construct a blade damage dataset, and divide it into a training set and a test set according to a predetermined ratio. The training set is input into the intelligent detection model for training;

[0011] Step S3: After the training is completed, obtain a model file in PT format. First, convert it into a model file in onnx format, and then convert the converted onnx model file into a model file in RKNN format;

[0012] Step S4: Design the interface of the detection unit through the QT interface design tool, add the opencv library, RGA library, and rknn_api C library, and input the RKNN model into the QT project. After the QT project design is completed, perform compilation and construction to generate an executable file of the QT project;

[0013] Step S5: Copy the executable file compiled by QT and the required library files to the development board for deployment, such as the RK3588S development board with the Ubuntu system already burned;

[0014] Step S6: Connect the camera to the development board with the program and system encapsulated, start and open the detection unit, and click the "Start" button to realize the real-time detection and recognition of the damage of the aero-engine blade.

[0015] In the on-board intelligent detection method for the damage of the aero-engine blade, in step S1, constructing the blade damage dataset includes,

[0016] Collect the blade damage images on-site in the field, and collect various blade damage images of multiple aero-engine blades;

[0017] Sort out and classify the collected damage image data, and determine the blade damage categories. The damage categories involved in the present invention include chipping, breakage, ablation, and crack;

[0018] Use an image annotation tool to annotate the sorted damage picture data, mark the damage positions and export them.

[0019] In the on-board intelligent detection method for the damage of the aero-engine blade, in step S2, construct a blade damage dataset based on the damage image data and the annotation data.

[0020] In the on-board intelligent detection method for the damage of aero-engine blades, in step S2, the blade damage data set is divided into a training set and a test set according to a ratio of 7:3, and the training set is input into the intelligent detection model for training. The intelligent detection model includes an Input input part, a Backbone part, and a Head part; the Head part includes a Neck part and a Detect part. The Backbone part uses New CSPDarkNet-53 as the backbone, and SPPF and New CSP-PAN as the feature fusion network, and three detection heads are used to predict targets of different scales. In the backbone network, the CBS module contains a 1×1 convolutional layer, a batch normalization layer, and a Silu activation function layer. In addition to extracting features from the image, it also performs downsampling through 3×3 or 6×6 stride convolutions to reduce the size of the feature map. The C3 module is composed of three CBS modules and a Bottleneck module. After entering the C3 module, the input is divided into two paths. One path passes through a CBS and a Bottleneck, and the other path only passes through a CBS. Finally, the two paths are connected together and then pass through a CBS to further make the feature extraction more sufficient. The feature map output by New CSPDarkNet-53 is first output to the SPFF, and after further max pooling, it is connected to obtain a feature map with a larger number of channels. Then, through the CBS module, the number of channels will return to the number of channels of the feature map output by New CSPDarkNet-53; for the model input part, the damaged image is scaled to the required size, padded with black edges at the same time, and normalized.

[0021] Further, in the on-board intelligent detection method for the damage of aero-engine blades, the size of the damaged image input for model training is 640*640, the training batch size is 16, the initial number of training iteration rounds epochs is 100, the learning rate is 0.01, and two different weight models will be generated after the training iteration ends: the best model weight parameter file and the model weight parameter file after the last training iteration ends, the training fitting curve, and the detection results of the damaged images in the first three batches. Among them, if the training fitting curve does not converge, it is necessary to increase the number of training iteration rounds and retrain.

[0022] In the on-board intelligent detection method for damage of aero-engine blades, in step S3, after the training is completed and the fitting curve converges, the test set is input into the detect function and the best.pt model obtained through training is inferred to further evaluate the feasibility of the model. If the model test effect is qualified and the recognition bounding box is accurate, the best model weight parameter file in PT format obtained after the training is first converted into an onnx format model file through the export function, and then the converted onnx model file is imported into the netron visualization tool to be opened to check whether its network structure output conforms. If it conforms, the converted onnx model file is converted into an RKNN format model file through the RKNN toolkit.

[0023] In the on-board intelligent detection method for damage of aero-engine blades, in step S4, the detection unit interface is designed based on the finally converted RKNN model through the QT interface design tool. Among them, before the software interface design, the QT compilation environment is also configured, and the compilation kit is configured so that the construction can be carried out after the interface design is completed. Secondly, the opencv source code is compiled through the cross-compilation chain tool, and the generated opnecv library file after compilation is added to the QT project file to be designed, and the RGA library and rknn_api C library related to the RKNN model are added. The designed interface program project is compiled and built to generate an executable file of the QT project.

[0024] In the on-board intelligent detection method for damage of aero-engine blades, in step S5, the executable file compiled and built by QT, as well as the used opencv library file, RGA library file and rknn_api C library file are copied to the development board for deployment, such as the RK3588S development board with the Ubuntu system already burned. The designed on-board detection unit is implemented as a GUI interface to realize the detection function in real time and the program running process is as follows: First, the aero-engine blades to be detected are read frame by frame through the USB camera module. The size of the readable picture is 640*480. The read image is further filled with black edges through the resize preprocessing function to be filled to the 640ⅹ640 image size supported by the model and converted into the RGB format suitable for the model. Then the processed image is input into the converted RKNN model for inference. After the model inference, data including bounding box coordinates, label categories and confidence levels are output. Then, it is compared with the QT image display area. The preprocessed picture is scaled to the fixed size of the detection interface display image area, and the damage category, border and confidence level are drawn on the image through the cv::rectangle and cv::putText functions of the opencv library. At the same time, the blade picture with the detection box, damage category and confidence level is displayed in the QT image display area.

[0025] In the on-board intelligent detection method for damage of aero-engine blades, the on-board detection unit is implemented as a GUI interface to realize the detection function in real time and includes the detection of damage of aero-engine blades, taking pictures, saving and recording the damage detection results, and viewing and playing the pictures and videos of the damage detection results. By calling a USB camera, each frame of the blade image is inferred in real time, and the results are displayed in the interface.

[0026] Compared with the prior art, the present invention has the following advantages: By applying an on-board algorithm for damage of aero-engine blades to the borescope technology, the present invention realizes the real-time detection and intelligence of damage of aero-engine blades, improving the efficiency and accuracy of borescope detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] By reading the following detailed description of the preferred embodiments, various other advantages and benefits of the present invention will become clear to those of ordinary skill in the art. The accompanying drawings in the specification are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Obviously, the following described drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts. Moreover, throughout the drawings, the same reference numerals are used to represent the same components.

[0028] In the drawings:

[0029] Figure 1 is the technical roadmap of an on-board intelligent detection method for damage of aero-engine blades according to the present invention;

[0030] Figure 2 is the training effect diagram of an intelligent detection and recognition algorithm for damage of aero-engine blades according to the present invention;

[0031] Figure 3 is the network structure diagram of an intelligent detection model for damage of aero-engine blades according to the present invention.

[0032] The present invention will be further explained below with reference to the drawings and embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] The specific embodiments of the present invention will be described in more detail below with reference to the drawings. Although the specific embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0034] It should be noted that in the specification and claims, certain terms are used to refer to specific components. Those skilled in the art should understand that technicians may use different nouns to refer to the same component. The specification and claims do not distinguish components by the difference in nouns, but by the difference in the functions of the components. As mentioned throughout the specification and claims, "comprising" or "including" is an open-ended term and should be interpreted as "including but not limited to". The subsequent description in the specification is the preferred implementation manner for implementing the present invention, but the description is for the purpose of the general principles of the specification and is not used to limit the scope of the present invention. The protection scope of the present invention shall be subject to that defined by the appended claims.

[0035] For the convenience of understanding the embodiments of the present invention, the following will further explain with specific embodiments in conjunction with the drawings, and each drawing does not constitute a limitation to the embodiments of the present invention.

[0036] As Figures 1 to 3 shown, the on-board intelligent detection method for damage of aero-engine blades includes the following steps:

[0037] Step S1, collect damage images of multiple aero-engine blades and label the damaged parts;

[0038] Step S2, construct a blade damage data set based on the damage images, and divide it into a training set and a test set according to a predetermined ratio. The training set is input into the intelligent detection model for training;

[0039] Step S3, after the training is completed, obtain a model file in PT format, first convert it into a model file in onnx format, and then convert the converted onnx model file into a model file in RKNN format;

[0040] Step S4, design the interface of the detection unit through the QT interface design tool, add the opencv library, RGA library and rknn_api C library, and input the RKNN model into the QT project;

[0041] Step S5, copy the executable file compiled by QT, as well as the pencv library, RGA library and rknn_apiC library files that need to be used, to the RK3588S development board with the Ubuntu system already burned;

[0042] Step S6, connect the camera to the development board with the program and system encapsulated, start and open the detection unit, and click the "Start" button to realize real-time detection and recognition of aero-engine blade damage.

[0043] In the preferred implementation manner of the on-board intelligent detection method for damage of aero-engine blades, step S1 includes

[0044] Collect images of blade damage in the field, and collect various images of blade damage of multiple aero-engine blades;

[0045] Sort and classify the collected damage image data, and determine the blade damage categories. The damage categories involved in the present invention include chipping, breakage, ablation, and cracks;

[0046] Label the sorted damage picture data through an image annotation tool, mark the damage positions and export them as.

[0047] In the preferred implementation of the on-board intelligent detection method for aero-engine blade damage, in step S2, a blade damage data set is constructed based on the damage image data and annotation data. The blade damage data set is divided into a training set and a test set according to a ratio of 7:3, and the training set is input into the intelligent detection model for training. The intelligent detection model includes an Input input part, a Backbone part, and a Head part; the Head part includes a Neck part and a Detect part. The Backbone part uses New CSPDarkNet-53 as the backbone, and SPPF and New CSP-PAN as the feature fusion network, and three detection heads are used to predict targets of different scales. In the backbone network, the CBS module contains a 1×1 convolutional layer, a batch normalization layer, and a Silu activation function layer. In addition to extracting features from the image, it also performs downsampling through 3×3 or 6×6 stride convolutions to reduce the size of the feature map. The C3 module consists of three CBS modules and a Bottleneck module. After entering the C3 module, the input is divided into two paths. One path passes through a CBS and a Bottleneck, and the other path only passes through a CBS. Finally, the two paths are connected together and then pass through a CBS to further make the feature extraction more sufficient. The feature map output by NewCSPDarkNet-53 is first output to the SPFF, and after further max pooling, a feature map with a larger number of channels is connected. For example, the 20×20×1024 feature map output by New CSPDarkNet-53 is output to the SPFF, and after further max pooling, a feature map with a scale of 20×20×4096 is connected. Then, through the CBS module, the number of channels will return to 1024. For the model input part, the damage pictures are scaled to the required size, black edges are filled at the same time, and normalization operations are performed.

[0048] Further, in step S2, the size of the damaged images input for model training is 640*640, the training batch size is 16, the initial number of training epochs is 100, the learning rate is 0.01. At the end of the training iteration, two different weight models will be generated: the best model weight parameter file and the model weight parameter file after the last training iteration, the training fitting curve, and the detection results of the damaged images in the first three batches. Among them, if the training fitting curve does not converge, the number of training epochs needs to be increased and the training is restarted.

[0049] In the preferred embodiment of the on-board intelligent detection method for aero-engine blade damage, in step S3, after the training is completed and the fitting curve converges, the test set is input into the detect function and inference is performed on the best model weight parameter file obtained from the training to further evaluate the feasibility of the model; if the model test effect is qualified and the recognition bounding box is accurate, the best model weight parameter file in PT format obtained after the training is first converted into an onnx format model file through the export function, and then the converted onnx model file is imported into the netron visualization tool to be opened to check whether its network structure output conforms. If it conforms, the converted onnx model file is converted into an RKNN format model file through the RKNN tool kit.

[0050] In the preferred embodiment of the on-board intelligent detection method for aero-engine blade damage, in step S4, based on the finally converted RKNN model, the detection unit interface is designed through the QT interface design tool. Among them, before the software interface design, the QT compilation environment is also configured, and the compilation kit is configured so that the construction can be carried out after the interface design is completed; secondly, the opencv source code is compiled through the cross-compilation chain tool, and the generated opnecv library file after the compilation is added to the QT project file to be designed, and the RGA library and rknn_api C library related to the RKNN model are added. The designed interface program project is compiled and built to generate the executable file of the QT project.

[0051] In the preferred embodiment of the on-board intelligent detection method for aero-engine blade damage, the detection unit interface reads the aero-engine blade to be detected through the USB camera module, and the read image is scaled through the resize preprocessing function to the fixed size of the image display area on the detection interface and converted into the RGB format suitable for the model; the image is inferred by the RKNN model to output the bounding box coordinates and label categories, and the damage category, border, and confidence are drawn and displayed on the image to achieve real-time blade damage detection.

[0052] In the preferred embodiment of the on-board intelligent detection method for aero-engine blade damage, the executable file compiled and built by QT, as well as the used opencv library file, RGA library file, and rknn_api C library file, are copied to the development board, such as the RK3588S development board with the Ubuntu system already burned. The designed on-board detection unit is implemented as a GUI interface to realize the detection function in real time, and the program running process is as follows: First, the aero-engine blade to be detected is read frame by frame through the USB camera module. The size of the readable picture is 640*480. The read image is further filled with black edges through the resize preprocessing function to be filled to the 640ⅹ640 image size supported by the model, and is converted to the RGB format suitable for the model. Then, the processed image is input into the converted RKNN model for inference. After the model inference, data including bounding box coordinates, label categories, and confidence levels are output. Then, it is compared with the QT image display area. The preprocessed picture is scaled to the fixed size of the detection interface display image area, and the damage category, border, and confidence level are drawn on the image through the cv::rectangle and cv::putText functions of the opencv library. At the same time, the blade picture with the detection box, damage category, and confidence level is displayed in the QT image display area.

[0053] The on-board detection unit is implemented as a GUI interface to realize the detection function in real time and includes aero-engine blade damage detection, taking pictures, saving, and recording of damage detection results, and viewing and playing functions of damage detection result pictures and videos, so as to perform inference on each frame of blade image in real time by calling the USB camera and display the results in the interface.

[0054] In one embodiment, the on-board intelligent detection method includes the following steps:

[0055] Step S1, collect damage images of multiple aero-engine blades and label the damaged parts;

[0056] Step S2, construct a blade damage data set based on the damage images and divide it into a training set and a test set according to a predetermined ratio. The training set is input into the intelligent detection model for training;

[0057] After the training is completed, a model file in PT format is obtained. First, it is converted into a model file in onnx format, and the converted onnx model file is then converted into a model file in RKNN format;

[0058] Step S4: Design the interface of the detection unit using the QT interface design tool, add the opencv library, RGA library, and rknn_api C library, input the RKNN format model into the QT project, and after the QT project design is completed, perform compilation and build to generate an executable file for the QT project;

[0059] Step S5: Copy the executable file compiled by QT, as well as the opencv library, RGA library, and rknn_api C library files required, to the development board, such as the RK3588S development board with the Ubuntu system already burned;

[0060] Step S6: Connect the camera to the development board with the program and system encapsulated, start and open the detection unit, and click the "Start" button to achieve real-time detection and recognition of aero-engine blade damage.

[0061] The blade damage dataset includes four damage category labels: breakage, ablation, chipping, and crack. It is achieved through annotation using an image annotation tool. The dataset is divided into a training set and a test set in a ratio of 7:3 and input into the intelligent detection lightweight model for training. The input size of the aero-engine blade damage image data for model training is 640*640, the training batch size is 16, the initial number of training epochs is set to 100, the learning rate is 0.01, and the training will generate two weight models, a training fitting curve, and the detection results of the damage images in the first three batches. If the fitting curve does not converge, the number of training epochs needs to be increased and retrained, otherwise it will cause random boxes to appear in the test of the converted onnx model. After training is completed, the best weight parameter model best.pt is obtained, and this model is converted into an onnx model through the model conversion file export.py. The converted onnx model is opened using the netron visualization tool to check whether its output conforms. The three feature map data output by the present invention are 1*27*80*80, 1*27*40*40, and 1*27*20*20. The conversion of the Onnx model to RKNN needs to be carried out in the RKNN environment. Before conversion, the RKNN toolkit needs to be built and installed first, and the rknn-toolkit2 environment needs to be configured. Then, the convert interface is called in the rknn environment to convert the onnx model into an rknn model.

[0062] Before the interface design of the QT detection unit, the opencv source code is first compiled through the cross-compilation chain tool. The source code version of this invention is opencv4.5.0. Add the generated lib library file after compilation to the QT project file; at the same time, add the library files related to the RKNN model. The QT detection unit interface mainly includes functions such as the detection of aero-engine blade damage, taking pictures, saving and recording the damage detection results, and viewing and playing the pictures and videos of the damage detection results. This invention uses a USB camera to read images, and its detection operation process includes the following steps: The USB camera reads pictures frame by frame, and the size of the pictures that can be read is 640*480; the read picture data is scaled using the resize preprocessing function to the fixed size of the image display area in the detection interface and filled; it is converted to the RGB format suitable for the rknn model using RGA hardware acceleration. The preprocessed picture data is input into the converted rknn format model for inference. The model inference outputs labels, confidence levels, and bounding box coordinates. After completion, the preprocessed pictures are compared with the QT image display area, scaled to the specified display area size, and the labels, confidence levels, and bounding box coordinates output after inference are used to draw bounding boxes on the read image using the cv::rectangle and cv::putText functions of the opencv library, print the damage category and confidence level, and at the same time display the results in the QT image display area.

[0063] After the interface design of the QT detection unit is compiled to generate an executable program, copy the program and the opencv library, the RGA library related to the RKNN model, and the rknn_api C library file required for the program to run to the RK3588S development board installed with the Ubuntu system, and connect the camera module to achieve real-time detection of aero-engine blades. In addition, with the various functions of the detection interface, the detection efficiency can be further improved.

[0064] Although the embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields. The above specific embodiments are merely illustrative and guiding, rather than restrictive. Those of ordinary skill in the art can also make many forms under the inspiration of this specification and without departing from the scope protected by the claims of the present invention, and these all belong to the scope of protection of the present invention.

Claims

1. An onboard intelligent detection method for aircraft engine blade damage, characterized in that: The method comprises the following steps: Step S1, collecting damage images of multiple aircraft engine blades and marking the damaged parts; Step S2, constructing a blade damage data set based on the damage image, and dividing it into a training set and a test set according to a predetermined ratio, and inputting the training set into an intelligent detection model for training; Step S3, after the training is completed, the model file in PT format is obtained, which is first converted into a model file in onnx format, and the converted onnx model file is then converted into a model file in RKNN format; Step S4, design the detection unit interface through the QT interface design tool, add the opencv library, RGA library and rknn_api C library, input the RKNN format model into the QT project, and after the QT project design is completed, compile and build to generate an executable file of the QT project; Step S5, copy the executable file compiled by the QT interface design tool, as well as the required opencv library, RGA library and rknn_api C library files to the development board for deployment; Step S6, connect the camera to the development board that encapsulates the system and program, start and open the detection unit, click the "Start" button, and realize real-time detection and identification of aircraft engine blade damage.

2. The onboard intelligent detection method for aircraft engine blade damage according to claim 1, characterized in that: Preferably, in step S1, a plurality of blade damage images of a plurality of aircraft engine blades are collected by collecting blade damage images in the field; secondly, the collected damage image data are sorted and classified to determine the blade damage category, which includes chipping, breakage, ablation and cracking; and then the sorted damage image data are annotated by an image annotation tool, and the damage location is annotated and exported.

3. The onboard intelligent detection method for aircraft engine blade damage according to claim 2, characterized in that: A blade damage dataset is constructed based on the damage image data and the annotation data.

4. The onboard intelligent detection method for aircraft engine blade damage according to claim 3, characterized in that: The size of the damage image input for model training is 640*640, the training batch size is 16, the initial training iteration number of epochs is 100, and the learning rate is 0.

01.

5. The onboard intelligent detection method for aircraft engine blade damage according to claim 4, characterized in that: At the end of the training iteration, two different weight models will be generated: the best model weight parameter file and the model weight parameter file after the last training iteration.

6. The onboard intelligent detection method for aircraft engine blade damage according to claim 4, characterized in that: After the training is completed and the fitting curve converges, the test set is input into the detect function and the trained best.pt model is inferred to further evaluate the feasibility of the model.

7. The onboard intelligent detection method for aircraft engine blade damage according to claim 6, characterized in that: Based on the RKNN model obtained by the final conversion, the detection unit interface is designed using the QT interface design tool.

8. The onboard intelligent detection method for aircraft engine blade damage according to claim 7, characterized in that: Copy the executable file compiled by QT, as well as the used opencv library file, RGA library file, and rknn_api C library file to the development board for deployment.

9. The onboard intelligent detection method for aircraft engine blade damage according to claim 8, characterized in that: The onboard detection unit is implemented as a GUI interface to realize the detection function in real time and includes aircraft engine blade damage detection, taking pictures, saving and recording damage detection results, and viewing and playing pictures and videos of damage detection results.

Images