Borescope image damage recognition model and application system based on improved convolutional neural network
By improving the convolutional neural network, combining ResNet-152 and Darknet-53 networks, using the ECA attention mechanism and PAFPNet module, the problem of reduced image quality and insufficient recognition accuracy in hole probe image recognition is solved, and the problem of in-aviation engines is realized in the aircraft engine Efficient and accurate damage identification.
Patent Information
- Application Number
- CN202411011788.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-26
AI Technical Summary
The acquisition, transmission and conversion of hole probe images in the prior art result in a decrease in image quality, and most of them are for artificial damage. The real working image of the wing engine is not selected for judgment, resulting in the recognition accuracy, speed and false alarm rate that cannot meet practical requirements. .
The improved convolutional neural network is adopted, including the backbone feature extraction network, the enhanced feature extraction network and the prediction network. The ResNet-152 and Darknet-53 networks combine the ECA attention mechanism and the PAFPNet module to extract and fuse features to achieve different angles and different angles. The identification of pixels and different damage types reduces the amount of parameters and calculation complexity, and enhances the generalization ability and robustness of the model.
It realizes accurate identification of ablation, crack and other damage under actual working conditions, reduces false alarm rates, improves processing speed, meets the actual working needs of aircraft engine hole detection image recognition, and improves the accuracy and adaptability of hardware requirements.
Smart Images

Figure CN118537708B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a borescope damage recognition model, and more specifically, the present invention mainly relates to a borescope image damage recognition model and an application system based on an improved convolutional neural network. Background Art
[0002] Convolutional neural network is an artificial intelligence model built with bionic visual perception as the mechanism goal. It is a type of feedforward neural network that includes convolution calculation and deep network. It is one of the representative models of deep learning and is also the mainstream basic algorithm model of current image processing. The aircraft engine borescope inspection technology is a technology that uses borescope equipment to inspect the internal structure of the engine in detail without disassembling the engine. It is the core supporting technology for quality control and reducing the overall cost during the entire life cycle of aircraft engines. It plays an indispensable role in the qualitative or quantitative evaluation of internal structural damage of aircraft engines.
[0003] During the on-wing operation of the engine, a borescope is required to conduct qualitative and quantitative assessments of whether the engine is damaged and whether the damage exceeds the standard. This process currently needs to be completed manually. However, due to the complex internal structure and harsh working environment of aircraft engines, as the flight time increases, its internal blades and linings are prone to suspected damage such as paint peeling and carbon deposition on the edges. This situation greatly increases the working hours and judgment difficulty of the staff, and is prone to dangerous situations such as misjudgment and missed judgment. At the same time, during the process of aircraft engine borescope, affected by the endoscope hardware facilities, personnel operation, and the internal environment of the engine, the acquisition, transmission and conversion process of the borescope image is inevitably accompanied by a decrease in image quality, which is also easy to affect the detection rate of damage. In addition, domestic airlines currently do not have real-time monitoring technology for aircraft engine borescopes that meets the expected accuracy requirements and has been put into production.
[0004] Although there are many papers on related topics published, they are basically limited to theory. The accuracy, recognition speed, false alarm rate, hardware requirements, etc. of their models cannot meet practical requirements at the same time. Most of them target artificial damage and do not select real working images of on-wing engines for judgment. The videos and pictures used have high clarity and are accurately focused on the defect locations. Most of the defects are late defects and occupy too large a proportion of the image without interference display. They are not obtained from normal working shots, but are mostly posed and targeted shots, and their accuracy has little reference value.
[0005] For example, the Chinese invention patent with publication number CN109800708A uses a fully convolutional neural network to identify damage in aircraft engine borescope images. Its disadvantages are: 1. The borescope images need to be preprocessed to meet training and testing requirements; 2. Key information such as detection rate, accuracy, operating speed, and false alarm rate are not specified; 3. It is only an application innovation, and no on-demand improvements are made to the engine borescope images.
[0006] Another example is the Chinese invention patent with publication number CN114842192A. This patent improves the backbone structure of YOLO V4, realizes the recognition of three types of defects in borescope images and videos, and completes the construction of the application. Its disadvantages are: 1. The image used is not the actual working perspective, the defect accounts for too large a proportion of the image, the clarity of such images is too high, which is not the normal level of clarity of borescope defects, and there are no interference items in the rest of the image; 2. Different damages are displayed in the form of box selection, which is not conducive to observing the direction of crack extension, and the following characteristics in the video processing process are not explained; 3. The video frame rate improvement brought by the optimization mentioned in the patent has no application and reference value in the actual borescope process; 4. The amount of data used in this patent is large, so it takes a lot of time to complete the model training. This problem is not mentioned in the patent.
[0007] Another example is the preliminary study on the application of image processing technology in the determination of borescope inspection results of aircraft engines published in Aviation Maintenance and Engineering. The document performs multiple preprocessing on the borescope image to make the cracks and other related displays more obvious, thereby achieving the purpose of assisting the completion of borescope monitoring. The disadvantages are as follows: 1. The image preprocessing cannot be completed adaptively; 2. The overall research stops at the image processing stage, and no further attempts are made on the adaptive recognition of image defects.
[0008] Based on the survey of the above-mentioned existing technologies, it is found that it is necessary to make a series of targeted optimizations and adjustments to the damage identification network design and system architecture according to the damage characteristics, so as to solve the problems of high complexity, low recognition speed and poor recognition speed of existing detection methods and models. Summary of the invention
[0009] One of the purposes of the present invention is to address the above-mentioned shortcomings and provide a borescope image damage recognition model and application system based on an improved convolutional neural network, in the hope of solving the technical problems that the acquisition, transmission and conversion processes of borescope images in the prior art are inevitably accompanied by a decrease in image quality, and most of the research on aircraft engine damage is aimed at artificial damage, and the real working images of the on-wing engine are not selected for judgment.
[0010] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0011] On one hand, the present invention provides a borescope image damage recognition model based on an improved convolutional neural network, and the borescope image damage recognition model includes:
[0012] The backbone feature extraction network is used to extract preliminary features of the actual borescope image; the backbone feature extraction network includes a ResNet-152 network and a Darknet-53 network. The ResNet-152 network and the Darknet-53 network are respectively applied to corresponding damage identification requirements based on their differences in convolution kernel size, channel number restrictions, network layer number, and iteration time.
[0013] The above-mentioned ResNet-152 network is a ResNet-152 network optimized by introducing the ECA attention mechanism, and the above-mentioned Darknet-53 network is a Darknet-53 network in which the C3 module is replaced by the C2f module. In the application, the above-mentioned two networks are both networks after borescope image damage optimization.
[0014] The enhanced feature extraction network is connected to the backbone feature extraction network to fuse and filter the preliminary features, extract effective features therein and output them to the prediction network.
[0015] The enhanced feature extraction network includes a PAFPNet module, which is used to aggregate shallow feature maps and deep feature maps, shorten the information path between bottom-level features and top-level features, and enhance the expression capability of multi-scale features.
[0016] The above-mentioned PAFPNet module is a PAFPNet module (Path Aggregation Feature Pyramid Network, PAFPNet) that combines the Feature Pyramid Networks (FPNet) with the bottom-up Path Aggregation Networks (PANet). Based on the top-down strong semantic feature communication of the conventional FPNet, this module adds the bottom-up strong positioning feature communication of PANet, aggregates the shallow feature maps (low resolution but weak semantic information) and the deep feature maps (high resolution but rich semantic information), shortens the information path between the bottom features and the top features, and further enhances the expression ability of multi-scale features.
[0017] The prediction network is connected to the enhanced feature extraction network, which is used to utilize the extracted effective features to complete the extraction of target position and category information through decoupling, loss calculation and proportional weighting; and then determine the damage information in the borescope image in the unknown state, wherein the damage information includes damage type, damage size and damage position.
[0018] The prediction network includes a decoupling head without an anchor frame and a loss calculation module, which is used for extracting the position and category information of the aggregated effective features by the decoupling head, inputting them into the corresponding network branches for loss calculation for learning respectively, and then outputting the final result by proportional weighting; thereby realizing parallel calculation at different scales, effectively reducing the number of parameters and calculation complexity, and enhancing the generalization ability and robustness of the model.
[0019] Preferably, a further technical solution is: the backbone feature extraction network extracts preliminary features of the actual borescope image through the training set in the labeled data set; the labeled data set is obtained by: obtaining a sufficient number of actual borescope images and videos, and capturing and screening images containing different damage types as the original data set; marking the damage through closed geometric shapes and recording the corresponding damage categories together, using the marked damage images as the labeled data set, and the unlabeled images as the damage-free data set.
[0020] A further technical solution is that: the labeled data set includes a test set, a training set and a validation set, and the labeled images in the labeled data set correspond to the names of the corresponding label files one by one, and the test set, training set and validation set are respectively stored in corresponding subdirectories; the label files include the following information in sequence: damage type label, Arabic integers with 0 as the initial value, each number represents a damage type; coordinates of each label point, every two numbers is a group, representing the position of the marker point in the image, and the coordinates of the marker point are normalized by the formula , Calculated, where is the original pixel data of the marker point in the image, The height and width of the image in pixels.
[0021] A further technical solution is: the effective feature extraction method of the enhanced feature extraction network is to obtain a previous fused feature layer through lateral connection and feature fusion from top to bottom, and then obtain a further fused feature layer through lateral connection and feature fusion from bottom to top, and repeatedly repeat the effective feature extraction method.
[0022] A further technical solution is that the labeled data set is used for training and verification of a damage identification network, which extracts damage target location and category information by decoupling feature layers, calculating losses and weighting proportions, thereby realizing identification, prediction and verification of network performance;
[0023] The loss calculation function includes the classification loss function Classification Loss, L ClsAnd regression positioning loss function Regression Loss,L Reg ;
[0024] The classification loss function Classification Loss,L Cls The calculation formula is:
[0025]
[0026] In the formula, q The intersection over union (IoU) of the predicted box Bounding box (Bbox) and the ground truth box (GTbox) of any damage category is the intersection of the predicted box Bbox and the ground truth box (GTbox) divided by the union of the two boxes, which represents the overlap between the predicted box Bbox and the ground truth box GTbox. p is the probability of this damage type, γ is an adjustable factor, with a value range of [0,5]. α is the weight coefficient, and its value range is [0,1].
[0027] The regression positioning loss function Regression Loss, L Reg The calculation formula is:
[0028]
[0029] In the formula, are the center points of Bbox and GTbox respectively, p is the Euclidean distance between the center points of Bbox and GTbox, c is the diagonal distance of the closed area of the two rectangular frames, v is the consistency of the relative proportions of the two rectangular frames, β is the weight coefficient, and its value range is [0,1]. q is the intersection-over-union ratio of Bbox and GTbox of any damage category.
[0030] A further technical solution is: the borescope image damage recognition model is also used to identify real machine borescope images with different viewing angles and different occlusion conditions, and to assist in counting the blades in the real machine borescope images; the backbone feature extraction network performs preliminary feature extraction on the labeled data set and then inputs it into the enhanced feature extraction network, and the enhanced network fuses and screens the different scale features output by the backbone feature extraction network to form a fused feature layer, and repeatedly extracts the available feature information of the fused feature layer; the blade auxiliary counting is to follow and capture the real machine blade image appearing in the video, and complete the bidirectional counting of the real machine blades by counting the disappearance in the left and lower screen range as an increase, and the disappearance in the right and upper screen range as a decrease.
[0031] A further technical solution is that during the counting process of the blade auxiliary counting, the intersection-and-combination ratio threshold is controlled within [0.15, 0.25] according to the actual blade ablation state.
[0032] On the other hand, the present invention provides an application system of the above-mentioned borescope image damage recognition model, the system includes a camera video frame real-time capture and recognition module, a video frame damage recognition module, an image damage recognition module and a folder batch processing module; the camera video frame real-time capture and recognition module is used to identify and capture defective borescope images in the current video; the video frame damage recognition module and the image damage recognition module are used to respectively identify and mark the damage positions in the video frame and the image through the borescope image damage recognition model; the folder batch processing module is used to batch process all identifiable videos and images in the corresponding folder.
[0033] Based on the borescope image damage recognition model described above, it is also possible to pause the recognition at any time in the application; the found damage image or video screenshot and the comparison image containing only the damaged area can be saved as viewable pictures respectively; the damage information and the corresponding damage image or video screenshot can be saved as a damage data set and used to expand the training set in the labeled data set.
[0034] Compared with the prior art, one of the beneficial effects of the present invention is that based on the characteristics of different damages in the working borescope images of actual blades and the interference of the surrounding environment, the feature extraction network in the convolutional neural network is selected in a targeted manner, as well as attention improvement and selection of calculation weights, so that the improved network model can achieve the accuracy, false alarm rate, processing speed and hardware requirements required for actual work when performing continuous image recognition of damage at different angles, different pixels, different damage types and different periods. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 The figure is a schematic diagram of the structure of a borescope image damage recognition model for illustrating an embodiment of the present invention.
[0036] Figure 2 A first comparative display diagram of damage marks for illustrating an embodiment of the present invention.
[0037] Figure 3 A second comparative illustration of damage marks for illustrating an embodiment of the present invention.
[0038] Figure 4 A diagram showing a blade counting function for illustrating an embodiment of the present invention. DETAILED DESCRIPTION
[0039] Based on the characteristics of different damages in the actual machine working borescope images and the interference of the surrounding environment, the selection of feature extraction network and attention improvement, the selection of calculation weights in the convolutional neural network are completed in a targeted manner, so that the improved network model can achieve the accuracy, false alarm rate, processing speed and hardware requirements when performing continuous image recognition of damages at different angles, different pixels, different damage types and different periods, and achieve the following functions:
[0040] 1. Accurately identify various types of damage such as ablation, cracks, and burn-through, select them, highlight the damaged area and extension direction, and indicate the confidence level of the damage judgment.
[0041] 2. Realize the automatic screenshot and annotation function of the damaged area.
[0042] 3. The accuracy of identifying early, middle and late stage injuries is sufficient, and the false alarm rate is lower than the level that affects actual work.
[0043] 4. Accurately identify various forms of engine blades (different observation angles, partial areas covered, etc.) and achieve precise target tracking, and on this basis, realize the two-way counting function of the blades.
[0044] 5. When performing video detection and real-time monitoring, if crack alarms continue to appear in consecutive frames and positions, take screenshots of the period for subsequent review.
[0045] 6. Identify equipment operation behaviors in the video that are not related to the borescope (such as modifying notes, measuring, etc.) and accelerate them, so as to help speed up the video review.
[0046] 7. Complete the integration, packaging and visualization of the functions of each module to enable the defect identification program to run smoothly on the borescope equipment.
[0047] The present invention will be further described below in conjunction with the accompanying drawings.
[0048] Items 1-5 above are the main functions of the borescope defect recognition network. The main structure diagram of the model is as follows: Figure 1 As shown, it specifically includes the following main parts:
[0049] The backbone feature extraction network is mainly used to extract several preliminary features of the blade boresight image.
[0050] The backbone feature network includes two optional forms: the ResNet-152 network that introduces the effective channel attention (ECA) attention mechanism optimization and the Darknet-53 network that replaces the C3 module with the C2f module. Based on their convolution kernel size, channel number restrictions, number of network layers, and iteration time, they are applied to different damage identification needs.
[0051] The enhanced feature extraction network mainly includes the PAFPNet module, which is connected to the above-mentioned backbone feature extraction network and is used to fuse and screen several of the above-mentioned preliminary features, extract effective features and output them to the prediction network.
[0052] As mentioned above, the enhanced feature extraction network includes the PAFPNet module (Path Aggregation Feature Pyramid Network, PAFPNet), which combines the Feature Pyramid Networks (FPNet) with the bottom-up Path Aggregation Networks (PANet). This module adds the strong positioning features of PANet from bottom to top on the basis of the top-down strong semantic feature transmission of the conventional FPNet, and aggregates the shallow feature maps (low resolution but weak semantic information) and the deep feature maps (high resolution but rich semantic information), shortening the information path between the bottom features and the top features, while further enhancing the expression capability of multi-scale features.
[0053] The prediction network mainly includes a decoupled head without an anchor frame and a loss calculation module. It is connected to the enhanced feature extraction network and is used to use the acquired effective features to determine the damage information and network performance in the borescope image of unknown state through decoupling and weighted regression calculation. The damage information includes damage type, damage size, and damage location.
[0054] The above-mentioned anchor-free decoupled head and loss calculation module extracts the position and category information of the aforementioned aggregated features through the decoupled head, inputs them into the corresponding network branches for loss calculation for learning respectively, and finally outputs the final result in a proportional weighted manner, thereby realizing parallel calculations at different scales, effectively reducing the number of parameters and computational complexity, and enhancing the generalization ability and robustness of the model.
[0055] Optionally, in order to enhance the feature extraction and prediction capabilities of the network, different network types and calculation weights are applied to different damage types, thereby achieving the purpose of targeted improvement of network performance.
[0056] The following introduces the technical solutions for building a defect recognition model, a blade counting model, an automatic damage identification system for aircraft engine borescope images, and its related (visualization and environment-dependent) construction technical solutions.
[0057] 1. Construction of defect recognition model (including data set acquisition, convolutional neural network improvement, training and complete technical solution for defect recognition).
[0058] 1.1 Obtain a sufficient number of real-machine borescope images and videos, and extract and filter images containing different damage types as the original data set. Mark the damage with closed geometric shapes and record the corresponding damage categories together. The marked damage images are used as the marked data set, and the unmarked images are used as the damage-free data set.
[0059] Furthermore, for the labeled data set, its damage data needs to cover different periods of each damage type, and the edge area of the defect must be accurately segmented through video verification and other methods.
[0060] Furthermore, if the data samples are insufficient, in order to improve the generalization of the damage identification model, it is necessary to increase the diversity of the labeled data set by cropping, rotating, flipping, and expanding.
[0061] Furthermore, the names of the labeled images and the corresponding labeled files in the labeled dataset need to correspond one to one and be stored in the same subdirectory.
[0062] Furthermore, the tag file needs to include the following information in sequence:
[0063] Damage type label, an Arabic integer with 0 as the initial value, each number represents a damage type.
[0064] Coordinates of each marking point, every two numbers Coordinates of each marking point, every two numbers is a group, each representing the position of the marker point in the image. The coordinate normalization calculation method is as follows:
[0065] By formula , , normalize the original pixel data, where is the original pixel data of the marker point in the image, The height and width of the image in pixels.
[0066] Furthermore, the labeled data set needs to be used for training, validation, and testing of the damage identification network. To ensure the training quality of unknown samples, the training set, validation set, and test set cannot have any intersection.
[0067] Optionally, the ratio of the training set, validation set, and test set is 8:1:1 by default. If the total number of labeled sample sets is insufficient, the ratio of the validation set needs to be appropriately increased.
[0068] 1.2 Use the above labeled data set to train and verify the damage recognition model to obtain the optimal recognition network corresponding to each damage type. The specific process is as follows:
[0069] 1.2.1 Training: The labeled data set used for training is input into the backbone feature extraction network, and then input into the enhanced feature extraction network after preliminary feature extraction through the backbone feature extraction network. The enhanced network fuses and screens the different scale features output by the backbone feature extraction network, and obtains the previous fusion feature layer through lateral connection and top-down feature fusion, and then obtains a further fusion feature layer through lateral connection and bottom-up feature fusion. The information of the obtained preliminary feature layer is repeatedly extracted through the above network to obtain as many effective features as possible.
[0070] Optionally, the backbone feature network can select one of the following two networks to complete preliminary feature extraction according to the specific type of damage:
[0071] The ResNet-152 network optimized by the ECA attention mechanism is introduced. The network implements the attention mechanism in a lightweight way. Compared with the original Darknet-53 network, it adopts residual learning and increases the number of network layers several times. By introducing the ECA optimization to weight different channels, the performance of the network can be improved without increasing the computational cost, and the network's cross-channel information acquisition ability is improved. It is more suitable for damage types that are difficult to extract features. In the patent of this invention, it is used to judge and detect turbine blade images with mainly damages such as cracks and unclear edge features.
[0072] The Darknet-53 network replaces the C3 module with the C2f module. This network includes 1×1 and more 3×3 convolution layers, and performs convolution, merging, bolding and other operations. Since it performs 1×1 convolution at the beginning of the convolution and uses fewer network layers, its performance such as the number of channels, inference speed and training speed are improved, but it is not effective in recognizing features with more details, and it is difficult to complete edge judgment when performing fine segmentation tasks. In the patent of this invention, it is used for the judgment and detection of images such as combustion chamber areas with more damage such as ablation and burn-through, and blade counting.
[0073] After the above feature extraction, the feature layer sizes of each 640×640 original image input to the prediction network are reduced to (20, 20), (40, 40), and (80, 80), respectively, that is, the height and width of the three feature layers are 1 / 32, 1 / 16, and 1 / 8 of the height and width of the original input image, respectively. In the prediction network, the effective feature layer obtained is used to obtain the damage dimensional information in the unknown state borescope image after decoupling, weighted regression, and decoding calculation, and the following results are output:
[0074] The damage category (Classification, Cls) corresponding to each feature area has a dimension of ( Damage category number).
[0075] The damage confidence (Objective, Obj) contained in each feature area has the dimension .
[0076] The coordinates of the damage prediction box (Bounding box, Bbox) corresponding to each feature area obtained by regression coordinates have a dimension of The determination method is as follows: are the coordinates of the center point of the feature area, are the feature area height and width respectively.
[0077] 1.2.2 Verification: Use the above damage recognition network to predict the damage information in the marked damage images in the marked database. The predicted damage information is calculated by the damage recognition network, and the predicted damage information includes the damage type, coordinates and location of the damaged area, and the damage confidence. The obtained predicted damage information is compared with the marked status in the marked data set to determine the final performance of the network.
[0078] Furthermore, in order to evaluate the performance of the damage identification network, a loss definition is introduced, which quantitatively determines the network performance and guides the subsequent network training. The loss function includes the classification loss, L Cls and regression positioning loss Regression Loss,L Reg , where the classification loss represents the difference between the predicted results and the actual results of the categories in the prediction network classification process, and the regression positioning loss combines factors such as overlapping area, center point distance, aspect ratio, etc., and comprehensively describes the difference between the predicted effect and the actual effect of the prediction network. The greater the loss, the greater the difference. The calculation formulas of the above two loss functions are as follows:
[0079]
[0080] in, qIt is the intersection-over-union (IoU) of the predicted box (Bounding box, Bbox) and the real box (GroundTruth box, GTbox) of any damage category. IoU is the intersection of the predicted box and the real box divided by the union of the two boxes, which represents the overlap between the predicted box and the real box. p is the probability of this damage type, γ is an adjustable factor, with a value range of [0,5]. α is the weight coefficient, and its value range is [0,1].
[0081]
[0082] in, Represent the center points of Bbox and GTbox respectively, p represents the Euclidean distance between the center points of Bbox and GTbox, c Indicates the diagonal distance of the closed area of two rectangular frames. v Indicates the consistency of the relative proportions of the two rectangular boxes. β is the weight coefficient, and its value range is [0,1]. q is the intersection-over-union ratio of Bbox and GTbox of any damage category.
[0083] Optionally, if the trained recognition network is used to identify damage types such as cracks whose edge information is not obvious, the weight coefficient needs to be appropriately increased based on the dark area on the blade edge to avoid the network from misidentifying dark areas such as the blade edge as cracks, etc. In this patent, the default weight coefficient for network training of damage such as ablation and burn-through is 0.5, and the default weight coefficient for network training of damage such as cracks is 0.7.
[0084] Furthermore, the loss function value of each damage type is calculated by the above method, and compared with the preset threshold to determine whether the loss function value meets the requirements, thereby determining whether the current network meets the damage identification performance requirements. If not, the variables in each layer of the network of the aircraft engine blade damage identification model are reversely updated according to the loss function value, and the "damage identification network training" step is returned. If the requirements are met, the training is terminated to obtain the trained aircraft engine blade damage identification network of each damage type.
[0085] 1.2.3 Use the above-mentioned trained damage recognition model to identify the unknown state of the borescope image or video, obtain the damage information of the image to be identified, and compare the damage marked image with the original image as shown below: Figure 2 and Figure 3 shown.
[0086] Furthermore, when damage such as ablation and burn-through is found, it is marked with a box and the confidence level of the damage judgment is indicated. When crack damage is found, the crack extension direction is highlighted.
[0087] Furthermore, an alarm instruction is added. During the process of video damage identification, if crack alarms are found to appear continuously in consecutive frames and positions, screenshots of time periods with higher confidence levels are captured for subsequent review.
[0088] Optionally, when outputting the video, the equipment operation behaviors not related to the borescope in the video are accelerated by frame extraction (such as modifying notes, measuring, etc.), so as to speed up the video review without affecting the review requirements.
[0089] 1.3 Auxiliary blade counting function
[0090] Obtain a sufficient number of engine blade borescope images, mark blades of different viewing angles and different occlusion conditions one by one through closed geometric shapes, and obtain and mark blades of different shapes one by one in order to enhance the following performance of the subsequent blade recognition model. The marked images are used as the marked data set, and the marking method and requirements are the same as 1.1. Figure 4 shown.
[0091] The leaf dataset input for training is introduced into the ResNet-152 network optimized by the ECA attention mechanism. After preliminary feature extraction by the backbone feature extraction network, it is input into the enhanced feature extraction network. The enhanced network fuses and filters the different scale features output by the backbone feature extraction network, thereby realizing repeated extraction of available feature information of the feature layer. The network is trained using the extracted feature data to obtain the corresponding leaf recognition model. The bidirectional counting of leaves is completed by following and capturing the leaf information that appears in the video and counting the leaf information that disappears on the left side and the lower side of the screen as an increase, and disappearing on the right side and the upper side of the screen as a decrease. The function display and following performance are shown in Figure 1. Figure 3 shown.
[0092] Furthermore, since the leaf counting function requires the detection frame's followability and anti-interference ability to be much greater than the edge recognition performance, the intersection-over-union ratio threshold is controlled within [0.15, 0.25] according to the leaf ablation state during the counting process. At the same time, a strategy for associating the previous and next frames is added, and the confidence of the previous frame exceeding a certain threshold is multiplied by the weight coefficient and superimposed on the corresponding grid position of the next frame to achieve the purpose of improving the counting performance.
[0093] Furthermore, the blade counting model distinguishes types by engine model, blade area, and leading and trailing edges. The corresponding counting model is better trained with the corresponding data set.
[0094] 1.4 Build a system for automatic identification of aircraft engine borescope image damage and its required environment, including the following: complete the design and integration of the functions of each module, including the camera video frame real-time capture and identification module, video frame damage identification module, image damage identification module, folder batch processing, etc., complete the corresponding UI design, and create an aircraft engine borescope damage automatic identification system based on this.
[0095] Optionally, the application environment of the borescope image damage recognition network model is containerized, and the real-time monitoring video is processed by uniform frame extraction based on the computer graphics card performance, so as to facilitate the establishment and smooth operation of the recognition system in computers with poor performance.
[0096] Optionally, in order to facilitate unified management and ensure the security of database information, the automatic damage identification system is restricted to the Internet and login and registration functions are added.
[0097] Optionally, to ensure the lightweight of the system, the training and verification of the previous model is completed in a high-performance image processing computer or GPU server, and only the damage recognition of images and videos and the damage recognition of frame images of real-time images are integrated into the damage recognition system.
[0098] It can be seen from the above embodiments that the improved model proposed in the present invention has a significant improvement in mean average precision (mAP) compared with the YOLOv8 damage recognition model. The mAP50 (mAP value at 50% IoU threshold) for the recognition of damage such as cracks is increased by about 17.4%, and the recall rate is increased by about 4.8%; the mAP50 for the recognition of damage such as burn-through and ablation is increased by about 12.6%, and the recall rate is increased by about 11.5%. The damage recognition system reduces the average reasoning time per frame of the image by 5.3%, and has made a number of optimizations for practical applications, which improves the reasoning speed and reduces the requirements for computer configuration while ensuring the recognition accuracy.
[0099] At the same time, the present invention integrates functions such as model training, model verification, video review, real-time monitoring, and account management, which greatly enhances the practicality of the invention.
[0100] In addition to the above, it should be noted that "one embodiment", "another embodiment", "embodiment", etc. mentioned in this specification refer to the specific features, structures or characteristics described in conjunction with the embodiment included in at least one embodiment generally described in this application. The same expression appearing in multiple places in the specification does not necessarily refer to the same embodiment. Further, when describing a specific feature, structure or characteristic in conjunction with any embodiment, it is claimed that the realization of such feature, structure or characteristic in conjunction with other embodiments also falls within the scope of the present invention.
[0101] Although the present invention is described herein with reference to a number of illustrative embodiments of the present invention, it will be appreciated that those skilled in the art may devise many other modifications and implementations that fall within the scope and spirit of the principles disclosed herein. More specifically, within the scope of the present disclosure, drawings, and claims, a variety of variations and improvements may be made to the components and / or layout of the subject combination layout. In addition to variations and improvements made to the components and / or layout, other uses will also be apparent to those skilled in the art.
Claims
1. A borescope image damage recognition model based on an improved convolutional neural network, characterized in that The borescope image damage recognition model includes: A backbone feature extraction network is used to extract preliminary features of the actual borescope image; the backbone feature extraction network includes a ResNet-152 network and a Darknet-53 network, and the ResNet-152 network and the Darknet-53 network are respectively applied to corresponding damage identification requirements based on their different convolution kernel sizes, channel number restrictions, network layers, and iteration times; the ResNet-152 network is a ResNet-152 network optimized by introducing an ECA attention mechanism; the Darknet-53 network is a Darknet-53 network with a C2f module; The enhanced feature extraction network is connected to the backbone feature extraction network to fuse and filter the preliminary features, extract effective features therein and output them to the prediction network; The enhanced feature extraction network includes a PAFPNet module, which is used to aggregate shallow feature maps and deep feature maps, shorten the information path between bottom-level features and top-level features, and enhance the expression ability of multi-scale features; The prediction network is connected to the enhanced feature extraction network, and is used to use the extracted effective features to complete the extraction of target position and category information through decoupling, loss calculation and proportional weighting; and then determine the damage information in the borescope image of the unknown state, wherein the damage information includes damage type, damage size and damage position; The prediction network includes a decoupling head without an anchor frame and a loss calculation module, which is used for extracting the position and category information of the aggregated effective features by the decoupling head, inputting them into the corresponding loss calculation network branches for learning respectively, and then outputting the final result in a proportional weighted manner; The backbone feature extraction network extracts preliminary features of the real machine borescope image through the training set in the labeled data set; the labeled data set is obtained in the following manner: Obtain a sufficient number of real-machine borescope images and videos, and extract and filter out images containing different damage types as the original data set; The damage is marked by closed geometric shapes and the corresponding damage categories are recorded together. The marked damage images are used as the marked data set, and the unmarked images are used as the non-damage data set. The marked data set includes different periods of each damage type and the damage edge area is segmented. The borescope image damage recognition model is also used to recognize real machine borescope images of different viewing angles and different occlusion conditions, and mark the leaves in the real machine borescope images one by one, use the marked images of the leaves as a marked data set, and perform auxiliary counting on the leaves; After the backbone feature extraction network performs preliminary feature extraction on the labeled data set, the feature extraction network is input into the enhanced feature extraction network, and the enhanced network fuses and screens the different scale features output by the backbone feature extraction network to form a fused feature layer, and repeatedly extracts the available feature information of the fused feature layer; The blade auxiliary counting is to follow and capture the real machine blade image appearing in the video, and complete the bidirectional counting of the real machine blade by counting when it disappears in the left and lower screen range, and when it disappears in the right and upper screen range.
2. The borescope image damage recognition model based on improved convolutional neural network according to claim 1 is characterized in that: The labeled data set includes a test set, a training set and a validation set, and the labeled images in the labeled data set correspond to the names of the corresponding label files one by one, and the test set, the training set and the validation set are respectively stored in corresponding subdirectories; The tag file includes the following information in sequence: Damage type label, Arabic integer with 0 as the initial value, each number represents a damage type; The coordinates of each marker point, each pair of numbers (W, H') is a group, representing the position of the marker point in the image. The coordinates of the marker point are normalized by the formula Calculated, where w and h are the original pixel data of the marker point in the image, and W and H are the height and width pixel values of the image.
3. The borescope image damage recognition model based on improved convolutional neural network according to claim 1 is characterized in that: The effective feature extraction method of the enhanced feature extraction network is to obtain a preceding fused feature layer through lateral connection and feature fusion from top to bottom, and then obtain a further fused feature layer through lateral connection and feature fusion from bottom to top, and repeatedly repeat the effective feature extraction method.
4. The borescope image damage recognition model based on an improved convolutional neural network according to claim 1 is characterized in that: The labeled data set is used for training and verification of the damage identification network. The damage identification network extracts the location and category information of the damage target by decoupling the feature layer, calculating the loss and weighting the proportion, thereby realizing identification, prediction and verification of network performance. The loss calculation function includes the classification loss function Classification Loss, L Cls And regression positioning loss function Regression Loss,L Reg ; The classification loss function Classification Loss,L Cls The calculation formula is: Wherein, q is the intersection over union (IoU) of the predicted box Bbox and the real box GTbox of any damage category. The IoU is the intersection of the predicted box Bbox and the real box GTbox divided by the union of the two boxes, which represents the overlap between the predicted box Bbox and the real box GTbox. p is the probability of the damage type. γ is an adjustable factor with a value range of [0,5]. α is a weight coefficient with a value range of [0,1]. The regression positioning loss function Regression Loss, L Reg The calculation formula is: In the formula, b, b gt are the center points of Bbox and GTbox respectively, p is the Euclidean distance between the center points of Bbox and GTbox, c is the diagonal distance of the closed area containing the two rectangular boxes, v is the consistency of the relative proportions of the two rectangular boxes, β is the weight coefficient, and the value range is [0,1], q is the intersection and union ratio of Bbox and GTbox of any damage category.
5. The borescope image damage recognition model based on improved convolutional neural network according to claim 1 is characterized in that: During the counting process of the blade auxiliary counting, the intersection-and-combination ratio threshold is controlled within [0.15, 0.25] according to the actual blade ablation status.
6. An application system of the borescope image damage recognition model according to any one of claims 1 to 5, characterized in that The system includes a camera video frame real-time capture and recognition module, a video frame damage recognition module, an image damage recognition module and a folder batch processing module; The camera video frame real-time capture and recognition module is used to identify and capture the defective borescope image in the current video; The video frame damage recognition module and the image damage recognition module are used to respectively recognize and mark the damage positions in the video frame and the image through the borescope image damage recognition model; The folder batch processing module is used to batch process all identifiable videos and images in the corresponding folder.
Citation Information
Patent Citations
Aero-engine hole detection image damage intelligent identification method based on deep learning
CN109800708A
Aero-engine blade damage identification model, damage identification method and system
CN114842192A
Deep learning-based aero-engine nondestructive testing method, device, equipment and storage medium
CN112581430A
Garbage classification collection and transportation supervision method based on improved YOLOv3 network
CN115035442A
Aero-engine damage automatic detection method
CN116758395A
