METHOD AND COMPUTER PROGRAM PRODUCT FOR AUTOMATED DEFECT DETECTION IN ENGINE BOROSCOPY

DE502020009575D1Active Publication Date: 2024-11-14LUFTHANSA TECHNIK AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502020009575
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-08-29
Filing Date
2020-08-26
Publication Date
2024-11-14
Estimated Expiration
2040-08-26

AI Technical Summary

Technical Problem

The manual inspection of engine blades in jet engines is time-consuming and lacks standardization, with existing methods requiring several hours to inspect all stages and often resulting in unreliable defect detection due to human error.

Method used

An automated method using a video borescope that tracks engine blades through image recognition, identifying defects by comparing successive images and confirming their presence based on movement and speed, reducing false positives and negatives through optical image recognition and deep learning techniques.

Benefits of technology

This method significantly reduces the time and error rate in defect detection, achieving a reliability comparable to or exceeding manual inspection, with the ability to partially automate the inspection process and provide standardized reports.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method and computer program product for automated defect detection during engine boroscopy.

[0002] Engines, especially aircraft jet engines, must be inspected regularly to verify compliance with technical safety requirements and detect any damage at an early stage. Especially for inspections while the jet engine is mounted on the aircraft (onwing), side flaps on the jet engine are opened and / or individual components are removed so that an inspector can view the interior of the jet engine directly or with the aid of a borescope to inspect the engine blades. The inspector's task is to examine the engine blades directly through the eyepiece of the borescope or—in the case of a video borescope—on the connected video monitor, and to reliably identify defects on the engine blades, such as notches or dents, in the tenths of a millimeter range.

[0003] In order to inspect the engine blades of a single rotating engine stage of the jet engine, it is known to rotate the engine stage and move them past a viewing opening of the jet engine or past a borescope inserted into the jet engine in such a way that all the engine blades of the engine stage of the jet engine can be inspected one after the other in the viewing opening or the field of view of the borescope.

[0004] The inspection of all engine stages of a jet engine requires several hours; the inspection of the nine stages of the high-pressure compressor of a CFM56 engine alone requires four hours. The result is an evaluation by the inspector, which reflects the inspector's visual impression but usually does not meet the requirements for a standardized assessment.

[0005] Document JP 2014-32363 A discloses a video endoscope for inspecting turbine blades of a gas turbine. Nozzles are provided in the area of ​​the image acquisition sensor, from which compressed air can be discharged in a controlled manner. This compressed air can interact with the turbine blades in such a way that the affected turbine stage is set in rotation or decelerated. By edge analysis of images acquired in chronological order, the rotation speed of the turbine stage can be determined and ultimately controlled.

[0006] It is an object of the present invention to provide a method by which the inspection of the engine blades of rotating engine stages of a jet engine can be improved.

[0007] This problem is solved by a method according to the main claim. Advantageous further developments are the subject of the dependent claims.

[0008] Accordingly, the invention relates to a method for automated defect detection in engine boroscopy, in which a video boroscope is inserted into an engine in such a way that when an engine shaft is rotated, the engine blades of an engine stage attached thereto are moved one after the other through the image area of ​​the video boroscope, characterized by the steps: Identifying a possible defect in an engine blade by image recognition based on a single image from the video borescope; Detecting the movement of the engine blades in the image field of the video borescope by comparing two consecutive individual images; Tracking the possible defect on the video image along the recorded movement by optical image recognition based on the individual images used for motion detection; If the track of a possible defect on the video image matches the recorded movement in direction and speed over a specified length: identifying the possible defect as an actual defect.

[0009] The invention further relates to a computer program product comprising program parts which, when loaded into a computer, are designed to carry out the method according to the invention.

[0010] First, some terms used in connection with the invention will be explained.

[0011] A "video borescope" is a borescope that provides a continuous analog or digital video image of the borescope's image area for further processing. The image captured by an optical system at the free end of the borescope can be transmitted via an optical cable to a camera at the other end of the borescope, where it is converted into the desired video image. It is also possible, however, for the captured image to be converted into a video image by a suitable chip (e.g., a CCD or CMOS chip) directly at the free end of the borescope. This video image is then transmitted via a data cable to the other end of the borescope, where it is available for further processing.

[0012] The "video image" is a continuous sequence of "individual images" in which the time interval between the recording of the individual images is usually constant, which is why the frame rate can also be used as a measure of the number of individual images per given time period.

[0013] "Two consecutive frames" refers to two consecutive frames. These do not necessarily have to be two directly consecutive frames of the video image. It is also possible that the video image contains intermediate frames between the two consecutive frames, which, however, are not taken into account, at least in the inventive determination of the movement of the engine blades in the image field of the video borescope.

[0014] The invention provides a method with which the demanding, yet monotonous task of individually inspecting the multitude of engine blades of an aircraft engine can be at least partially automated. For this (partial) automation of engine inspection to even be considered, a high degree of defect detection reliability is required. The method according to the invention achieves this because both false negative and false positive results can be reduced to such an extent that the error rate of the method according to the invention is comparable to, if not lower than, that of a manual inspection.

[0015] The method according to the invention is based on the arrangement known from engine boroscopy, in which a video borescope is inserted through a suitable opening into an aircraft engine and aligned so that at least one engine blade of the engine stage to be examined is located within the image field of the video borescope. Subsequently, by rotating the engine shaft to which the engine blades of the engine stage to be examined are attached, the individual engine blades of the engine stage can be moved one after the other through the image field of the video borescope.

[0016] Although, for the sake of clarity, it is assumed below that the image field of the video borescope is generally large enough to capture the entire length of the engine blades, the method according to the invention is also suitable for inspecting individual longitudinal sections of the engine blades. By repeating the method sufficiently for the individual longitudinal sections, a complete inspection of the engine blades of an engine stage can be performed over their entire length.

[0017] Using image recognition, potential defects in the engine blade(s) in the individual frames of the video image, or at least a portion of them, are identified. At this stage, these are initially only "defect candidates," i.e., those features on the engine blades that can be identified by image recognition and that could potentially represent a defect.

[0018] Because the video borescope not only records individual images as needed, but also provides a continuous video image, any possible movement of the engine blades can be detected by comparing two consecutive individual images. Such movement can usually only be achieved by rotating the engine shaft to which the engine blades observed by the video borescope are attached. Since the movement of the engine blades is recorded solely from the video image, it is irrelevant how the rotational movement of the engine shaft is actually achieved. Even manual rotation of the engine shaft remains possible, allowing the inventive method to be used flexibly.

[0019] If movement of the engine blades is detected, the previously identified possible defect is tracked on the video images along the recorded movement. For this purpose, optical image recognition is applied to those individual frames of the video image that are also used to record the movement. It is possible to apply image recognition to the entire individual images in order to, for example, also identify possible defects in the image area of ​​the video borescope (see above), or to limit image recognition to those areas of the individual images in which the possible defect should be located according to the recorded movement, which can save considerable computing power. A combination of the two options is also conceivable, e.g.Every fourth individual image is completely submitted to image recognition, while for the individual images in between, only the areas in which previously identified possible defects should be located are analyzed.

[0020] If an initially identified potential defect is repeatedly identified by image recognition on several consecutive individual images, a trace of the potential defect is created across the individual images. If this trace matches the direction and speed of the movement of the engine blades recorded on the basis of the same individual images, and if the trace also has a specified length, the defect previously classified as only a possible candidate can be identified as an actual defect.

[0021] Because the method according to the invention does not rely on image recognition of individual images to identify defects, false results—for example, due to light reflections, shadows, dirt, dust, etc.—can be avoided. Only if a potential defect is identified and thus confirmed as a possible defect across multiple individual images, each of which is located at a different location due to the movement of the engine blades, can it be assumed with a very high degree of probability that it is indeed a defect. Conversely, if a potential defect is identified only in individual, non-consecutive images, it can be assumed with a very high degree of probability that it is not an actual defect.

[0022] It has been shown that the method according to the invention can achieve a defect detection accuracy at least comparable to that of a manual inspection. Often, even better results can be achieved.

[0023] Among other things, in order to easily verify the defects identified according to the method according to the invention, it is preferred if the detection of the movement of the engine blades is used to identify the position of the individual engine blades relative to an arbitrarily selected starting engine blade. For example, if the engine blade that is most completely located within the image field of the video borescope when a possible defect is first identified based on the video image is assumed to be the starting engine blade, the other engine blades can be unambiguously determined in their relative position relative to the starting engine blade by appropriately taking the detected movement into account. For example, it is thus possible to "number" the individual engine blades starting from the starting engine blade.This allows a specific, previously analyzed engine blade to be moved into the image field of the video borescope by appropriately rotating the engine shaft and recording the movement. This allows an inspector, for example, to re-examine the defect detected by the method on the engine blade in question, which is clearly identifiable by its numbering, using a live video image to confirm or reject the defect detection. When numbering the engine blades, the direction of rotation of the engine shaft can be taken into account, so that the numbering, for example, always increases clockwise.

[0024] It is preferred if the method comprises controlling the rotational movement. The rotational movement can be controlled, for example, by appropriately controlling the air starter of the aircraft engine or an external drive temporarily arranged on the aircraft engine. Even if a corresponding control alone generally does not allow for an exact determination of the movement of the engine blades in the image field of the video borescope, at least the presence of such a movement itself and, in particular, the basic direction of the movement can be taken into account when comparing two consecutive individual images to detect the movement. In particular, specifying the direction can simplify the final detection of the movement.In addition, by appropriate control of the rotational movement, it can be ensured that the engine blades move through the image area of ​​the borescope only at a sufficiently low speed so that their movement can still be recorded from the individual images and, if necessary, the position of the individual engine blades can be identified relative to an arbitrarily selected initial engine blade.

[0025] To capture the movement of the engine blades, it is preferable to segment the individual images into regions that precisely reflect the outlines of the individual engine blades, with the remaining region then being identified as the non-blade region. Compared to image capture methods that determine a bounding box around specific objects contained in an image, segmentation yields the precise outline of the visible part, e.g., of an engine blade. This enables a more precise assignment of the individual image regions to individual engine blades. Segmentation can be performed, for example, using a convolutional neural network, such as the Mask-RCNN method, preferably comprising an inception feature extractor, as described by He, Gkioxari, Dollar, and Girshick in the article "Mask R-CNN" (ar-Xiv.org, document number 1703.06870v3).Motion capture can then be achieved by changing the segmentation of two consecutive frames. This only requires that the movement of the engine blades be sufficiently slow, that the position of the individual engine blades does not change excessively from one frame to the next, and, in particular, that a specific engine blade is depicted in more than two consecutive frames used to capture the motion. This can be ensured by appropriately selecting the frame rate of the video borescope, the number of disregarded intermediate frames between two frames, and / or the speed of the engine blades' movement within the image field of the video borescope.

[0026] To reduce the computing power required for segmenting individual images, it is preferable to define exclusion regions within which no segmentation of the individual images takes place. For example, if the video image contains regions that do not change even when the engine blades move – such as stator blades or stationary walls – these can be excluded by specifying suitable exclusion regions prior to segmentation, so that segmentation only needs to be performed for a section of the individual images.

[0027] Regardless of whether the movement is captured using the described segmentation and subsequent comparison of consecutive individual images or any other method, it is preferable if a direction of movement of the engine blades is specified in the image field of the video borescope, which is then also maintained during the actual movement. If a direction of movement is specified, the computational effort required to assign the individual turbine blades or the individual turbine blade segments during a movement is reduced. Specifying the direction of movement of the turbine blades in the image field of the video borescope essentially corresponds to specifying the direction of rotation for the engine shaft to which the turbine blades in question are attached.

[0028] The actual identification of defects in the engine blades, such as notches or dents, is preferably performed using a convolutional neural network, e.g., a Faster-RCNN method, as described by Ren, He, Girshick, and Sun in the article "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks" (arXiv.org, document number 1506.01497v3). This method is sufficiently fast to be used not only for initial defect identification but also for tracking a potential defect during engine blade movement.

[0029] Both the Mask-RCNN method and the Faster-RCNN method are based on convolutional neural networks. However, other deep learning methods can also be used. In particular, the deep learning method ultimately used for defect identification can be continually improved through the confirmation or rejection of defect detection by an inspector, as described above. Furthermore, it is of course necessary to train the deep learning methods used before their first use using suitable manually processed or evaluated training data sets, each comprising individual images on which the areas or features to be automatically detected are marked. Depending on the deep learning method used, the training effort orThe effort required to create training data sets can be reduced by transferring a single image underlying a training data set into another training data set by mirroring, rotating or adjusting contrast and / or brightness, whereby the final, complex marking of the individual images - if necessary with comparable mirroring and / or rotation - can be carried out without the need for manual rework.

[0030] For an explanation of the computer program product, reference is made to the above statements.

[0031] The invention will now be further described using an exemplary embodiment with reference to the accompanying drawings. They show: Figure 1: an arrangement designed to carry out the method according to the invention; Figure 2a, b: two series of, from the video borescope of the arrangement according to Figure 1recorded consecutive individual images; Figure 3: an exemplary representation of the segmentation of individual images; and Figure 4a, b: an exemplary representation of the motion detection and determination of the relative position of engine blades based on segmented individual images.

[0032] In Figure 1 An exemplary arrangement 1 designed to carry out the method according to the invention is shown during use for the borescope inspection of an aircraft engine 80 - only indicated in the figure.

[0033] The aircraft engine 80 is a turbofan engine with a low-pressure compressor stage 81 comprising the fan 82, which is connected to a low-pressure turbine 84 via a first shaft 83, and a high-pressure compressor stage 85, which is connected to a high-pressure turbine stage 87 via a second shaft 86. The combustion chamber arranged between the high-pressure compressor stage 85 and the high-pressure turbine stage 87 is not shown for reasons of clarity. Figure 1 shown.

[0034] For the inspection of the engine blades 90 of the individual stages 81, 84, 85, and 87, the engine cover 88 is open (and therefore shown in dashed lines), allowing access to the core engine 89. Various borescope openings 91 are provided on the core engine 89, into which a video borescope 2 can be inserted. Figure 1Two borescope openings 91 are shown as examples, into each of which a borescope 2 can be inserted in order to inspect the engine blades 90 of a specific turbine stage - in the example shown, either the high-pressure compressor stage 85 or the high-pressure turbine stage 87 - through the borescope 2. By rotating the shaft 83, 86 to which the engine blades 90 to be inspected are attached - in the example, the second shaft 86 - all of the engine blades 90 of the engine stage located in the image area of ​​the borescope 2 can be inspected one after the other. A direction of rotation is specified for this purpose, which is indicated by the arrow 95. The actual rotation of the second shaft 86 is effected by a drive unit (not shown) integrated into the aircraft engine 80.

[0035] The video borescope 2 is connected to a computer unit 3, which is designed to process the video image originating from the video borescope 2 as described below. Furthermore, the computer unit 3 also includes an output 4 for connection to the control unit of the aircraft engine 80, via which it controls the drive unit for the second shaft 86. A terminal 5 is also connected to the computer unit 3, via which the video image or individual images thereof can be examined by an inspector.

[0036] In Figure 2a are exemplary from left to right three temporally consecutive individual images of the video borescope 2 according to Figure 1recorded video image of a single stage of the high-pressure turbine stage 87. Between the individual images shown there are two intermediate images, which are not taken into account in the processing described below and are therefore also not included in the display of the Figure 2a can be omitted.

[0037] Due to a rotation of the second shaft 86 initiated by the computer unit 3 via the output 4 into the Figure 1 In the direction of rotation 95 shown, the engine blades 90 of the engine stage 87 move through the image field of the video borescope 2. As a result, the engine blade 90 shown moves in the individual images of the Figure 2a from left to right.

[0038] For each of the individual images 10 according to Figure 2aImage recognition is performed to detect possible defects. A suitably trained Faster-RCNN method is used for defect detection, which is specifically trained to detect notches and dents. All possible defects on the individual images according to Figure 2a are marked by a box 11.

[0039] The individual images 10 according to Figure 2a are also used to determine the movement of the engine blades 90 in the image area of ​​the video borescope 2.

[0040] As in Figure 3 As shown by way of example using other individual images 10` of a video image recorded by a borescope 2, each individual image 10` is segmented into individual engine blade areas 90' using a Mask-RCNN method with Inception Feature Extractor ( Figure 3 , below).

[0041] An exclusion region 12, indicated by dashed lines, is defined in which no segmentation of the individual images 10' is required, since a stationary element is located in this region 12 of the video image, thus precluding the possibility of an engine blade 90 being visible in this region. By defining an exclusion region 12, the computing power required for segmentation can be reduced.

[0042] Since the computer unit 3 has caused the second shaft 86 to rotate in the direction 95 via the output 4, whereby the basic direction of movement of the engine blades 90 in the image area of ​​the borescope 2 is known, the speed and direction of movement of the engine blades 90` can be determined in a simple manner using the engine blade areas 90', which in Figure 3 indicated by the vector 95`.

[0043] The intended segmentation of the individual images 10` makes it possible, in addition to detecting the movement of the engine blades 90, to determine the relative position of the individual engine blades 90 with respect to an arbitrarily selected initial engine blade 90", which is subsequently determined using the Figures 4a and b is explained in more detail.

[0044] As in Figure 4a A single image 10" is displayed initially according to Figure 3 segmented, whereby - if a corresponding selection has not yet been made - one of the engine blade segments 90' is defined as the starting engine blade 90". Subsequently, the engine blades 90 or the engine blade segments 90' can be numbered starting from the starting engine blade 90".

[0045] Subsequently, taking into account the recorded movement of the engine blades 90, each engine blade segment 90' can be assigned a unique number which corresponds to a counting of the individual engine blades 90 or engine blade segments 90' (cf. Figure 4b ). By means of the position of the individual engine blades 90 thus detected, it is possible to individually control the drive of the shaft 86 via the output 4 of the computer unit 3 and to simultaneously check the position according to Figure 4a, b be targeted in order to enable, for example, the follow-up inspection of a specific identified possible defect on the corresponding engine blades 90 using a live video image via Terminal 5.

[0046] Already on the basis of the Figure 3 detected movement of the engine blades 90, the Figure 2aidentified possible defects on the video image, more precisely on the individual images 10 used for motion detection. For this purpose, an optical image recognition is carried out for each of the individual images 10 in question, whereby - as in the present case - the Figure 2a The image acquisition described above can be used to identify possible defects and their results. In combination with the detected movement, a track 92 can be determined across the video image for each possible defect, provided it is present in all individual images, as shown in Figure 2b. If one of the tracks 92 thus determined has a length that exceeds a predetermined length, the possible defect forming the track 92 can be identified as an actual defect, which can be documented in a suitable, standardized manner.

[0047] If a sufficiently long, continuous trace 92 cannot be determined for a possible defect, e.g., because no defect attributable to a specific trace 92 could be identified in a single image 10, the possible defect is highly likely not an actual defect. Such a possible defect can nevertheless be documented and, in particular, as mentioned above, can be inspected again by an inspector via Terminal 5 by deliberately approaching the engine blade 90 with the possible defect in order to verify the automatically determined assessment. The same naturally applies to defects identified as actual.

[0048] The results of the inspector's verification can be used to optimize image detection of potential defects.

Claims

1. Method for automated defect detection during borescoping of an engine, in which a video borescope (2) is inserted into an engine (80) in such a way that, during rotation of an engine shaft (83, 86), the engine blades (90) of an engine stage (81, 84, 85, 87) which are secured to said engine shaft are moved successively through the image region of the video borescope (2), wherein the following steps are carried out: - Identifying a possible defect on an engine blade (90) by image recognition on the basis of a frame (10) of the video borescope; - Detecting the movement of the engine blades (90) in the image region of the video borescope by comparing in each case two successive frames (10, 10'); - Tracking the possible defect on the video image along the detected movement by optical image recognition on the basis of the frames (10, 10') used for detecting the movement; - If the trace (92) of a possible defect on the video image corresponds to the detected movement in terms of direction and speed over a predefined length: identifying the possible defect as an actual defect.

2. Method according to Claim 1,characterized in thatthe detection of the movement of the engine blades (90) is used to identify the position of the individual engine blades vis-à-vis an arbitrarily chosen initial engine blade.

3. Method according to either of the preceding claims,characterized in thatthe method comprises controlling the rotational movement of the engine axle.

4. Method according to any of the preceding claims,characterized in thatfor the detection of the movement of the engine blades, the frames are segmented into individual blade regions, preferably by a mask RCNN method, with further preference comprising an inception feature extractor.

5. Method according to Claim 4,characterized in thatexclusion regions are defined in which the frames are not segmented.

6. Method according to any of the preceding claims,characterized in thata direction of movement for the engine blades in the image region of the video borescope is predefined.

7. Method according to any of the preceding claims, characterized in that • the possible defects that are identifiable comprise notches and dents.

8. Computer program product comprising program parts which, when loaded in a computer, are designed for carrying out the method according to any of Claims 1 to 7.