Computer-implemented method and system for generating a defective image of a machine through a defect mask

Abstract

According to the innovative method a defective image of a portion of a machine is generated through at least the steps of: receiving (240) an non-defective image of the portion of the machine, receiving (250) a segmentation image mask corresponding to a defect, receiving (260) defect coordinates corresponding to a position of the defect, generating (270) a masked image by blanking an area of the non-defective image, and using a conditional Generative Adversarial Network for generating (280) the defective image; notably, it is the masked image that is input to the conditional Generative Adversarial Network as image input and not the whole non-defective image; notably, both the segmentation image mask and the defect coordinates are input to the conditional Generative Adversarial Network as condition input and not simply a label.

Description

TITLEComputer-implemented method and system for generating a defective image of a machine through a defect maskDESCRIPTIONTECHNICAL FIELD

[0001] The subject matter disclosed herein relates to a computer-implemented method for generating a defective image of a portion of a machine, in particular a turbomachine, through a defect mask.BACKGROUND ART

[0002] Artificial Intelligence (usually abbreviated as “Al”) may be used for example in order to identify defects in manufactured machines or parts thereof. A machine or a part containing a defect may have been manufactured since a short time, for example few seconds or minutes; in this case, the defect is typically due to a manufacturing problem. Alternatively, a machine or a part containing a defect may have been manufactured since a long time, for example days, weeks, months or years, and may have been used in operation or stored in warehouse after being manufactured; in this case, the defect is typically due to its use in operation and / or its storage in warehouse.

[0003] A “defect” in a machine or part may be defined as a deviation in the value of one of its features that affects the correct operation or performance of the machine or part, i.e. that leads to operation or performance not in accordance with predefined limits; the refence for evaluating the deviation is the “rated value”, i.e. the value corresponding to the design of the machine or part. A “defect” in a machine or part may also be defined as a deviation in the value of one of its features not in accordance with predefined limits. Forexample, a “crack” in e.g. a part of a machine is a defect as a surface of the part having a crack does not correspond to the surface according to its design, i.e. its ideal surface.

[0004] Some kinds of defect may be identified simply by watching e.g. a part of a machine or by watching an image, e.g. a photograph, of the part. In the case of images, visibility of a defect may depend on one or more factors, for example the observation point and the illuminating light.

[0005] Other kinds of defects cannot be seen, and may require measurements by non-destructive techniques and / or invasive examinations.

[0006] Assuming that a defect in e.g. a part of a machine is visible, a captured image of this part shows the defect in that the captured image is different from an image of the same part that contains no defect. In the following, the expression “defective image” will be used to mean an image that shows, in a more or less evident way, a defect in a machine or a part of a machine, and the expression “non-defective image” will be used to mean an image that does not show a defect.

[0007] In general, it is difficult to identify defects based on images even for a so-called “subject matter expert” as differences in images may not be evident and / or may be caused by other reasons, i.e. not defects, and more difficult for a computer-based system. Therefore, nowadays, systems based on Artificial Intelligence are sometimes used for automatically identifying defects in machines or parts starting from images, in particular for quality tests after manufacturing.

[0008] Such a system based on Artificial Intelligence usually requires three main phases: 1) data collection, 2) data annotation, and 3) optimization of an Al model. For example, in order to detect cracks in industrial combustor chambers, one must first collect a large amount of photos of industrialcombustor chambers containing cracks; then, one must involve one or more experts to annotate these photos, i.e. mark the cracks in the photos; finally, one can use the annotated photos to optimize an Al model in order to perform automated crack detection, in other words, as well known in the field of Al, the system “learns” how to perform a task, before performing the task, directly from data provided to the system during a so-called “training phase”.

[0009] Unfortunately, collecting a large annotated dataset is not only timeconsuming but also requires experts.

[0010] Therefore, it is desirable to reduce the temporal bottleneck introduced by data collection and annotation as much as possible.

[0011] From the article of Rajhans Singh et al. entitled “Generative Adversarial Networks for Synthetic Defect Generation in Assembly and Test Manufacturing" in the 31 st Annual SEMI ASM Conference held in 2020, there are known Artificial Intelligence-based synthetic defect generation techniques to augment the training image sets for CNNs-based defect detection and classification systems to be used in semiconductor manufacturing processes of e.g. “chips” or “ICs”; Generative Adversarial Networks (GANs) are described to create various modes of the defects which are difficult to create manually; they declare that the output of their adapted GANs are images of realistic- looking defects for a wide variety of common manufacturing defects including foreign material, misplaced epoxy, scratches, and die chipping defects among others. According to this article, two specific GAN techniques were adapted: Style-Transfer GAN and Cycle-Consistency GAN. This article mentions also the possibility of using conditional GANs wherein a label information is provided as condition, the label corresponding to the type of defect (for example, foreign material, misplaced epoxy, scratches, and die chipping). However, there is little control on the defect generation process; in particular, only the type of defect is provided as input, while it would be advantageous to make more use of expert knowledge during this process.

[0012] From the article of Luka Posilovic et al. entitled “Generative adversarial network with object detector discriminator for enhanced defect detection on ultrasonic B-scans" in ARXIV, from Neurocomputing vol. 459 at pages 361-369 published in 2021, there are known two distinct methods (see paragraph 3): the first one is based on a “copy / paste” approach (see paragraphs 3 and 4.1) and is considered “current state-of-the-art method' and the second one is based on a “deep learning approach” through “GAN” (see paragraphs 3 and 4.2) and is considered “novel” mainly due to “the usage of a pre-trained object detector for training the GAN” , i.e. the GAN that is used for generating synthetic defective images that in turn may be used for training Artificial Intelligence systems able to automatically identify defects in machines image; the only input to this GAN is a “position mask” (that corresponds to the combination of what are referred in the following as “position” of the defect and “bounding area” of the defect); no input is given to this GAN about the type and shape of the defect(s) to be used for generating the synthetic defective image(s); furthermore, no input is given to this GAN about the context of the defect(s) to be used for generating the synthetic defective image(s); in other words, this GAN is almost totally free to generate synthetic defective images apart from their position masks.

[0013] It is to be noted that Generative Adversarial Networks and conditional Generative Adversarial Networks (i.e. GANs wherein the image generation is dependent on one or more conditions input to the network) are architectures generally known in the field of image processing. However, many variants to the architecture and to the kind of input data are possible, and architecture and input data affect the performances of the network.SUMMARY

[0014] Therefore, it would be now desirable to improve further synthetic defective images generation techniques in order to achieve, afterwards, better automatic defects identification.

[0015] In particular, it would be now desirable to have more control on the generated synthetic defective images using GAN.

[0016] The Applicant has decided to focus on defects in turbomachines.

[0017] According to a first aspect, the subject matter disclosed herein relates to a computer-implemented method for generating a defective image of a portion of a machine. According to the innovative images generation method a defective image of a portion of a machine is generated through at least the steps of: receiving an non-defective image of the portion of the machine, receiving a segmentation image mask corresponding to a defect, wherein the segmentation image mask defines a bounding area, receiving defect coordinates corresponding to a position of the defect, generating a masked image by blanking an area of the non-defective image, and using a conditional Generative Adversarial Network for generating the defective image, wherein, notably, the masked image is input to the conditional Generative Adversarial Network as image input, wherein, notably, both the segmentation image mask and the defect coordinates are input to the conditional Generative Adversarial Network as condition input. A “segmentation mask” allows to “partition” an image into regions; in the present case, the mask allows to distinguish a defect from the rest; in particular, the mask allows to divide an area (for example the area of a “bounding box”) into two portions: a first portion corresponding to a defect (for example, each pixel of the first portion is to be considered a detail representation of the defect) and a second portion not corresponding to a defect (for example, each pixel of the second portion is notto be considered a detail representation of the defect). Preferably, the segmentation image mask may be a line-shape mask or an area-shape mask or a pixel-wise mask.

[0018] According to a second aspect, the subject matter disclosed herein relates to a computer-based system configured to carry out an innovative defective image generation method. The innovative images generation systemcomprises a conditional Generative Adversarial Network for generating defective image including synthetic defects. Notably, the conditional Generative Adversarial Network is configured to receiving and process, as a condition, segmentation masks corresponding to defects.

[0019] According to a third aspect, the subject matter disclosed herein relates to a computer-implemented method for automatically identifying defects in images of a machine. The innovative defects identification method uses defective images generated by the innovative images generation method.BRIEF DESCRIPTION OF THE DRAWINGS

[0020] A more complete appreciation of the disclosed embodiments of the invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:Figs. 1A-1E show images used in an embodiment of an innovative images generation method,Fig. 2 shows a flowchart of an embodiment of an innovative images generation method,Fig. 3 shows a schematic block diagram of an embodiment of a generator of an innovative images generation system, andFig. 4 shows a schematic block diagram of an embodiment of a discriminator of an innovative images generation system.DETAILED DESCRIPTION OF EMBODIMENTS

[0021] According to the subject matter disclosed herein a conditional Generative Adversarial Net or Network (usually abbreviated as “GAN”) is used for generating a defective image, i.e. an image that shows a defect (for example a crack or an oxidation in a machine); such a defective image may becalled “synthetic defective image” as it derives from a synthesis process and not simply by a capturing process. Only a small portion of the defective image is actually synthetically generated while the rest of the image is taken from a real, i.e. captured, image of the machine. Notably, a segmentation mask corresponding to a defect to be synthetized is provided to the conditional GAN as a condition for the image generation process; preferably, the mask is designed by a subject matter expert based on his knowledge and experience. As already mentioned, a “segmentation mask” allows to “partition” an image into regions; in the present case, advantageously, the mask allows to distinguish a defect from the rest. The image generation process takes advantage inter alia of the image information contained in the real image surrounding the place where the defect is to be placed.

[0022] The innovative idea has been conceived by focusing on defects in turbomachines. However, it is not to be excluded that it is applicable to other machines or apparatuses.

[0023] The innovative idea will be introduced in the following through a graphic example that should not be construed as a limitation of the subject disclosed herein.

[0024] Fig. 1A shows schematically a real (i.e. captured from reality) nondefective image 110 of a portion of e.g. a turbomachine. In particular, such image is rectangular and has a width (that may be expressed as a pixel number) and a height (that may be expressed as a pixel number).

[0025] According to the subject-matter disclosed herein, if a synthetic defective image is desired, a segmentation image mask is necessary. Fig. IB shows an exemplary segmentation mask 120 in the form of a line-shape mask that may correspond for example to a crack.

[0026] Fig. IB shows also a “bounding area” 125, specifically a “boundingbox”. In particular, it is rectangular shape and has a width (that may be expressed as a pixel number) and a height (that may be expressed as a pixel number). The bounding box may be a “minimum bounding box”, i.e. having the minimum size necessary for containing the segmentation mask, or may be adequately marginated with respect to the “minimum bounding box”, i.e. including a border surrounding the “minimum bounding box”, as shown in Fig. IB (the border may be uniform thus having a size of e.g. few pixels in each direction). Alternatively, the “bounding area” (minimum or not) may have a different shape, for example a square, a circle or an ellipse.

[0027] According to the subject-matter disclosed herein, if a synthetic defective image is desired, a defect position where the synthetic defect will be located is necessary. Such defect position may correspond for example to the center of the defect or the center of the “bounding area” of the defect or to any of the corners of a “bounding box” of the defect (upper left corner, upper right corner, lower right corner, lower left corner).

[0028] According to the subject-matter disclosed herein, the real nondefective image 110 of a portion of a machine is blanked as shown for example in Fig. 1C. Fig. 1C shows an image 130 corresponding to image 110 but wherein an area 135 corresponding to the bounding box 125 at the above- mentioned defect position is blanked (for example all its pixels as set to zero). For example, a computer system receiving a defect position (that may be expressed as X and Y pixel coordinates) as well as a width and e height of the bounding box of the defect mask (that may be expressed as pixel numbers) may set to zero all the pixels of the real non-defective image width the box so located (compare Fig. 1C with Fig. 1 A).

[0029] Starting from image 130 and mask 120, a “patch” 140, that may also be called “defect image”, will be generated through an adequately trained conditional GAN. An example of such “patch” or “defect image” 140 isschematically shown in Fig. ID. Comparing Fig. IB and Fig. ID, one might erroneously think that patch 140 derives simply from superimposing mask 120 over a portion of the background of image 110. However, the generative process carried out by a conditional GAN is extremely more complex as will be explained in the following and has nothing to do with superimposing two images or mixing / fusing two images. It is to be noted that the “defect image” may be generated “directly” or “indirectly”; “indirectly” means that the “defect image” may derive from another artificial image, for example a bigger-size image suitably cut.

[0030] Detailed information regarding GAN in general may be derived for example from the article of Ian Goodfellow et al. entitled “Generative Adversarial Nets” in Advances in Neural Information Processing Systems 27, published in 2014 (or the article still of Ian Goodfellow et al. entitled “Generative Adversarial Networks” in Communications of the ACM, Vol. 63, No. 11, published a bit later i.e. in 2020), and regarding conditional GAN in general from the article of Mehdi Mirza et al. entitled “Conditional Generative Adversarial Nets” in arXiv: 1411.1784, published in 2014. In the following, an advantageous specific embodiment of a conditional GAN will be described, with the reference to Fig. 3 and Fig. 4, that may be used for implementing the present inventive method.

[0031] Finally, the generated patch 140 is placed in the blanked area 135 of image 130 (in other words image 130 and image 140 are combined) and a synthetic defective image 150 is thus created as shown for example in Fig. IE.

[0032] Synthetic defective image 150, or similarly generated images, may be used for training a system for automatically identifying defects in images of a machine, in particular a turbomachine.

[0033] As already explained, the innovative images generation method is based on the use of a conditional GAN. Typically, the conditional GAN needsto be trained, referred in the following as step “a” (see for example block 220 in Fig. 2), and mask data (see for example block 230 in Fig. 2) need to collected, referred in the following as step “b”, before starting generating one or more (usually several) synthetic defective images. In other words, as it is known in the field of Al, the present conditional GAN “learns” how to perform the task of generating synthetic defective images, before performing the task, directly from data provided to the conditional GAN during a so-called “training phase”. Neither generation rules nor defects definitions are preliminary provided to the conditional GAN, but simply annotated defective and nondefective images.

[0034] An innovative images generation method such as for example the one corresponding to the flowchart 200 of Fig. 2, comprises the steps of c) receiving (block 240) a non-defective image of the portion of the machine, wherein the non-defective image has a first bidimensional size, d) receiving (block 250) a segmentation image mask corresponding to a defect, wherein the segmentation image mask defines a bounding area, wherein the bounding area has a second bidimensional size, the second bidimensional size being smaller than the first bidimensional size, e) receiving (block 260) defect coordinates corresponding to a position of the defect, f) generating (block 270) a masked image by blanking an image area of the non-defective image, wherein the image area corresponds to the bounding area and has a size equal to the second dimensional size, wherein the image area is at the position of the defect, and g) using a conditional GAN for generating (block 280) the defective image.

[0035] Notably, the masked image is input to the conditional GAN as image input, and not the whole non-defective image.

[0036] Notably, both the segmentation image mask and the defect coordinates are input to the conditional GAN as condition input, and not simply a label.

[0037] Basically, according to the subject matter disclosed herein, there are two possibilities: either the conditional GAN is used for generating an image having the above-mentioned first bidimensional size (e.g. the size of element 150 in Fig. IE) or it is used for generating an image having the above- mentioned second bidimensional size (e.g. the size of element 140 in Fig. ID). For the sake of clarity, the generated image according to the first possibility will be called “artificial image” and according to the second possibility will be called “defect image”.

[0038] According to a first advantageous possibility, the conditional GAN generates the artificial image having the first bidimensional size, afterwards a partial image having the second bidimensional size and corresponding to the bounding area is derived from the artificial image, and afterwards the partial image is located at the image area of the masked image thus creating the defective image. Typically, the partial image is simple a smaller portion of the artificial image; it must be a portion selected, typically automatically selected, so that the defect is shown.

[0039] According to a second advantageous possibility, the conditional GAN generates the defect image having the second bidimensional size and corresponding to the bounding area (using the terminology introduced with the example, the Conditional GAN generates directly a “patch”), and afterwards the defect image is located at the image area of the masked image thus creating the defective image.

[0040] It is to be noted that, according to this second possibility, while only a “defect image” or “patch”, having the same size as the bounding box of thesegmentation image mask, may be generated directly by the conditional GAN based on the segmentation mask, the generation process takes advantage of whole images of portions of machines. This leads to a high quality of the “defect image” or “patch” and high match between the “defect image” or “patch” and the surrounding image.

[0041] The “bounding area” may be for example a square or a rectangle or a circle or an ellipse or even a combination of these figures; alternative, a “bounding area” may be delimited by a free-form or free-shape closed line.

[0042] Typically, the “bounding area” corresponding to the defect is smaller than the area of the non-defective image (as well as the defective image resulting from the innovative method); in other words, the second bidimensional size is smaller than the first bidimensional size. For example, it may be at least 4 times smaller, for example 2 times smaller in the X direction and example 2 times smaller in the Y direction. Often, it is much smaller, for example 10-100 times smaller. The ratio between the area of the defect and the area of the image may depend on the size of the non-defective image and the size (and type) of the defect.

[0043] The flowchart 200 of Fig. 2 has a BEGIN block 210 and an END block 290 corresponding respectively to the begin and the end of a process for generating a single synthetic defective image. If another synthetic defective image, all these steps may be repeated. However, the activity corresponding to block 230, i.e. collecting mask data, and, especially, the activity corresponding to block 220, i.e. training of the conditional GAN, may be not repeated.

[0044] The conditional GAN may be prepared through the training steps (block 220) of al) receiving a first plurality of images of machines similar to the machine of interest and a corresponding first plurality of segmentation imagemasks, the images of the first plurality being defective images, and training the conditional GAN through said first pluralities, and possibly and advantageously also a2) receiving a second plurality of images of machines similar to the machine of interest and a second plurality of segmentation image masks, the images of the second plurality being non-defective images, and training the conditional GAN through said second pluralities.As it is apparent from the flowchart 200 of Fig. 2, the training steps precede the generation steps, i.e. steps from “c” to “g”.

[0045] As already explained, the generation process requires a segmentation image mask at step “b”. In general, a set of masks may be available prior to carrying out the generation steps, i.e. steps from “c” to “g” .

[0046] According to a first possibility, the segmentation image mask used at step “b” may derive from the step (block 230) of: bl) receiving mask data from a computer program, wherein the computer program is configured to output real defective images through a user interface and to input mask data drawn by a user, for example a subject matter expert, through the user interface. In practice, it is e.g. the subject matter expert that draws the segmentation mask.

[0047] It is to be noted that the subject matter expert may not only contribute to create a segmentation image mask, but also to create a corresponding “bounding area”. Alternatively, a “bounding area” may be determined automatically by a computer program based on e.g. an actual shape of a defect for example after annotation on an image by a subject matter expert.

[0048] According to a second possibility, the segmentation image mask may derive from the step (this is different from the embodiment of Fig. 2) of:b2) receiving mask data from a computer program, wherein the computer program is configured to process real defective images based on predetermined rules and to generate mask data from the processed real defective images.Such rules may derive for example from the knowledge of a subject matter expert. In practice, the segmentation mask is drawn automatically through a piece of software that is configured to extract masks from real images based on such rules.

[0049] The expression “mask data” used in the previous paragraphs means data that define a segmentation image mask, for example its contour and size and at least data that allows to determine whether a pixel in an image to be processed by the mask belongs to the “defect portion” or not.

[0050] According to the subject matter disclosed herein, the segmentation image mask may be of two different types: shape mask or pixel-wise mask. A shape mask is defined through a geometric shape: for example a line (for example straight, curve, piecewise-linear, polyline, spline, ...) or an area (for example rectangle, square, circle, ellipse or a combination of these shapes). A pixel-wise mask is generally an array of pixels (usually a rectangular array of pixels), and the mask may be defined by those pixels being “0” or alternatively by those pixels being “1”. It is to be noted that different types and / or shapes of masks may be used in the same innovative system to better match for example with the type of defect as it will become clearer in the following.

[0051] According to the subject matter disclosed herein, the segmentation mask “partitions”, i.e. divides, an area (for example the area of a “bounding box”) into two portions: a first portion corresponding to a defect and a second portion not corresponding to a defect. In particular, each pixel of the segmentation image mask has a digital value selected between only two different values: a first digital value (for example the binary value “0” or “1”)and a second digital value (for example, correspondingly, the binary value “1” or “0”). The first digital value may correspond to a label value; in this way, by reading this value it is possible for example to extract other information from the mask itself (for example defect type and / or defect cause, as it will become clearer in the following).

[0052] Advantageously, a segmentation image mask may be associated to at least one label, the at least one label providing information regarding the defect for example the type of defect or the cause (e.g. root cause) of defect; a first label may be used for the type and a second label may be used for the cause; the initial n bit of a label may used for the type and the final m bit may be used for the cause. Such annotation of the mask and the defect provide more accurate training and more accurate generation of defective images.

[0053] The type of the defect may be selected for example from “crack”, “erosion”, “oxidation”, “discoloration”, and “spallation”. The following table shows a possible correspondence between the type of defect and the shape of the mask:Defect type Mask shapeCrack LineOxidation AreaDiscoloration Diffused area = pixel -wiseErosion Diffused area = pixel -wiseSpallation Diffused area = pixel -wise

[0054] The cause of the defect (if provided), in particular its accuracy, depends on the knowledge and ability of the subject matter expert. It is to be noted that it is far from immediate to identify the cause from the appearance of the defect. An expert may identify the cause from the appearance of what surrounds the defect. As it is evident being able to determine (or at least “guess”) the cause of a defect from an image showing the defect is extremely advantageous and it goes beyond a simple observation of the image.

[0055] According to advantageous embodiments, a label of the segmentation image mask may correspond to a dimensional indication, in particular (real world) length of the defect (expressed for example in mm) or (real world) area of the defect (expressed for example in mm2) or (real world) length of the bounding area (expressed for example in mm) or (real world) width of the bounding area (expressed for example in mm); more than one label this kind may be used, for example two or three or four labels; the dimensional indication may be for example a number or a range. In other words, when the subject matter expert designs a defect mask, he may provide also an indication of the dimension of the defect he has designed and possibly an indication of the dimension of the associated bounding area.

[0056] According to the above-mentioned embodiments, it is preferable that also the non-defective image used for generating the synthetic defective image carries dimensional information. For example, such dimensional information may be (real world) width of the image (expressed for example in mm) and / or height of the image (expressed for example in mm) and / or size of a detail included in the image. In this way, the defect and possibly its bounding area may be scaled to the image during generation, including blanking. Scaling may be uniform along any direction or, for example, different along two distinct (for example perpendicular) directions or even follow some specific bidimensional formula; especially in this last case, for example, the shape of the scaled defect mask is “distorted” with respect to the shape of the defect mask designed by the subject matter expert.

[0057] Depending on the tool used for designing the defect mask and / or the settings of the tool, dimensional information may be input by the user of the tool or determined automatically by the tool or determined semi-automatically (i.e. with the assistance of the user) by the tool.

[0058] The above described dimensional considerations are not simply aimed at generating very realistic synthetic defective images but have an important impact on the Al system trained through such images. In fact, depending on the size of a defect detected on a component of a machine (but taking also the type of the defect and / or the shape of the defect into account), the component may be considered still operative or to be repaired or to be replaced. Therefore, an Al system that is able not only to detect defects automatically but also to determine their type, their shape and their size automatically would be a better tool for assisting maintenance personnel; this requires corresponding adequate training based on training (defective) images, advantageously synthetic training (defective) image.

[0059] The word “receiving” used in the previous paragraphs with regard to data and / or information means that the computer program carrying out the inventive method obtains by means of hardware and / or software such data and / or information. Such data and / or information may be received for example from a human-machine interface or from a data storage device, such as e.g. a hard-disk, or from a computer network or from a communication line or from another software application running on the same computer or on another computer. Such data and / or information may be generated fully manually, i.e. by a human being, or fully automatically, i.e. through a computer program, or by a human being assisted by a computer program.

[0060] According to preferred embodiments, the conditional GAN used for implementing the present innovative method or for realizing the present innovative system may have a particularly advantageous architecture, as explained through embodiments in the following.

[0061] Typically, the conditional GAN comprises only one generator being neural -network based and only one discriminator being neural -network based. At some point in time, both the generator and the discriminator has to be trained;typically, their training is carried out at the same time, i.e. while the generators learns to generate the discriminator learns to discriminate.

[0062] According to some embodiments as shown for example in Fig. 3, the generator 300 is a neural network with a decoding architecture. Such generator 300 comprises e.g. six decoding blocks 320 being sequentially coupled; any or all of these decoding blocks 320 may comprises a SPADE ResBlock 322 followed by an Upsample 2x block 324. Such generator 300 receives an input signal 310 that is used to produce a synthetic image; the input signal 310 may be a vector VI (e.g. size NxM) containing random numbers (drawn by a Gaussian distribution); optionally, it is possible to input an additional vector V2 containing information about the life history of the portion of machine containing the defect. For example, such life history information may relate to the aging of and / or the level and / or time use of and / or the wear (in particular the estimated wear) of this portion. Such life history information may be provided by a subject matter expert as two values in the range of e.g. 0-100 (similar to the ordinal vector containing patient age and health details in the article of Tian Xia et al. entitled “ Learning to synthesise the ageing brain without longitudinal data" in Medical Image Analysis, published in 2021).

[0063] The sequentially-coupled decoding blocks 320 receive as further input a masked image 330 and a segmentation image mask 340 (as a conditional input); image 330 and image 340 may be “concatenated” in the sense that for each pixel a first set of bits (for example 3x4 or 3x6 or 3x8 bits, in case of an RGB image) derive from the corresponding pixel of image 330 and a second set of bits (for example 1 bit or few bits) derive from image 340. These further input are specifically provided (suitably rescaled) to two SPADE blocks of the SPADE ResBlock. The decoding blocks 320 are followed first by a Conv Block 350 and then by a 3x3 Conv Layer 360; at the end a (synthetic) defective image 370 is output.

[0064] Such generator 300 processes the input vector VI, or the concatenationof vectors VI and V2, using the decoding blocks 320. Each decoding block 320 processes the input and produces a features map. At the end of each decoding block 320, a 2x interpolation is operated to increase the size of the features map so gradually approaching a dimension equal to the desired size of the synthetic image (e.g. size 64Nx64M, being 64 equal to 2 raised to the 6thpower).

[0065] After the decoding blocks 320, the convolutional block 350 projects the features map to a three-channeled array, representing the RGB defective image, that need to be processed by the convolutional layer 360 in order to output the (synthetic) defective image 370.

[0066] Each of the blocks shown in Fig. 3 are known per se to the person skilled in the art of image processing; however, such architecture and the corresponding processing of input data are not known and are inventive: For the sake of completeness, some further explanations and some sources of information will be provided in the following.

[0067] A SPADE ResBlock is known e.g. from the article of Taesung Park et al. entitled “ Semantic Image Synthesis with Spatially-Adaptive Normalization” in the proceedings of the IEEE / CVF conference on computer vision and pattern recognition published in 2019 and shown in figure 2; as shown in figure 4 of this article, the SPADE ResBlock comprises two SPADE blocks. The Upsample 2x block upsamples an input matrix along the width and height dimensions, in particular it processes the input tensor so that the output width and height are twice the initial width and height; upsampling may be performed using the so-called “nearest-neighbour” interpolation. The Conv Block contains a series of two or more so-called “convolutional layers”; the convolutional layers are applied to the block input in a sequential manner. The 3x3 Conv Layer applies a 2D convolution over an input matrix; the convolutionis operated with a series of 3x3 filters. In mathematics, convolution is a mathematical operation on two functions (f and g) that produces a third function (f*g), expressing how the shape of one function is modified by the other.

[0068] According to some embodiments as shown for example in Fig. 4, the discriminator 400 is a neural network with an encoding architecture. Such discriminator 400 comprises e.g. four encoding blocks 430 being sequentially coupled; any or all of these encoding blocks 430 comprises a convolutional layer 432, followed by a Spectral Normalization block 434, followed by a Downsample 2x block 436.

[0069] The sequentially-coupled encoding blocks 430 receive as input a pair of a defective image 410 (that can be either a real one or a synthetic one) and a segmentation image mask 420; image 410 and image 420 may be “concatenated” in the sense that for each pixel a first set of bits (for example 3x4 or 3x6 or 3x8 bits, in case of an RGB image) derive from the corresponding pixel of image 410 and a second set of bits (for example 1 bit or few bits) derive from image 420. A real pair consists of the concatenation of a real defective image and a defect mask. A synthetic pair consists of the concatenation of a synthetic image (previously generated by the generator as described above) and a defect mask used (previously used for generating the synthetic image).

[0070] Such discriminator 400 uses the predicted probability to classify the input pairs as real or synthetic through a Convolutional Layer 440 that receives the output of the sequentially-coupled encoding blocks 430.

[0071] Each of the blocks shown in Fig. 4 are known per se to the person skilled in the art of image processing; however, such architecture and the corresponding processing of input data are not known and are inventive: Forthe sake of completeness, some further explanations and some sources of information will be provided in the following.

[0072] Each spectral normalization block 434 may be implemented according to the teaching contained in e.g. the article of Takeru Miyato et al. entitled “Spectral Normalization for Generative Adversarial Networks'" in the proceedings of the ICLR published in 2018. The Downsample 2x block downsamples an input matrix along the width and height dimensions, in particular it processes an input tensor so that the output width and height are half the initial width and height; downsampling may be performed using the so-called “strided convolution”; a layer performing strided convolution is a variant of the standard convolutional layer; in such a variant, the filters used during convolution are convolved only with a fraction of the values contained in the input tensor; as a result, the output tensor has smaller width and height than the input. In our case, we use a stride equal to two, ensuring that the output width and height of the matrix are half of the input width and height.

[0073] Detailed information regarding convolution (that is implemented i.a. by Conv blocks and Conv Layer blocks) may be found for example in the article of Christian Szegedy et al. entitled “Going deeper with convolutions'" in the proceedings of the CVPR published in 2015.

[0074] If information about life history is fed to the generator, an additional classifier may be used in order to process the features maps extracted by the discriminator and to use them in order to predict the input life history details. Such classifier may be optimized together with the conditional GAN to force the generator to synthesize images matching the provided life history information, similarly to what is described in the article of Xi Chen et al. entitled “Infogan: Interpretable representation learning by information maximizing generative adversarial nets" in Advances in the conference Neural Information Processing Systems 29 published in 2016.

[0075] Embodiments of the innovative method may be incorporated in or associated with a computer-implemented method for automatically identifying defects in images of a machine (in particular a turbomachine). In this case, the defects identification method uses defective images generated through the innovative method, in particular uses them during its training.

[0076] The innovative method may be implemented through a computer-based innovative system configured to carried out such method. The innovative system may incorporate for example the architectures shown in Fig. 3 and / or the architecture shown in Fig. 4 or variants thereof according to what previously described. Essentially, the innovative system may consist in the combination of such architectures or variants.

[0077] Embodiments of the innovative system may be incorporated in or associated with a computer-implemented system for automatically identifying defects in images of a machine (in particular a turbomachine).

[0078] The subject matter described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structural means disclosed in this specification and structural equivalents thereof, or in combinations of them. The subject matter described herein can be implemented as one or more computer program products, such as one or more computer programs tangibly embodied in an information carrier (e.g., in a machine readable storage device), or embodied in a propagated signal, for execution by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). A computer program (also known as a program, software, software application, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarilycorrespond to a file. A program can be stored in a portion of a file that holds other programs or data, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[0079] The processes and logic flows described in this specification, including the method steps of the subject matter described herein, can be performed by one or more programmable processors executing one or more computer programs to perform functions of the subject matter described herein by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus of the subject matter described herein can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0080] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processor of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks, (e.g., internal hard disks or removabledisks); magneto optical disks; and optical disks (e.g., CD and DVD disks). The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0081] To provide for interaction with a user, the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, (e.g., a mouse or a trackball), by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well. For example, feedback provided to the user can be any form of sensory feedback, (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0082] The techniques described herein can be implemented using one or more modules. As used herein, the term “module” refers to computing software, firmware, hardware, and / or various combinations thereof. At a minimum, however, modules are not to be interpreted as software that is not implemented on hardware, firmware, or recorded on a non-transitory processor readable recordable storage medium (i.e., modules are not software per se). Indeed “module” is to be interpreted to always include at least some physical, non-transitory hardware such as a part of a processor or computer. Two different modules can share the same physical hardware (e.g., two different modules can use the same processor and network interface). The modules described herein can be combined, integrated, separated, and / or duplicated to support various applications. Also, a function described herein as being performed at a particular module can be performed at one or more other modules and / or by one or more other devices instead of or in addition to the function performed at the particular module. Further, the modules can beimplemented across multiple devices and / or other components local or remote to one another. Additionally, the modules can be moved from one device and added to another device, and / or can be included in both devices.

[0083] The subject matter described herein can be implemented in a computing system that includes a back end component (e.g., a data server), a middleware component (e.g., an application server), or a front end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described herein), or any combination of such back end, middleware, and front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0084] Although some specific embodiments have been described herein, other embodiments are within the scope and spirit of the subject matter as per the annexed claims.

[0085] It is noted that one or more references are incorporated herein. To the extent that any of the incorporated material is inconsistent with the present disclosure, the present disclosure shall control. Furthermore, to the extent necessary, material incorporated by reference herein should be disregarded if necessary to preserve the validity of the claims.

Claims

CLAIMS1. A computer-implemented method (200) for generating a defective image of a portion of a machine, wherein the defective image shows the portion of the machine containing a defect, wherein the method comprises the steps of: c) receiving (240) an non-defective image of the portion of the machine, wherein the non-defective image shows the portion of the machine containing no defect, wherein the non-defective image has a first bidimensional size, d) receiving (250) a segmentation image mask corresponding to a defect, wherein the segmentation image mask defines a bounding area, wherein the bounding area has a second bidimensional size, the second bidimensional size being smaller than the first bidimensional size, e) receiving (260) defect coordinates corresponding to a position of the defect, f) generating (270) a masked image by blanking an image area of the non-defective image, wherein said image area corresponds to said bounding area and has a size equal to the second bidimensional size, wherein said image area is at said position of the defect, and g) using a conditional Generative Adversarial Network for generating (280) the defective image, wherein the masked image is input to the conditional Generative Adversarial Network as image input, wherein the segmentation image mask and the defect coordinates are input to the conditional Generative Adversarial Network as condition input.

2. The method of claim 1, wherein the segmentation image mask is a line-shape mask or an area-shape mask or a pixel-wise mask.

3. The method of claim 1,wherein each pixel of the segmentation image mask has a digital value selected between only a first digital value and a second digital value.

4. The method of claim 1, wherein the segmentation image mask is associated to at least one label, the at least one label providing information regarding the defect.

5. The method of claim 4, wherein the at least one label corresponds to a type of defect, the type of defect being in particular selected from “crack”, “erosion”, “oxidation”, “discoloration”, and “spallation” .

6. The method of claim 4, wherein the at least one label corresponds to a cause of defect.

7. The method of claim 4, wherein the at least one label corresponds to a dimensional indication, in particular defect length and / or defect area and / or bounding area length and / or bounding area width.

8. The method of claim 7, wherein the non-defective image of step c is associated to a dimensional indication, and wherein the image area to be blanked at step f takes into account the dimensional indication of the segmentation image mask and the dimensional indication of the non-defective image.

9. The method of claim 1, wherein the bounding area is a square or a rectangle or a circle or an ellipse.

10. The method of claim 1, wherein the bounding area is determined automatically by a computer program.

11. The method of claim 1, wherein, at step f, the conditional Generative Adversarial Network generates an artificial image having the first bidimensional size, afterwards a partial image having the second bidimensional size and corresponding to said bounding area is derived from the artificial image, and afterwards the partial image is located at said image area of the masked image thus creating the defective image.

12. The method of claim 1, wherein, at step f, the conditional Generative Adversarial Network generates a defect image having the second bidimensional size and corresponding to said bounding area, and afterwards the defect image is located at said image area of the masked image thus creating the defective image.

13. The method of claim 1, wherein the conditional Generative Adversarial Network is prepared through the training steps (220) of al) receiving a first plurality of images of machines similar to the machine and a corresponding first plurality of segmentation image masks, the images of the first plurality being defective images, and training the conditional Generative Adversarial Network through said first pluralities, and a2) receiving a second plurality of images of machines similar to the machine and a second plurality of segmentation image masks, the images of the second plurality being non-defective images, and training the conditional Generative Adversarial Network through said second pluralities.

14. The method of claim 1, wherein the segmentation image mask derives from the step (230) of bl) receiving mask data from a computer program, wherein the computer program is configured to output real defective images through a user interface and to input mask data drawn by a user through the user interface.

15. The method of claim 1, wherein the segmentation image mask derives from the step of: b2) receiving mask data from a computer program, wherein the computer program is configured to process real defective images based on predetermined rules and to generate mask data from the processed real defective images.

16. The method of claim 1, wherein the conditional Generative Adversarial Network of step e) comprises only one generator being neural -network based and only one discriminator being neural -network based, wherein the generator is trained, and wherein the discriminator is trained.

17. A computer-based system configured to carry out the defective image generation method according any of claims from 1 to 16.

Images