Method for detecting and characterizing defects on micrograph of object by artificial intelligence

Abstract

A method for identifying and characterizing surface defects on an object by means of electronic processing is described. The method comprises the following steps: acquiring, by means of a microscope, at least one microphotograph or digital image of an object or a part of the object to be confirmed for the defect; then providing the acquired at least one microphotograph or digital image to an algorithm trained by means of artificial intelligence and / or machine learning technology; a defect is then identified by a trained algorithm and the identified defect is ascertained by means of a first mask, and a first processed image is provided as an output of the trained algorithm, in which the identified defect is ascertained by the first mask. The method also provides for processing the first processed digital image by means of filtering and morphological closing operations for highlighting the edges of the defects in a cleaner and clearer manner and by means of segmentation operations adapted to highlight the individual defects visually confirmed in the image, a second processed image and a second processed mask adapted to confirm the defect and highlight and characterize the contour and shape of the defect in an improved manner are obtained. The method further comprises the step of applying a computer vision technique / algorithm to the aforementioned second processed image to determine characterization information for each of the detected defects. For each of the identified defects, this characterization information comprises at least the following information: a defect category, a position of the defect relative to a reference coordinate system associated with the image, at least one geometric / dimensional parameter of each defect. The at least one trained algorithm is trained in a preliminary training step performed on training digital images, each of which is marked by means of marking a known defect and filtered to produce a mask corresponding as much as possible to the marked defect. A system for detecting and characterizing defects on a micrograph of an object is also described, which system is capable of performing the above method.

Description

Technical Field

[0001] The present invention relates to a method for detecting and characterizing defects on micrographs of objects through artificial intelligence (AI) and electronic processing. Background Art

[0002] The use of artificial intelligence (AI) and computer vision (CV) techniques to detect photographically reproduced defects and to quantify them in terms of size is currently known. These known techniques and tools involve the analysis of manually or automatically (partially or completely) acquired images of possible defects by means of appropriately designed algorithms (usually one or more neural networks).

[0003] Among the types of images to which such known techniques can be applied include microscopic images of general materials (eg, metals and / or crystals).

[0004] Micrographs allow the observation of the material's microstructure. This is highly interesting and can be used for a variety of purposes, including, for example, checking the quality of the product's internal matrix. In fact, the macroscopic mechanical properties of an alloy are largely determined by the microstructure.

[0005] In the case of components with a safety function, such as brake calipers, fulfilling the mechanical resistance requirements is even more important.

[0006] The ability to identify, classify, and measure defects in a product's microstructure in a timely manner is crucial for providing feedback to production, thereby avoiding financial losses and improving production efficiency. Furthermore, systematically collecting information related to defects is a prerequisite for improving manufacturing processes and design guidelines.

[0007] However, the methods proposed in the prior art for detecting defects on micrographs by leveraging the potential of artificial intelligence and computer vision have some obvious limitations and disadvantages.

[0008] First, during the training step, the tagging process commonly employed involves drawing a rectangle around each observable defect, without taking into account the often irregular shapes of the defects themselves. Therefore, tagging according to known techniques can significantly limit the performance of artificial intelligence algorithms.

[0009] Furthermore, in the prior art, in order to identify which pixels of the considered digital image belong to defects, a segmentation operation needs to be applied to the output of a properly trained neural network. This can negatively impact the segmentation results because the masks generated by the neural network often do not completely correspond to the defects.

[0010] Furthermore, in existing techniques, neural networks are applied to a single image, whereas the analysis typically must be performed on the entire sample.

[0011] If the sample to be analyzed is larger than the field of view of the microscope used to capture a single image or micrograph, the sample is scanned as follows: the image acquisition system implemented in the microscope moves the sample, capturing an image of each part of the sample until the captured images cover the entire surface of the sample.

[0012] Since the number of images acquired for the entire sample is typically multiple (e.g., in the case of a portion of a brake caliper, approximately hundreds of digital images), it is not possible in the prior art to apply one or more neural networks to a combined image of the individual images (e.g., due to known limitations on the number of pixels that the neural networks described in the literature can process). A combined image is an image obtained by juxtaposing the individual images belonging to a single sample in the order in which they were acquired, horizontally and vertically, in a plane without leaving space or creating overlap.

[0013] Knowing the location of a defect in the reference frame of a specimen (and therefore of the entire specimen) is very important in metallurgy, since the presence of the same defect in different locations of a product can affect its mechanical properties in different ways.

[0014] Based on the above description, there is a clear need to design a method for detecting and characterizing defects on micrographs of objects by means of an appropriate combination of artificial intelligence (AI) and computer vision (CV), which method is improved compared to the above-mentioned known solutions and therefore meets the above-mentioned technical needs that have not yet been fully met by the prior art. Summary of the Invention

[0015] The object of the present invention is to provide a method for detecting and characterizing defects on micrographs of objects by electronic processing, at least partially obviating the drawbacks mentioned above with respect to the prior art and responding to a need felt in particular in the considered technical field.

[0016] This object is achieved by a method according to claim 1 .

[0017] Further embodiments of the method are defined in claims 2 to 24 .

[0018] A further object of the present invention is to provide a system for detecting and characterizing defects in micrographs of objects, which system is capable of implementing the above method. This object is achieved by a system according to claim 25.

[0019] Further embodiments of the system are defined in claims 26 to 30 . BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other characteristics and advantages of the method and system according to the invention will become apparent from the following description of a preferred embodiment given by way of non-limiting indication with reference to the accompanying drawings, in which:

[0021] - Figure 1 shows a digital image processed according to the steps of an embodiment of the method and, in particular, in the upper part of the figure, shows the image on which a first labeling operation is performed, and in the lower part of the figure, shows the image further processed by applying an Otsu filter;

[0022] - Figure 2 Two examples of defect categories that can be identified by a trained algorithm included in an embodiment of the method according to the invention are shown;

[0023] - Figure 3 In the upper part, a micrograph (digital image) acquired by means of a microscope is shown, and in the lower part, the corresponding digital image processed by applying a deep learning algorithm and a segmentation operation according to the steps of the method of the invention is shown;

[0024] - Figure 4 shows a digital image further processed according to the steps of the method of the present invention, wherein size parameters calculated for individual defects and defect clusters are graphically shown;

[0025] - Figures 5 to 8 1 shows corresponding examples of graphical user interfaces (GUIs) and related functions provided by embodiments of the method according to the present invention; in particular, Figure 5 shows an example of a GUI in which a user can select an image to be processed; Figure 6 shows an example of a GUI in which the user can view the mask processed by the algorithm on each image; Figure 7 An example of a GUI is shown in which a user can view the position information and relative size information of each defect in the image reference frame; Figure 8 shows an example of a GUI in which a user can manually define a region around each defect and restart the computer vision algorithm if the previous results are deemed unsatisfactory;

[0026] - Figure 9 shows an exemplary microscopic analysis report generated by a method according to an embodiment;

[0027] - Figure 10 A simplified block diagram of a system suitable for performing the above method according to an embodiment of the present invention is shown;

[0028] - Figure 11A simplified block diagram shows a further embodiment of the system according to the invention, which is capable of carrying out the method according to the invention in another manner. DETAILED DESCRIPTION

[0029] Reference Figures 1 to 11 , a method for identifying and characterizing surface defects on an object by means of electronic processing is described.

[0030] The method comprises the following steps: acquiring, by means of a microscope, at least one micrograph or at least one digital image of an object or a part of the object for which defects are to be confirmed; then, providing the acquired at least one micrograph or digital image to an algorithm trained with the aid of artificial intelligence and / or machine learning techniques; then, identifying defects with the aid of the trained algorithm and confirming the identified defects with the aid of a first mask; and providing a first processed image as an output of the trained algorithm, in which the identified defects are confirmed with the aid of the first mask.

[0031] The method also provides for processing the first processed digital image by means of filtering and morphological closing operations for highlighting the edges of defects in a cleaner and clearer manner, and by means of a segmentation operation suitable for visually highlighting individual defects identified in the image, thereby obtaining a second processed image and a second processed mask suitable for identifying defects and highlighting and characterizing their contours and shapes in an improved manner.

[0032] The method further includes the step of applying computer vision techniques / algorithms to the second processed image to confirm characterization information of each of the detected defects.

[0033] For each of the identified defects, such characterization information comprises at least the following information: defect category, location of the defect relative to a reference coordinate system associated with the image, at least one geometric / dimensional parameter of each defect.

[0034] The at least one trained algorithm is trained in a preliminary training step performed on training digital images based on a training data set, each of the training digital images being labeled by labeling a known defect and filtered to produce a mask that corresponds as closely as possible to the labeled defect.

[0035] According to an embodiment, each of the training digital images is filtered by applying an Otsu filter.

[0036] The Otsu filter is a digital image processing technique known per se (see, for example, Wuli Wang et al., “Fast Image Segmentation Using Two-Dimensional Otsu Based on Estimation of Distribution Algorithm,” Journal of Electrical and Computer Engineering, Vol. 2017, No. 1735176, https: / / doi.org / 10.1155 / 2017 / 1735176).

[0037] According to an embodiment of the method, the aforementioned processing step by means of a morphological closing operation comprises image processing operations by means of dilation and erosion.

[0038] The above-mentioned "morphological closing" operations and in particular the "dilation" and "erosion" operations are known per se in the field of digital image processing / recognition.

[0039] For example, the closing operation is performed using a combination of "erosion" (i.e., removal of "scattered" black pixels) and "dilation" (compensation for the initial erosion). In this way, a "cleaner" mask is obtained with fewer isolated points (thus not negatively affecting the defect measurement) and, most importantly, a more precise demarcation of defects between individual points (avoiding the formation of "bridges").

[0040] According to an embodiment of the method, the above-mentioned training algorithm for identifying defects and confirming the identified defects by means of the first mask comprises a deep learning algorithm / model.

[0041] According to an implementation option, the above-mentioned training algorithm for identifying defects and confirming the identified defects with the help of the first mask includes a Mask-RCNN algorithm / model based on a neural network, which is trained based on the open source COCO training set.

[0042] According to an embodiment of the method, the aforementioned preliminary training step operates from a pre-trained algorithm based on a pre-trained dataset other than the training dataset, thereby applying transfer learning techniques to implement the trained algorithm.

[0043] In this context, transfer learning (TL) methods are used to build machine learning (ML) algorithms, i.e., to select an algorithm that has been pre-trained on another dataset. In the field of machine learning, transfer learning is a technique that involves applying knowledge gained while solving a similar problem to solve a new one. Repurposing, or transferring, information from previously learned tasks to learn new ones has the potential to significantly improve the recognition performance of machine learning algorithms, particularly deep learning algorithms.

[0044] According to an embodiment, the method is applied to at least one of the following objects: the object comprises a metallurgical / metal object.

[0045] According to another embodiment, the method is applied to at least one object comprising an object made of ceramic material in an industrial, geological or mineralogical sample.

[0046] According to another embodiment, the method is applied to a brake caliper of a vehicle brake system.

[0047] According to different possible implementations, the above-mentioned defects that can be detected with the aid of the method include the presence of one or more of the following defects: oxides, and / or shrinkage cavities, and / or bubbles, and / or inclusions, and / or cracks, and / or joints, and / or cleaning defects of the sample.

[0048] According to an embodiment of the method, the at least one geometric / dimensional parameter that can be determined for each defect comprises the total area of ​​the defect.

[0049] According to another embodiment of the method, the above-mentioned step of determining the characterization information of each defect among the detected defects includes: confirming all pixels belonging to the detected defect under consideration on the second processed image; determining the minimum area rectangle containing all pixels belonging to the detected defect under consideration; and determining the maximum length of the longest side of the minimum area rectangle.

[0050] In this case, the at least one geometric / dimensional parameter that can be determined for each defect therefore comprises the maximum length of the longest side of such a minimum area rectangle.

[0051] According to another embodiment of the method, the above-mentioned step of determining the characterization information of each defect in the detected defects includes: confirming all pixels of the extended group or cluster of adjacent defects on the second processed image; and determining a minimum area rectangle, which contains all pixels belonging to the extended group or cluster of adjacent defects under consideration; and determining the maximum length of the longest side of the minimum area rectangle.

[0052] In this case, the at least one geometric / dimensional parameter that can be determined for each defect therefore comprises the maximum length of the longest side of the minimum area rectangle.

[0053] According to an embodiment, the method is applied to a plurality of micrographs or digital images of a sample of the object in order to analyze the entire sample, even when the entire sample is larger than the field of view of the microscope from which the micrographs or digital images originate.

[0054] Thus, in this case, the acquisition step comprises sequentially acquiring the plurality of micrographs or digital images from a sample of the object.

[0055] According to an implementation option of this embodiment, the method comprises performing the above-mentioned steps of identifying and confirming defects on each of a plurality of micrographs or digital images by means of a trained algorithm, and providing a plurality of corresponding first processed images.

[0056] The method further includes juxtaposing the first processed images horizontally and vertically in a plane without leaving space or generating overlap to obtain an extended visual plane reconstructed image.

[0057] In this case, the method ultimately includes: performing the step of processing the first processed digital image on the extended visual plane reconstructed image to obtain a second extended processed image; and performing the above-mentioned step of applying computer vision technology / algorithm on the above-mentioned second extended processed image to determine the characterization information of each defect among the detected defects.

[0058] According to a specific implementation option, the extended visual plane reconstructed image is an overall image of the object sample, and the second extended processed image is a second overall processed image of the object sample.

[0059] According to another form of implementation, the method comprises providing an extended digital image by juxtaposing acquired photomicrographs or digital images of the sample, and in a preliminary training step providing the extended digital image and each of the acquired photomicrographs or digital images representing respective parts of the extended digital image to a first trained neural network in addition to the above-mentioned trained algorithm for selecting a subset of the respective digital images or micrographs among the acquired digital images or micrographs based on criteria related to the presence and / or significance and / or ease of detectability of defects.

[0060] The method also provides: performing the above steps of identifying defects and confirming the identified defects on each digital image of the above selected subset of digital images, and providing a corresponding first processed image through the above trained algorithm (in this case, including the second neural network).

[0061] In this case, the method ultimately includes: performing a processing step by means of filtering and morphological closing operations on each of the first processed images to obtain a plurality of corresponding second processed images; and performing a step of applying computer vision techniques / algorithms to each of the second processed images to determine information characterizing each of the detected defects.

[0062] Depending on the specific implementation option, the above criteria for selecting the digital images to be analyzed include:

[0063] - Select the digital image with the highest number of defects, and / or

[0064] - selecting a digital image comprising one or more defects of a type predetermined to be relevant, and / or

[0065] - Selecting digital images from a specific area of ​​the sample predetermined to be relevant and having the greatest number of defects, for example close to an edge.

[0066] According to an embodiment, the method further comprises the steps of providing a computerized graphical user interface to the operator and allowing the operator to perform one or more of the following actions with the aid of the graphical user interface:

[0067] - selecting one or more micrographs or digital images to be processed and analyzed from among the collected micrographs or digital images of the object sample; and / or

[0068] - displaying on each selected micrograph or digital image the mask processed by the trained algorithm and / or the first processed digital image and / or the second processed digital image; and / or

[0069] - for each defect identified, displaying the category of the defect, and / or the position of the defect relative to a reference coordinate system associated with the digital image, and / or said at least one geometric / dimensional parameter of the defect; and / or

[0070] If the result is not satisfactory, the area surrounding the defect in question is manually defined in the digital image and the execution of the method steps is restarted.

[0071] According to an embodiment of the method, all identified defects and all information determined in connection with each defect are summarized in an electronic report that can be accessed by a computer.

[0072] According to an implementation option of this embodiment, the electronic report comprises information about the exceeding of a threshold value relating to a maximum number of defects, a maximum number of defects of a certain type, a maximum size of defects or defect clusters.

[0073] According to an embodiment, the method is performed by a local electronic processing device in communication with a digital image or micrograph acquisition device.

[0074] According to another embodiment of the method, the collected digital image to be processed is stored in the cloud, and / or the first processed image and the second processed image are stored in the cloud so as to be accessible by a network application.

[0075] According to another embodiment of the method, the above steps of identifying defects and confirming the defects, processing the first processed digital image by means of filtering and closing operations, and applying computer vision techniques / algorithms to the second processed image are performed partially or completely in a distributed manner using resources provided by a cloud-type electronic processing architecture.

[0076] According to an embodiment, the method provides a mechanism for classifying defects based on their impact on the proper functioning and safety of the product, including a priori partitioning of the component into non-overlapping regions. For each region, constraints are set to adhere to, such as a maximum number of defects and a maximum length per defect.

[0077] Refer again Figures 1 to 11 , a system for detecting and characterizing defects on micrographs of objects is described.

[0078] The system includes: a digital image acquisition device configured to acquire at least one micrograph or at least one digital image of an object or a portion of the object for which defects are to be confirmed through a microscope; and an electronic processing device configured to receive the acquired at least one digital image, identify defects using at least one algorithm trained with the help of artificial intelligence and / or machine learning techniques and running in the electronic processing device, and confirm the identified defects with the help of a first mask.

[0079] The at least one trained algorithm is trained in a preliminary training step performed on training digital images based on a training set, wherein each of the training digital images is marked by marking a known defect and is filtered to produce a mask that corresponds as closely as possible to the marked defect.

[0080] The electronic processing device is further configured to:

[0081] - providing a first processed image as an output of the trained algorithm, in which first processed image the identified defects are confirmed by means of the first mask;

[0082] - processing the aforementioned first processed digital image by means of filtering and closing operations for highlighting the edges of defects in a cleaner and sharper manner, and by means of a segmentation operation suitable for visually highlighting the individual defects identified in the image, thereby obtaining a second processed image and a second processed mask suitable for identifying the defects and highlighting and characterizing their contours and shapes in an improved manner;

[0083] - Applying computer vision techniques / algorithms to the second processed image to determine characterization information for each of the detected defects, wherein, for each of the confirmed defects, the characterization information includes at least the following information: defect category, location of the defect relative to a reference coordinate system associated with the image, and at least one geometric / dimensional parameter of each defect.

[0084] According to an embodiment of the system, the electronic processing device is configured to provide a graphical user interface to the user / operator, the graphical user interface being configured to allow the user / operator to perform one or more of the following actions:

[0085] - selecting one or more micrographs or digital images to be processed and analyzed among the micrographs or digital images acquired of the object sample; and / or

[0086] - displaying the mask processed by the trained algorithm, and / or the first processed digital image, and / or the second processed digital image on each selected micrograph or digital image; and / or

[0087] - for each defect identified, displaying the category of the defect, and / or the position of the defect relative to a reference coordinate system associated with the digital image, and / or said at least one geometric / dimensional parameter of the defect; and / or

[0088] If the result is not satisfactory, the area surrounding the defect in question is manually defined in the digital image and the execution of the method steps is restarted.

[0089] According to an embodiment of the system, the above-mentioned micrograph or digital image acquisition device comprises an electron microscope.

[0090] According to an embodiment of the system, the electronic processing device includes a local electronic processor or an edge device operatively connected to the acquisition device.

[0091] According to another embodiment of the system, the electronic processing device includes a user device, a remote server and a remote computer, which are connected to each other in the cloud and are operatively connected to the digital image acquisition device and the user device capable of providing a graphical user interface through a network.

[0092] According to multiple possible implementations, the system is configured to execute the method according to any one of the implementations of the above method.

[0093] Hereinafter, the embodiment of the present invention will be referred to Figures 1 to 11 Additional details of the method are given for illustrative purposes only and not for limiting purposes.

[0094] The image acquisition system implemented in the microscope is capable of acquiring digital images of a sample of the object to be analyzed, which has been appropriately prepared for analysis and placed in the observation area. The digital images thus acquired represent the input to a machine learning (ML) model or algorithm, which is capable of identifying possible defects in the digital images.

[0095] The following describes how this embodiment of the method performs the steps typically provided for developing a machine learning algorithm; input preparation, labeling, and model training.

[0096] In this method, input preparation steps are avoided because the algorithm used uses digital images acquired directly by the microscope as input. This is advantageous from the perspective of computational load and thus time savings.

[0097] "Marking" activities (often qualified by the term "tagging") include:

[0098] - manually marking defects shown in the image generated by the microscope, for example by tracing the smallest rectangle that contains the entire defect (“weak marking” or “weak labeling”);

[0099] - Applying an Otsu filter to the digital image labeled in the manner described above to produce a mask that overlaps as closely as possible with the labeled defects in the image.

[0100] Figure 1 Shown are examples of weakly labeled images before and after applying the Otsu filter.

[0101] Accurate labeling (achieved by embodiments of the method, as shown above) is a fundamental requirement for a well-functioning deep learning algorithm to ensure that labeled pixels correspond only to locations where defects exist, thereby improving the performance of the model.

[0102] The labeling step is followed by a training process: a subset of the training dataset with labeled images—for example, comprising at least 100 images for each defect type to be identified—is provided as input to an artificial intelligence / machine learning algorithm to calibrate the model parameters and adapt them to make predictions.

[0103] According to specific implementation options, the aforementioned subset of the labeled training dataset is enriched by data augmentation techniques.

[0104] In this embodiment, a transfer learning process is used to build a machine learning algorithm, that is, an algorithm pre-trained on another dataset is selected. In the example shown in this article, the Mask-RCNN model based on a neural network is selected from various available transfer learning processes (for example, see "Mask R-CNN" - Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, 2018 - https: / / arxiv.org / abs / 1703 - the actual implementation version is https: / / github.com / matterport / Mask_RCNN) and trained on the open source COCO dataset (for example, the version https: / / cocodataset.org / #home).

[0105] The identification of defects and the extraction of some of their information include the following steps.

[0106] (i) Apply the trained algorithm to each image collected by the microscope according to the above procedure to identify and classify possible defects. The categories of defects identified include: oxides, shrinkage cavities, bubbles, inclusions, cracks, see Figure 2 , where shrinkage defects (“shrinkage” in the figure) and defects caused by the presence of oxides (“oxides” in the figure) are shown.

[0107] (ii) Computer vision (CV) processing including OTSU-type filters, dilation, and erosion is applied at the output of the neural network to clean and fit the edges accurately.

[0108] (iii) applying a segmentation operation to the image processed as described in step (ii) above to visually highlight the defects present in the image (see Figure 3 ).

[0109] (iv) For each defect, extract and save some information from the image, including in this example:

[0110] a. Defect category;

[0111] b. Position in the image's reference frame;

[0112] Using classic computer vision techniques, the following parameters are also calculated, first expressed in number of pixels and then converted to metric units of measurement by standard methods (see e.g. Figure 4 ):

[0113] c.Total area;

[0114] da, the maximum length of the longest side of the minimum area rectangle that contains all pixels of the extended defect;

[0115] eL is the maximum length of the longest side of the minimum area rectangle that contains all pixels of the extended defect cluster. A cluster is defined as a group of defects such that for each defect i of length ai there is at least one other defect j of length aj at a distance d such that: d < (ai + aj) / 2.

[0116] For mechanical components that are subject to thermal cycling during their service life, such as brake calipers for example, clusters are very important for evaluating the material and therefore the mechanical properties of the product.

[0117] As the temperature rises during use, spatially close defects coalesce, forming discontinuities in the material matrix that are larger than the individual starting defects. The presence of clusters therefore represents a factor that deteriorates the mechanical properties of the material.

[0118] In this embodiment, the choice of applying computer vision (CV) operations (step (ii)) and segmentation (step (iii)) downstream of the object recognition algorithm (object detection - step (i)) is because most state-of-the-art "object detection" algorithms return rectangular bounding boxes, which are insufficient for calculating the aforementioned size parameters for each defect. In fact, bounding boxes tend to merge nearby defects in a more or less arbitrary manner.

[0119] Furthermore, state-of-the-art semantic segmentation algorithms fail to return masks that are accurate enough to compute defect areas and often combine defective regions in an arbitrary manner. To correct this behavior and improve the quality of the generated masks, step (ii) above is applied, which is therefore an improvement.

[0120] According to an embodiment, the operator has available a graphical user interface with the help of which he can:

[0121] - Select one or more images to be processed (see e.g. Figure 5 );

[0122] - View the mask processed by the algorithm on each image (see for example Figure 6 );

[0123] - For each defect, the position information in the image reference system and at least the size information as described above are displayed (see e.g. Figure 7 );

[0124] - For each defect, if the result was considered unsatisfactory, the region around the defect was manually defined and the algorithm was restarted (see e.g. Figure 8 ).

[0125] In this implementation example, the information collected on one or more analyzed images is ultimately advantageously summarized in a report (see, e.g., Figure 9 ).

[0126] The described method is applicable to any microscopic image acquired with a microscope. However, defect detection is usually performed on the entire sample. If the sample to be analyzed is larger than the microscope's field of view, the described method can be applied to the entire sample in two ways, namely according to two implementation options, hereinafter referred to as the "bottom-up approach" and the "top-down approach."

[0127] A. Bottom-up approach

[0128] The described algorithm is applied individually to each of the acquired images. The images output by the algorithm are juxtaposed horizontally and vertically in a plane against each other without leaving space or creating overlaps.

[0129] B. "Top-down" approach

[0130] The second method involves training a neural network to select a selected number of individual images from the composite image and applying the algorithm to these selected images individually. The selected individual images can, for example, correspond to images with the highest number of defects or images with a specific type of defect, or they can be images from specific areas of the sample, such as near an edge. An operator can also manually select which areas of the sample to apply the algorithm to using a graphical user interface.

[0131] This second method is advantageous in terms of computing time since it is limited to examining only a subset of the images forming the sample, ie only the images selected as being the most important.

[0132] Regardless of the method used, the computer implementing the algorithm stores the position information of the sample in the reference system for each individual image, so the information about the defect contained therein is mapped into the reference system of the sample.

[0133] Depending on implementation options, this information can therefore be displayed in the sample reference frame by means of a graphical user interface.

[0134] Furthermore, for each defect identified, the distance from the outer surface of the product can be inferred. This information is also communicated to the operator via a graphical user interface.

[0135] Depending on the implementation option, information about defects identified by means of the analysis of a single image or about a plurality of images belonging to the same sample is aggregated at the image and sample level, respectively.

[0136] Depending on the implementation options, quality limits are established for the sample and if these quality limits are not met, a notification is sent to the operator via a graphical user interface.

[0137] Examples of such constraints include, but are not limited to, a maximum number of defects, a maximum number of defects of a certain type, a maximum size of a defect or defect cluster, and the like.

[0138] According to an embodiment, in a "top-down" architecture, computer vision algorithms (and in this case, not one or more neural networks) are used to identify the most severely defective areas. This embodiment has the advantage of reducing the time required to train deep learning algorithms due to their greater flexibility. For example, if images with a different scale than those used for training are fed as input to a deep learning algorithm, this can degrade its performance. In contrast, by using computer vision techniques to analyze images with a different scale than those used for training, performance should remain unchanged.

[0139] As can be seen, the above objects of the present invention are fully achieved by the above method with the help of the features shown in the above detailed description. The advantages and technical problems solved by the method according to the present invention have been described above with reference to the various features and aspects of the method.

[0140] Those skilled in the art may modify and adjust the embodiments of the above method, or replace elements with other functionally equivalent elements to meet possible needs without departing from the scope of the appended claims. Each of the features described above as belonging to possible embodiments can be implemented without considering the other embodiments described.

Claims

1. A method for detecting and characterizing defects on a micrograph of an object by means of electronic processing, the method comprising the following steps: - taking by means of a microscope at least one micrograph or at least one digital image of the object or a part of the object for which defects are to be identified; - providing said at least one acquired micrograph or digital image to at least one algorithm trained by means of artificial intelligence and / or machine learning techniques, wherein at least one trained algorithm is trained in a preliminary training step performed on training digital images based on a training set, each of said training digital images being marked by marking known defects and each of said training digital images being filtered in order to generate a mask that corresponds as closely as possible to the marked defects; - identifying defects by means of the trained algorithm and confirming the identified defects by means of a first mask, and providing a first processed image as an output of the trained algorithm, in which first processed image the identified defects are confirmed by means of the first mask; - processing said first processed digital image with the aid of filtering and morphological closing operations for highlighting the edges of defects in a cleaner and sharper manner, and a segmentation operation suitable for visually highlighting the individual defects identified in the image, thereby obtaining a second processed image and a second processed mask suitable for identifying defects and highlighting and characterizing their contours and shapes in an improved manner; - applying computer vision techniques / algorithms to the second processed image to determine characterizing information for each of the detected defects, wherein, for each of the identified defects, the characterizing information includes at least: Defect category, The position of the defect relative to the reference coordinate system associated with the image, At least one geometric / dimensional parameter for each defect.

2. The method according to claim 1, wherein Each of the training digital images is filtered by applying an Otsu filter.

3. The method according to any one of the preceding claims, wherein The processing step by means of the morphological closing operation comprises image processing operations by means of dilation and erosion.

4. The method according to any one of the preceding claims, wherein In order to achieve the trained algorithm, the preliminary training step operates from a pre-trained algorithm based on a pre-training dataset other than the training dataset, thereby applying a transfer learning technique.

5. The method according to any one of the preceding claims, wherein The trained algorithm for identifying defects and confirming the identified defects with the aid of the first mask comprises a deep learning algorithm / model.

6. The method according to any one of the preceding claims, wherein The trained algorithm for identifying defects and confirming the identified defects with the help of the first mask includes a neural network-based Mask-RCNN algorithm / model, which is trained based on the open source COCO training set.

7. The method according to any one of the preceding claims, wherein The object to which the method is applied includes a metallurgical / metal object, or the object to which the method is applied includes an object made of ceramic material in an industrial, geological or mineralogical sample.

8. The method according to any one of claims 1 to 6, wherein: The object to which the method is applied comprises a brake caliper for a vehicle braking system.

9. The method according to any one of the preceding claims, wherein The defects detectable by means of the method include the presence of one or more of the following defects: oxides, and / or shrinkage cavities, and / or bubbles, and / or inclusions, and / or cracks, and / or joints, and / or cleaning defects of the sample.

10. The method according to any one of the preceding claims, wherein The at least one geometric / dimensional parameter that can be determined for each defect includes the total area of ​​the defect.

11. The method according to any one of the preceding claims, wherein The step of determining the characterizing information for each of the detected defects comprises: identifying on the second processed image all pixels belonging to the detected defect under consideration; and determining a minimum area rectangle containing all pixels belonging to the detected defect under consideration; and determining the maximum length of the longest side of the minimum area rectangle, Therein, the at least one geometric / dimensional parameter that can be determined for each identified defect thus comprises the maximum length of the longest side of the minimum area rectangle.

12. The method according to any one of the preceding claims, wherein The step of determining the characterizing information for each of the detected defects comprises: identifying all pixels of an extended group or cluster of adjacent defects on the second processed image; and determining a minimum area rectangle containing all pixels belonging to the extended group or cluster of adjacent defects under consideration; and determining the maximum length of the longest side of the minimum area rectangle, The at least one geometric / dimensional parameter that can be determined for each confirmed defect includes the maximum length of the longest side of the minimum area rectangle.

13. The method according to any one of the preceding claims, wherein The method is applied to a plurality of micrographs or digital images of an object sample for analysis of the entire sample, even when the entire sample extends to be larger than the field of view of the microscope from which the micrographs or digital images originate, and wherein the acquisition step thus comprises sequentially acquiring a plurality of said micrographs or digital images from the object sample.

14. The method according to claim 13, comprising: - performing said step of identifying and validating defects by means of said trained algorithm on each of said plurality of micrographs or digital images and providing a plurality of corresponding first processed images; - juxtaposing the first processed images horizontally and vertically in a plane without leaving spaces or generating overlaps to obtain an extended visual plane reconstructed image; - performing said step of processing said first processed digital image on said extended visual plane reconstructed image to obtain a second extended processed image; - performing said step of applying computer vision techniques / algorithms on said second extended processed image to determine characterising information for each of the detected defects.

15. The method according to claim 14, wherein The extended vision plane reconstructed image is an overall image of the object sample, and wherein the second extended processed image is a second overall processed image of the object sample.

16. The method according to claim 13, comprising: - providing an extended digital image by juxtaposing said photomicrographs or digital images of the acquired sample; - in a preliminary training step, providing, in addition to the trained algorithm, both the extended digital image and each of the acquired micrographs or digital images representing respective portions of the extended digital image to a first trained neural network, in order to select a subset of the respective digital images or micrographs among the acquired digital images or micrographs based on criteria related to the presence and / or importance and / or detectability of defects; - performing, on each digital image of the selected subset of digital images, said step of identifying defects by means of said trained algorithm comprising a second neural network and confirming said identification, and providing a corresponding first processed image; - performing said processing step by filtering and morphological closing operations on each of said first processed images to obtain a plurality of corresponding second processed images; - performing said step of applying a computer vision technique / algorithm on each of said second processed images to determine characterising information for each of said detected defects.

17. The method according to claim 16, wherein The criteria used to select the digital images to be analyzed include: -Select the digital image with the highest number of defects, and / or - selecting digital images comprising one or more types that are predetermined to be related, and / or - Selecting digital images from a specific area of ​​the sample predetermined to be relevant and having the greatest number of defects, for example, said area being close to an edge.

18. The method according to any one of the preceding claims, further comprising the steps of: Provide a computerized graphical interface to the operator; and allowing the operator to perform one or more of the following actions with the aid of the graphical interface: - selecting one or more micrographs or digital images to be processed and analyzed among the acquired micrographs or digital images of the object sample; and / or - displaying on each selected micrograph or digital image the mask processed by the trained algorithm, and / or the first processed digital image, and / or the second processed digital image; and / or - for each defect identified, displaying the category of the defect, and / or the position of the defect relative to a reference coordinate system associated with the digital image, and / or said at least one geometric / dimensional parameter of the defect; and / or If the result is not satisfactory, the area surrounding the defect in question is manually defined in the digital image and the execution of the steps of the method is restarted.

19. The method according to any one of the preceding claims, wherein All identified defects and all information determined to be relevant to each defect are summarized in a computer-accessible electronic report.

20. The method according to claim 19, wherein The electronic report includes information regarding the exceeding of thresholds associated with: a maximum number of defects, a maximum number of defects of a certain type, a maximum size of a defect or a defect cluster.

21. The method according to claim 20, wherein For each of the groups of non-overlapping areas into which the object under inspection is divided a priori, the criteria of the maximum number of defects, and / or the maximum number of defects of a certain type, and / or the maximum size of defects or defect clusters are determined in order to classify the defects according to their impact on the correct function and / or safety of the object.

22. A method according to any preceding claim, performed by a local electronic processing device in communication with a digital image or micrograph acquisition device.

23. The method according to any one of claims 1 to 21, wherein The collected digital image to be processed is stored in the cloud, and / or the first processed image and the second processed image are stored in the cloud so as to be accessible by a network application.

24. A method according to any one of claims 1 to 21 or claim 23, wherein The steps of identifying defects and confirming the identified defects, the steps of processing the first processed digital image by means of filtering and closing operations, and the steps of applying computer vision techniques / algorithms to the second processed image are partially or fully performed in a distributed manner using resources that can be provided by a cloud-type electronic processing architecture.

25. A system for detecting and characterizing defects in a micrograph of an object, the system comprising: a digital image acquisition device configured to acquire, through a microscope, at least one micrograph or at least one digital image of the object or a portion of the object for which defects are to be identified; an electronic processing device configured to: receive the at least one acquired digital image; and identify defects by means of at least one algorithm running in the electronic processing device and to confirm the identified defects by means of a first mask, the algorithm being trained by means of artificial intelligence and / or machine learning techniques, wherein at least one of the trained algorithms has been trained in a preliminary training step performed on training digital images based on a training set, each of the training digital images being marked by marking known defects and each of the training digital images being filtered to generate a mask that corresponds as closely as possible to the marked defects; Wherein, the electronic processing device is further configured to: - providing a first processed image as an output of the trained algorithm, in which first processed image the identified defects are confirmed by means of the first mask; - processing said first processed digital image with the aid of filtering and morphological closing operations for highlighting the edges of defects in a cleaner and sharper manner, and a segmentation operation suitable for visually highlighting the individual defects identified in the image, thereby obtaining a second processed image and a second processed mask suitable for identifying defects and highlighting and characterizing their contours and shapes in an improved manner; - Applying computer vision techniques / algorithms to the second processed image to determine characterization information for each of the detected defects, wherein, for each of the confirmed defects, the characterization information includes at least the following: the defect category, the position of the defect relative to a reference coordinate system associated with the image, and at least one geometric / dimensional parameter of each defect.

26. The system of claim 25, wherein: The electronic processing device is configured to provide a graphical interface to a user / operator, the graphical interface being configured to allow the user / operator to perform one or more of the following actions: - selecting one or more micrographs or digital images to be processed and analyzed among the acquired micrographs or digital images of the object sample; and / or - displaying on each selected micrograph or digital image the mask processed by the trained algorithm, and / or the first processed digital image, and / or the second processed digital image; and / or - for each defect identified, displaying the category of the defect, and / or the position of the defect relative to a reference coordinate system associated with the digital image, and / or said at least one geometric / dimensional parameter of the defect; and / or If the result is not satisfactory, a region surrounding the defect in question is manually defined in the digital image and the execution of the method steps is restarted.

27. A system according to any one of claims 25 to 26, wherein: The micrograph or digital image acquisition device comprises an electron microscope.

28. A system according to any one of claims 26 to 27, wherein The electronic processing device includes a local electronic processor or edge device that is operatively connected to the acquisition device.

29. A system according to any one of claims 25 to 27, wherein The electronic processing device includes a user device, a remote server and a remote computer, wherein the remote server and the remote computer are connected to each other in the cloud, and the remote server and the remote computer are operatively connected to the digital image acquisition device and the user device capable of providing a graphical interface through a network.

30. The system of any one of claims 25 to 29, configured to perform the method of any one of claims 1 to 24.

Images