Method and system for detecting damage to mechanical component, in particular component of aircraft, by multispectral image processing

Abstract

A method implemented by a computer (16) for detecting damage to a component (1), comprising: applying (212) a first neural network (40) to at least one multispectral image (25) of the component (1) formed from a plurality of images (30) of the component (1) in a corresponding spectral band to generate a corresponding mask (62) comprising a respective plurality of pixels, each pixel of the mask (62) is relative to a corresponding portion of the component (1), the first neural network (40) further causing each pixel of the mask (62) to represent a corresponding first level estimate indicating a probability that a portion of the component (1) to which a pixel relates is damaged; applying a second neural network (70) to the data structure (65) based on the mask (62) to generate a second level estimate (999) indicating a probability that the component (1) is damaged; and detecting (228) whether the component (1) is intact or damaged based on the second level estimate (999).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority from European patent application No. 22217219.9 filed on December 29, 2022, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The present invention relates to a method and a system for detecting damage to mechanical components, in particular aircraft components, by means of multispectral image processing. Background Art

[0004] As is known, the need to detect the presence of anomalies (in the sense of damage, also called degradation) of mechanical components of an aircraft is particularly felt in the aviation field in order to ensure flight safety.

[0005] Typically, the search for possible anomalies in aircraft components is carried out by highly specialized personnel who perform visual and / or acoustic inspections of the components. In particular, in the case of acoustic inspections, the component being inspected is repeatedly struck with a mechanical impactor (e.g., a hammer) in order to generate an acoustic response to the impacts; based on this acoustic response, the person responsible for the inspection can detect the possible presence of anomalies in the component, such as, for example, the presence of detachment or delamination. Consequently, acoustic inspections, also known as "tap tests," require the presence of trained personnel, corresponding technical preparation, and considerable practical experience; moreover, this process cannot be automated and is inevitably subject to uncertainties related to the skills and human error of the personnel performing the process.

[0006] However, as far as visual inspection is concerned, it is also inevitably affected by uncertainties related to the skills and human error of the person performing it. In this regard, the results of the inspection can easily be distorted by the presence of dirt on the component or any reflections caused by lighting.

[0007] As an example, patent application US2020 / 0175669A1 discloses an inspection system including an imaging device that generates a first set of images and a second set of images of a workpiece, the first set of images and the second set of images being taken from a first position and a second position relative to the workpiece, respectively. The system generates a first predicted image and a second predicted image based on the first set of images and the second set of images, respectively. The first predicted image and the second predicted image include respective candidate regions. The system merges the first predicted image and the second predicted image and detects the presence of at least one defect of the workpiece depicted in at least one candidate region.

[0008] The paper "Pothole detection using location-aware convolutional neural networks" by Chen Hanshen et al., published in International Journal of Machine Learning and Cybernetics, Springer Berlin Heidelberg, Berlin / Heidelberg, Volume 11, No. 4, February 12, 2022, discloses an image processing method comprising: determining a heat map from an image; extracting a portion of the image based on the heat map; and performing binary classification on the extracted portion to determine the presence of a defect.

[0009] CN111325713A discloses a wood defect detection method, which includes inputting an image into a segmentation neural network to generate a defect prediction mask, and then converting the defect prediction mask into defect information. Summary of the Invention

[0010] It is therefore an object of the present invention to provide a solution that at least partially overcomes the disadvantages of the known art.

[0011] According to the present invention, there is provided a method and a system for detecting damage as defined in the accompanying claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a better understanding of the present invention, some embodiments thereof will now be disclosed, for illustrative and non-limiting purposes only, with reference to the accompanying drawings, in which:

[0013] - Figure 1 shows a block diagram of an anomaly detection system;

[0014] - Figure 2 、 Figure 5 、 Figure 11 、 Figure 16 、 Figure 17 and Figure 19 shows a flow chart according to the present method;

[0015] - Figure 3 shows images of the component acquired in different spectral bands;

[0016] - Figure 4 Three-dimensional multispectral and two-dimensional mask images are shown;

[0017] - Figure 6 An image of a component acquired without illumination is shown;

[0018] - Figure 7 Images of the sample acquired in different spectral bands are shown;

[0019] - Figure 8 a block diagram showing portions of a first neural network and a second neural network;

[0020] - Figure 9 A block diagram illustrating a data structure generated by executing a first neural network and a second neural network;

[0021] - Figure 10 A block diagram illustrating a data structure generated by executing a first neural network and a second neural network during a training step;

[0022] - Figure 12 and Figure 13 Two images in the visible band are shown;

[0023] - Figure 14 Shown from Figure 12 The binary image derived from the image shown;

[0024] - Figure 15 Shown from Figure 14 A smoothed image derived from the binary image shown; and

[0025] - Figure 18 A two-dimensional mask, a three-dimensional matrix and vectors are schematically shown. DETAILED DESCRIPTION

[0026] As an example, refer to Figure 1 The present method for detecting damage to a component of an aircraft is described by the inspection system 10 shown in FIG. 1 , which is shown as operating, for example, on a component 1 , where it is not known in advance whether the component 1 is intact or damaged. In particular, the inspection system 10 includes an illumination device 12 , an image acquisition device 14 , and a computer 16 .

[0027] The illumination device 12 is controllable so as to illuminate the component 1 with a plurality of radiation having various spectral bands (hereinafter referred to as illumination beams). In particular, the number of illumination beams is equal to NUM_BW; for example, it is assumed below that NUM_BW = 9. The spectral bands may have an amplitude of, for example, 20 nm and extend within various ranges centered around, for example, the following values: 720 nm, 655 nm, 614 nm, 585 nm, 520 nm, 490 nm, 465 nm, 440 nm, and 420 nm.

[0028] In particular, if Figure 2As shown, for each illumination beam, the illumination device 12 illuminates (block 100) the component 1 with the aid of the illumination beam, and the image acquisition device 14 acquires (block 102) a corresponding image of the component 1 illuminated by the illumination beam; this image is therefore an image relating to a single spectral band and is sent from the image acquisition device 14 to the computer 16. In this way, the computer 16 acquires nine images of the component, as shown in FIG. Figure 3 , wherein the image is denoted by 20 ; furthermore, in the following, the preliminary image 20 is referred to as the image 20 .

[0029] In more detail, the lighting device 12 and the image acquisition device 14 may be controlled by a computer 16 so as to time the illumination of the component 1 and the acquisition of the preliminary image 20 in a well-known manner.

[0030] Each preliminary image 20 is formed by a matrix of pixels; for example, assuming nine preliminary images 20 have dimensions equal to N×M (e.g., N=1200 and M=1920). In addition, the pixels of each preliminary image 20 indicate corresponding light intensity values in the spectral band to which the preliminary image 20 relates.

[0031] The computer 16 then generates (block 104, Figure 2 )Multispectral image 25( Figure 4 ).

[0032] For example, to generate the multispectral image 25, the computer 16 may control the lighting device 12 and the image acquisition device 14 to perform Figure 5 The operation shown.

[0033] In detail, the computer 16 turns off the lighting device 12 and controls the image acquisition device 14 to acquire (block 200, Figure 5 ) The first reference image 27 of component 1 ( Figure 6 ).

[0034] Thus, the first reference image 27 is an image of the component 1 obtained when the lighting device 12 is off and is therefore subject to any stray radiation from the environment in which the component 1 is arranged. In this regard, in this description it is assumed that the component 1 remains in the same arrangement relative to the lighting device 12 and the image acquisition device 14; thus, the arrangement of the component 1 during the acquisition of the first reference image 27 is the same as the arrangement of the component 1 during the acquisition of the nine preliminary images 20.

[0035] The computer 16 then subtracts (block 202, Figure 5 ) a first reference image 27 in order to generate a corresponding intermediate image (not shown) which relates to the same spectral bands of the corresponding preliminary image 20.

[0036] Thereafter, the computer 16 controls the lighting device 12 and the image acquisition device 14 to acquire ( Figure 5 204) of the reference sample (not shown) nine images 29 (e.g. Figure 7 ), hereinafter referred to as sample image 29. For example, the sample is formed of a flat surface having a high reflectivity (eg, greater than 99%) in all spectral bands.

[0037] In more detail, each sample image 29 relates to a corresponding spectral band.

[0038] The computer 16 then subtracts (block 206, Figure 5 ) a first reference image 27 in order to generate corresponding images (not shown), hereinafter referred to as standard images; each standard image relates to the same spectral band of the corresponding sample image 29.

[0039] The computer 29 then divides each intermediate image by (block 208, Figure 5 ) corresponding standard image, so as to generate the corresponding processed image 30 ( Figure 4 Nine (nine are shown in the figure), each processed image 30 relates to the same spectral band as the intermediate image and the standard image. In this way, nine processed images 30 are obtained, each image relating to a corresponding spectral band. More specifically, considering any processed image 30, the light intensity of each pixel of the processed image 30 is equal to the ratio of the light intensity of the corresponding pixel in the corresponding intermediate image to the light intensity of the corresponding pixel in the corresponding standard image.

[0040] Then, the computer 16 makes (block 210, Figure 5 ) The nine processed images 30 are aggregated to form the multispectral image 25 mentioned above.

[0041] For example, the aggregation occurs so that the multispectral image 25 is a three-dimensional matrix of pixels having dimensions 9xNxM. In other words, by indexing the three dimensions of the multispectral image 25 with k, i, j, the pixels of the multispectral image 25 can be indicated by the notation MS[k, i, j]. Furthermore, if k = k* (k* is equal to any integer between 1 and 9), then MS[k*, i, j] is equal to the k*th processed image 30. Furthermore, assuming k = 1, ..., 9, i = i* and j = j* (where i* is an integer in the range [1 to 1200] and j* is an integer in the range [1 to 1920]), the nine pixels M[k, i*, j*] relate to the same portion of the component 1.

[0042] The computer 16 then applies a first neural network 40 to the multispectral image 25, the first neural network being a convolutional neural network and Figure 8 In the qualitative and illustrative examples, a sequence of multiple convolutional layers is included; without any loss of generality, Figure 8 In the example shown, there are a first convolutional layer 41, a second convolutional layer 42, a third convolutional layer 43, and a fourth convolutional layer 44 connected in sequence.

[0043] In a well-known manner, each of the first, second, third and fourth hidden layers 41 to 44 provides for the execution of: one or more convolution operations based on a plurality of respective filters, each convolution operation being followed by an activation operation and an optional normalization operation; and a pooling operation (optional).

[0044] In practice, the multispectral image 25 is processed by a sequence of first, second, third and fourth convolutional layers 41-44 of the first neural network 40 (block 212, Figure 5 ). To this end, the multispectral image 25 is provided as input to a first convolutional layer 41 of a first neural network 40.

[0045] like Figure 9 As shown qualitatively in FIG, the execution of the operations of each of the first, second, third and fourth convolutional layers 41-44 of the first neural network 40 results in the generation of a three-dimensional matrix of corresponding numerical elements, which is represented by Figure 9 They are represented by 51, 52, 53 and 54 respectively.

[0046] In particular, reference is made to the feature matrix 54 to indicate the three-dimensional matrix generated by the fourth convolutional layer 44 of the first neural network 40, which has dimensions equal to, for example, 1024 x (N / 8) x (M / 8) under the assumption that the fourth convolutional layer 44 has a number of corresponding filters equal to 1024.

[0047] The first neural network 40 also includes a convolution filter 55 (e.g. Figure 9 ), which has dimensions 1x1. Therefore, the computer 16 applies the convolution filter 55 (block 214, Figure 5 ) in the characteristic matrix 54 in order to obtain a two-dimensional matrix 62 (e.g. Figure 4 and in a simplified manner in Figure 9 ), the two-dimensional matrix 62 has a dimension equal to, for example, (N / 8)×(M / 8). Hereinafter, the two-dimensional matrix 62 is referred to as a mask 62.

[0048] In detail, the first neural network 40 is such that each pixel of the mask 62 relates to a corresponding portion of the component 1 , that is to a corresponding portion of the multispectral image 25 , understood as a corresponding subset of pixels of the multispectral image 25 , said subset of pixels comprising the pixels of each of the nine processed images 30 relative to said portion of the component 1 , which in turn correspond to corresponding pixels of the nine preliminary images 20 , since there is a doubly unique relationship between each pixel of each processed image 30 and the corresponding pixel of the corresponding preliminary image 20 .

[0049] Specifically, with reference to MASK[J,K] for indicating a pixel of mask 62 having coordinates J and K, such a pixel from mask 62 may refer to pixel group MS[i,j,k], where i=1,2,...,9, j=J,J+1,...J+7 and j=K,K+1,...K+7, as in reference pixel MASK[1,1] in Figure 4 In other words, in this case, each pixel of the mask 62 relates to a corresponding portion of the component 1 represented by a corresponding group of 64 pixels in each processed image 30 of the multispectral image 25 .

[0050] Furthermore, the pixels of mask 62 have values ranging between 0 and 1. Specifically, the value of each pixel of mask 62 is an estimate of the probability that the portion of component 1 to which the pixel relates is intact or damaged. In other words, mask 62 represents the segmentation of multispectral image 25 estimated by first neural network 40.

[0051] Thereafter, the computer 16 aggregates the feature matrix 54 and the mask 62 (block 216, Figure 5 ) to generate the updated matrix 65, Figure 9 In fact, by observing that the feature matrix 54 is formed by a number NUM of two-dimensional matrices (in this example, it is assumed that NUM=1024) having the same dimensions as the mask 62, the updated matrix 65 includes i) the same two-dimensional matrices as the feature matrix 54 (the number is equal to NUM), and also includes the mask 62 as the NUM+1th two-dimensional matrix.

[0052] The computer 16 then applies a second neural network 70 to the updated matrix 65 , which together with the first neural network 40 forms a third neural network 75 .

[0053] As shown qualitatively and by way of example only, Figure 8 As shown in , the second neural network 70 comprises a corresponding sequence of multiple convolutional layers; without any loss of generality, Figure 8In the example shown, there are a first convolutional layer 71, a second convolutional layer 72, a third convolutional layer 73 and a fourth convolutional layer 74 connected in sequence.

[0054] In a well-known manner, each of the first, second, third and fourth convolutional layers 71-74 provides for the performance of: a plurality of convolution operations based on a plurality of respective filters, each convolution operation being followed by an activation operation and an optional normalization operation; and a pooling operation (optional).

[0055] In practice, the updated matrix 65 is processed by a sequence of first, second, third and fourth convolutional layers 71-74 of the second neural network 70 (block 218, Figure 5 ). To this end, the updated matrix 65 is provided as input to the first convolutional layer 71 of the second neural network 70.

[0056] like Figure 9 As shown qualitatively in FIG, the execution of the operations of each of the first, second, third and fourth convolutional layers 71-74 of the second neural network 70 results in the generation of a three-dimensional matrix of corresponding numerical elements, which is represented by Figure 9 They are represented by 81, 82, 83 and 84 respectively.

[0057] In particular, reference is made to the classification matrix 84 to indicate a three-dimensional matrix generated by the fourth convolutional layer 74 of the second neural network 70, which has, for example, dimensions equal to 32 x 10 x 10, assuming that the fourth convolutional layer 74 has a number of corresponding filters equal to thirty-two. In other words, in this example, the classification matrix 84 is formed by thirty-two two-dimensional matrices having dimensions of 10 x 10.

[0058] The second neural network 70 also provides, for each two-dimensional matrix of the classification matrix 84, calculating the average value of the relative pixels and extracting the maximum value between the pixels of the two-dimensional matrix (block 220, Figure 5 ).exist Figure 9 In , such operations of computing the average and extracting the maximum are indicated by blocks 90 and 91 respectively forming the pooling step. Furthermore, these operations result in the generation of the average vector (in Figure 9 92 in the figure) and the maximum value vector (in Figure 9 denoted by 93 in FIG); both the mean value vector and the maximum value vector have thirty-two elements.

[0059] The second neural network 70 also provides the average value of the pixels of the mask 62 to be calculated and the maximum value among the pixels of the mask 62 to be extracted (block 222, Figure 5 ).exist Figure 9, the operations of calculating the average value of the mask 62 and extracting the maximum value of the mask 62 are represented by blocks 94 and 95, respectively, which form a further pooling step. In addition, such operations result in the generation of a first data structure 96 storing the average value of the mask 62 and a second data structure 97 storing the maximum value of the mask 62.

[0060] The second neural network 70 also provides that the mean vector 92, the maximum vector 93 and the first data structure 9 and the second data structure 97 are aggregated (block 224, Figure 5 ) to form a macro vector 98 (in Figure 9 ), the macrovector 98 is formed in this example by a number of elements equal to sixty-six.

[0061] The second neural network 70 also provides the macro vector 98 to the fully connected layer 99 (such as Figure 9 As shown), the fully connected layer 99 has a plurality of input nodes (not shown) and output nodes equal to sixty-six, which generates ( Figure 5 Box 226) Estimate 999 of the probability that component 1 is damaged.

[0062] Based on the estimate 999, the computer 16 detects (block 228, Figure 5 ) whether the component 1 is intact or damaged. For example, if the estimate 999 is greater than or equal to the first threshold, the computer 16 detects the presence of damage, and if the estimate 999 is below the first threshold, the computer 16 detects that the component 1 is intact. In addition, in the event that the computer 16 has detected the presence of damage, the computer 16 identifies (block 230, based on the mask 62) Figure 5 ) damaged portion of component 1; to this end, the computer 16 identifies a set of the following portions of component 1 as damaged portions of component 1, which portions of component 1 correspond to pixels whose values of the mask 62 are higher than the second threshold. For example, Figure 18 Another example of a mask 62 is shown in which a damaged area 299 is highlighted, the damaged area 299 being formed by pixels of the mask 62 that exceed the second threshold and therefore corresponding to a damaged portion of the component 1; although in such an example, the damaged area 299 is formed by a single set of adjacent pixels, the damaged area 299 may include a plurality of damaged sub-areas, each formed by a corresponding set of adjacent pixels of the mask 62, the damaged sub-areas being separated from each other. Figure 18, the correspondence between the damaged area 299 and the damaged portion of the component 1 is also highlighted, which correspondence is represented in each processed image 30 of the multispectral image 25 by a corresponding set of pixels (denoted by 399), that is, by a corresponding sub-portion of the processed image 30. In the following, the set of pixels 399 is referred to as a single-band representation 399 of the damaged area 299. In the case where the damaged area 299 includes multiple separate sets of pixels (not shown), each single-band representation 399 is also formed by multiple separate sets of pixels.

[0063] To optimize performance, the first neural network 40 and the second neural network 70 can be trained in the following manner, assuming that there are a plurality of multispectral training images 125 (in Figure 10 1 ), each multispectral training image relates to a respective training component (not shown), not necessarily equal to component 1, and formed by a three-dimensional matrix with dimensions 9xNxM.

[0064] In more detail, each training multispectral image 125 is generated in the same manner as described for reference multispectral image 25 and relates to a component of the aircraft, which may be of the same type as the aforementioned component 1 or a different component; furthermore, the component to which each training multispectral image 125 relates may be intact or may have one or more damaged portions. Furthermore, by denoting the pixel matrix of the generic multispectral training image 125 by TM and assuming k=1, ..., 9, i=i*, and j=j* (where i* is an integer in the range [1 to 1200], and j* is an integer in the range [1 to 1920]), the nine pixels TM[k, i*, j*] relate to the same portion of the training component to which the multispectral training image 125 relates.

[0065] In detail, the multispectral training image 125 is divided in an arbitrary manner to form a training set and a validation set, respectively. Figure 11 The operations shown in .

[0066] For each training multispectral image 125 of the training set, the computer 16 generates ( Figure 11 The reduced image 400 corresponding to the frame 300) is shown. Figure 12 An example of a reduced image 400 is shown in FIG; for example only, Figure 12 The reduced image 400 shown in FIG. 1 relates to a training component of a preliminary image (denoted by 120) in a frequency band centered at 490 nm, which is a portion of a corresponding multispectral image 125, as shown in FIG. Figure 13 shown.

[0067] In more detail, the reduced image 400 is obtained, for example, by performing a well-known reduction operation so that the reduced image 400 has the same size as the mask 62 (in this example, N / 8 x M / 8). For example, by indicating the pixel matrix of the reduced image 400 by RED[i, j] and the pixel matrix of the preliminary image 120 by VIS[i, j], the reduced image 400 can be obtained by so-called resizing based on bilinear interpolation of the preliminary image 120. Therefore, the reduced image 400 is a smaller version of the preliminary image 120 and remains understandable to the observer. In this regard, the fact that the preliminary image 120 is in the frequency band centered at 490 nm is irrelevant; the preliminary image 120 can be in any of the nine spectral bands.

[0068] Thereafter, the user instructs the computer 16 to generate ( Figure 11 The binary image 410 has the same size as the reduced image 400 ( Figure 14 ). Specifically, each pixel of the binary image 410 is alternatively equal to '0' or '1', depending on whether the pixel of the binary image 410 relates to a corresponding intact portion or a damaged portion of the training component.

[0069] In more detail, as in Figure 14 As can be seen in FIG, the observer can control the computer 16 to assign a value of '1' to a set of adjacent pixels of the binary image 410, and if the observer believes that such a set of pixels relates to a damaged portion of the training component, the remaining pixels of the binary image 410 are set equal to '0'. In this way, if the training component is damaged, the binary image 410 has one or more damaged areas 412 ( Figure 14 , ie, groups of adjacent pixels having a value equal to '1', are hereinafter referred to as damaged training regions 412. The edges of each damaged training region 412 are sharp, ie, they are characterized by a transition from '1' to '0' between adjacent pixels.

[0070] Then, starting from each binary image 410, the computer 16 generates ( Figure 11 The smoothed image 420 corresponding to the box 304) is relative to Figure 14 The binary image 410 and the smoothed image 420 are shown as examples in FIG. Figure 15 Shown in.

[0071] In detail, for each binary image 410, if the corresponding smoothed image 420 does not have the damaged training region 412, it is the same as the binary image 410. On the contrary, if the binary image 410 includes at least one damaged training region 412, the computer 16 performs the following operations: processing the binary image 410 so as to expand the extension of each damaged training region 412 (optional operation); and then applying a Gaussian filter (e.g., having a dimension of 7×7) to the previously processed binary image 410.

[0072] For example, in order to expand the extension of each damaged training region 412, the computer 16 can modify the arrangement of the relative edges, for example by setting the pixels of the binary image 410 that were previously equal to '0' and were at a distance from the edge that did not exceed a distance threshold (measured according to a predetermined metric) to be equal to '1'. For example, for a general pixel, the distance can be calculated as the number of adjacent pixels that define the shortest path connecting the pixel to any pixel at the edge of the damaged training region 412. For the purposes of this method, the processing operations of increasing the extension of each damaged training region 412 and the relative details are irrelevant. Therefore, the Gaussian filtering operation can also be applied directly to the binary image 410. However, the same Gaussian filtering operation that allows minimizing any labeling errors (i.e., errors in the assignment of values '0' or '1') at the edges of the damaged training region 412 is optional; therefore, variants are possible in which the binary image 410 is used instead of the smoothed image 420. Furthermore, regardless of the possible enlargement of the damaged training area 412 and / or the application of a Gaussian filter, the binary image 410 may be obtained starting from the corresponding preliminary image 120 rather than the corresponding reduced image 400 and subsequently subjected to a downsizing operation in order to assume the same dimensions as the mask 62 .

[0073] Next, the first neural network 40 and the second neural network 70 are trained in a supervised manner.

[0074] Specifically, the computer 16 trains (block 306, Figure 11 ) The first and second neural networks 40, these labels are alternatively equal to zero if the smoothed image 420 has no damaged region, or equal to one if the smoothed image 420 has at least one damaged region.

[0075] In more detail, the training of the first neural network 40 and the second neural network 70 can be described as follows and as Figure 16In particular, in a well-known manner, the training multispectral images 125 of the training set are subdivided into a collection of training batches; such a collection of training batches forms an epoch; and, unless otherwise stated, the operations performed for each training multispectral image 125 of each training batch are described below, although such operations are performed for all training multispectral images 125 of the training batch.

[0076] The computer 16 initializes the parameter values of the first neural network 40 and the second neural network 70 in a well-known manner and then applies the first neural network 40 (block 500, Figure 16 ) on the multispectral training image 125 to obtain a corresponding mask, referred to as a training mask 162 ( Figure 10 In addition, the computer 16 calculates (block 502, Figure 16 ) as a function of the difference between the training mask 162 and the corresponding smoothed image 420, which serves as the labeled training mask; hereinafter, this error contribution is referred to as the segmentation error contribution. For example, the segmentation error contribution can be equal to the so-called focus loss calculated based on the smoothed image 420 and the training mask 162; however, it is possible to adopt a different metric.

[0077] Then, after applying the first neural network 40 to the multispectral training image 125, the computer 16 uses the reference Figure 5 The training mask 162 is aggregated with the feature matrix (not shown) provided by the first neural network 40 in the same manner as described in block 216 of FIG. 1 to obtain a corresponding update matrix, referred to as the updated training matrix 165 ( Figure 10 In addition, the computer applies the second neural network 70 (block 504, Figure 16 ) to the training mask 162 and the updated training matrix 165 to obtain a corresponding estimate 1000 of the probability that the training component involved in the training multispectral image 125 is damaged (e.g. Figure 10 Here, estimate 1000 refers to the training estimate 1000.

[0078] In a well-known manner, the computer 16 calculates (block 506, Figure 16 ) Another error contribution. Hereinafter, this further error contribution is referred to as a decision error contribution. For example, the decision error contribution may be proportional to the difference between the aforementioned label and the training estimate 1000.

[0079] Based on the decision error contributions and segmentation error contributions obtained for all training multispectral images 125 of the training batch, the computer 16 calculates (block 508, Figure 16 ) and is then updated according to the value of the error function (block 510, Figure 16 ) The values of the parameters of the first neural network 40 and the second neural network 70.

[0080] For example, in a well-known manner, updating the parameter values of the first neural network 40 and the second neural network 70 may envisage calculating the gradient of the error function with respect to each parameter and then updating the value of each parameter based on the corresponding gradient.

[0081] like Figure 16 As shown qualitatively in FIG, the computer can then iterate the operations mentioned in blocks 500 to 510 for the next training batch of multispectral training images 125. Once the training batch is completed, the computer 16 calculates (block 512, Figure 16 ) Estimates of the goodness of training of the parameters of the first neural network 40 and the second neural network 70 and the updated values of the predetermined metric.

[0082] Then, if Figure 17 As shown, once the training batch is completed, that is, once the epoch is completed, the computer 16 updates (block 600, Figure 17 ) error function, as explained in more detail below, and recombining the multispectral training images 125 of the training set to form (block 602, Figure 17 ) a set of new training batches, which form a new epoch. The computer 16 then iterates (block 604) based on the new training batches. Figure 16 , at the end of the new training batch, the computer 16 has updated the values of the parameters of the first neural network 40 and the second neural network 70 and the corresponding estimated values of the training goodness of the first neural network 40 and the second neural network 70.

[0083] The operations mentioned in blocks 600-604 are iterated a predetermined number of times, and then the computer 16 selects (block 606, Figure 17 ) The values of the parameters of the first neural network 40 and the second neural network 70 that correspond to the best of the previously calculated goodness of training estimates.

[0084] In more detail, regarding the operations of updating the error function mentioned in block 600, they may be performed as follows.

[0085] As described above, the operations at blocks 502, 506, and 508 are iterated for each epoch with respect to the calculation of the segmentation error contribution, the calculation of the decision error contribution, and the calculation of the value of the error function. Furthermore, the error function update operation at block 600 causes the calculation of the value of the error function at block 508 to be performed so as to change the weights assigned to the corresponding segmentation error contribution and the corresponding decision error contribution within the error function between one epoch and the next.

[0086] For example, and without any loss of generality, the calculation of the value of the error function mentioned in block 508 may be performed according to the following function:

[0087] F e (b) = f(CS b )+(1-λ e )*f'(CD b )

[0088] Where: index 'e' indicates the epoch, 'b' indicates the common training batch of the e-th epoch; f(CS b ) indicates the value of the first predetermined function calculated based on the segmentation error contribution of the b-th training batch; f'(CD b ) indicates the value of a second predetermined function calculated based on the decision error contribution of the b-th training batch; and λ e indicates a parameter that may, for example, initially be equal to one and may subsequently be decreased at each epoch during the operations of block 600 in order to gradually increase the weight of the decision error contribution within the calculation of the value of the error function.

[0089] In fact, the described training allows avoiding that initially inaccurate training of the first neural network 40 has a negative impact on the training of the second neural network 70, since the weight of the decision error contribution increases over time, ie as the training of the first neural network 40 improves.

[0090] However, a variant is possible in which the first neural network 40 and the second neural network 70 are trained separately. In this case, the first neural network 40 can be trained based on a first partial error function that depends only on the segmentation error contribution. Once the training of the first neural network 40 is completed, and then once the parameter values of the first neural network 40 are established, the second neural network 70 is trained based on a second partial error function that depends only on the decision error contribution; furthermore, during the training of the second neural network 70, the parameter values of the first neural network 40 are not changed.

[0091] Whatever the implementation details regarding the training of the third neural network 75 (understood as a set of first neural networks 40 and second neural networks 70), the structure of the third neural network 75 is such that the mask 62 provided by the first neural network 40 provides a spatial indication of the arrangement of potentially damaged portions in the component 1, these being portions of the component 1 exceeding the aforementioned second threshold relative to the pixels of the mask 62; this spatial and potential damage indication is interpreted by the computer 16 according to a corresponding estimate 999, which confirms whether the portion of the component 1 indicated by the mask 62 is actually damaged.

[0092] In other words, each pixel of mask 62 represents a corresponding first-level estimate indicating the probability that the portion of component 1 to which the pixel relates is damaged, said probability estimated by first neural network 40. Estimate 999 generated by second neural network 70 represents a second-level estimate indicating the probability that component 1 is damaged, i.e., the probability of having at least a damaged portion, said probability accurately estimated by second neural network 70. Therefore, if estimate 999 does not exceed the first threshold, computer 16 detects that component 1 is intact, regardless of the value of the pixel of mask 62. In this way, the presence of second neural network 70 allows for the reduction of false positives generated by first neural network 40.

[0093] Furthermore, the fact that mask 62 is provided at the input of the second neural network 70 causes the information provided by mask 62 with respect to the possibly damaged portion of component 1 to be weighted by the second neural network 70, thereby contributing to the generation of estimate 999 and thereby improving the accuracy of estimate 999.

[0094] According to a variant of the method, in the event that the computer 16 has detected the presence of damage in the component 1 based on the estimation 999, the computer 16 may execute Figure 19 The operation shown.

[0095] In particular, the computer 16 calculates (block 700, Figure 19 ) The spectral characteristics of the damaged area 299 and thus the spectral characteristics of the damaged portion of the component 1 are calculated. For this purpose, reference is made to e.g. Figure 18 In the case shown, the computer 12 calculates for each single-band representation 399 of the damaged area 299 of the mask 62 a corresponding average value of the values of the pixels of the single-band representation 399; Figure 18 In the example shown, such average values are denoted by C1 , C2 , C3 . . . , C9 . The set of nine calculated average values forms a vector VX , which is referred to as the spectral signature VX of the damaged portion of component 1 .

[0096] In practice, the spectral feature VX depends on the reflectivity value in the spectral band of the material forming the damaged portion of the component 1. As regards such a material, it can be, for example, rust or a metallic material (for example steel) which is coated with paint without damage but is exposed to the illumination beam due to damage to the component 1 (for example a scratch).

[0097] The computer 16 then identifies (block 702, Figure 19 )The material of the damaged part of component 1.

[0098] In particular, one or more known spectral features may be obtained in the same manner as described with reference to spectral feature VX, but with respect to a training component having a damaged portion formed of a known material (e.g., rust or a metallic material). It is also possible that the set of known spectral features includes spectral features generated differently; for example, it is possible that nine values of a known spectral feature of a material (e.g., an oily material or fat) are obtained by calculating the average value of processed images 30 of a sample of such a known material.

[0099] In fact, from a mathematical point of view, the known spectral features form a kind of basis of the space of spectral features. Therefore, the operation at block 702 may for example envisage performing a linear regression of the spectral feature VX in order to obtain VX=α1*VN1+α2*VN2+…α u *VN u , where 'u' indicates the number of known spectral features, which is represented by VN1, VN2, ... VN u indicates, and α1,…α u The computer 16 can then select, for example, a corresponding coefficient with respect to u The material with the highest coefficient α among the known spectral characteristics.

[0100] Still like Figure 19 As shown, the computer 16 may also signal the user in a well-known manner (block 704, Figure 19 ) identified material to allow the user to further assess the actual severity of the damage. If the known spectral signature does not include the spectral signature of the material in question, and therefore if the identification at block 702 is unreliable (e.g. because the coefficients a have values that are approximately equal to each other), the computer 16 signals this situation so that it is also possible to perform further analysis.

[0101] The advantages that this solution allows to obtain are clearly apparent from the preceding description.

[0102] In particular, the method makes it possible to detect the presence of surface defects of components under uncontrolled light conditions. Furthermore, the method has been shown to be effective even in the presence of dirt or paint on the component surface.

[0103] However, the system is characterized by low costs and high automation possibilities.

[0104] Finally, it is evident that modifications and variations may be made to the methods and systems described and illustrated herein without departing from the scope of the present invention as defined in the appended claims.

[0105] For example, the number of spectral bands NUM_BW may be different than described.

[0106] The first neural network 40 and the second neural network 70 may be different from those described; for example, they may include a different number of layers. In addition, the first neural network 40 and the second neural network 70 may be trained differently than described.

[0107] In general, the generation of the multispectral image may differ from that described, in which case the generation of the training multispectral image is adapted accordingly. It is also possible that the multispectral image is formed directly from the preliminary image without further processing of the latter, although this may result in reduced accuracy.

[0108] It is also possible that in order to reduce computational complexity, the values of N and M are reduced, in which case it is possible that for each component to be analyzed, multiple multispectral images need to be acquired, which relate to different parts of the component; however, each multispectral image is analyzed in the same manner as described above.

[0109] Additionally, the order in which some of the operations described above are performed may differ from the order described.

[0110] Finally, the method and the detection system may also be applied to mechanical components other than those of aircraft; for example, they may be applied to the structural monitoring of wind turbines or civil infrastructure, and more generally to monitoring the state of health of any mechanical component.

Claims

1. A method for detecting damage to a component (1) implemented by a computer (16), comprising: - applying (212) a first neural network (40) to at least one multispectral image (25) of the component (1) formed from a plurality of images (30) of the component (1) in corresponding spectral bands to generate a corresponding mask (62), the mask (62) comprising a respective plurality of pixels, each pixel of the mask (62) being relative to a corresponding portion of the component (1), the first neural network (40) being further configured such that each pixel of the mask (62) represents a corresponding first-level estimate indicating a probability that the portion of the component (1) to which the pixel relates is damaged; The method further comprises: - applying a second neural network (70) to a data structure (65) based on the mask (62) to generate a second level estimate (999) indicating a probability that the component (1) is damaged; as well as - Based on the second level estimation (999), detecting (228) whether the component (1) is intact or damaged.

2. The method of claim 1 , wherein applying (212) a first neural network (40) to at least one multispectral image (25) comprises generating a feature matrix (54) starting from the multispectral image (25); and wherein the first neural network (40) is configured such that the mask (62) is based on the feature matrix (54); and wherein the data structure (65) is based on the mask (62) and the feature matrix (54).

3. The method according to claim 2, wherein: The first neural network (40) includes a plurality of corresponding convolutional layers (41, 42, 43, 44), which are sequentially connected and configured to generate the feature matrix (54) starting from the multispectral image (25).

4. The method according to claim 2 or 3, wherein the first neural network (52) comprises a convolution filter (52) configured to generate the mask (62) starting from the feature matrix (54).

5. The method according to any one of the preceding claims, wherein the second neural network (70) comprises: - a convolution step (71, 72, 73, 74) configured to generate a data matrix (84) according to said data structure (65); - a first pooling step (90, 91) configured to generate data vectors (92, 93) starting from said data matrix (84); - a second pooling step (94, 95) configured to generate at least one value (96, 97) starting from said mask (62); - an aggregation step (98) configured to generate a macrovector (98) by aggregating the data vectors (92, 93) and the at least one numerical value (96, 97); as well as - a fully connected step (99) configured to generate said second level estimate (999) from the macrovector (98).

6. The method according to any one of the preceding claims, further comprising: If it has been detected (228) that the component (1) is damaged, the damaged portion of the component (1) is identified (230) based on the mask (62).

7. The method according to claim 6, further comprising: - determining a damaged area (299) of the mask (62), the damaged area (299) being formed by pixels of the mask (62) meeting a threshold condition, the damaged area (299) corresponding to a corresponding sub-portion (399) of the image (30) for each image (30) of the component (1); - calculating (700) a spectral signature of the damaged area (299), the spectral signature being formed by a corresponding numerical vector (VX) comprising values for each image (30), the values being based on pixels of a corresponding sub-portion (399) of the image (30); and - identifying (702) a material forming the damaged portion of the component (1) among a plurality of known materials based on the spectral signature (VX) and a plurality of known spectral signatures relative to known materials.

8. The method according to claim 7, wherein: For each image (30), the corresponding value of the numerical vector (VX) is equal to the average value of the pixels of the sub-portion (399) of the image (30).

9. A method according to any one of the preceding claims, wherein The first and second neural networks (40, 70) have been trained based on multispectral training images (125) relative to a training component, based on corresponding labeled training masks (420), and based on corresponding labels, each label indicating whether the training component is intact or damaged.

10. The method according to claim 9, wherein: Each labeled training mask (420) includes pixels, each pixel being relative to a corresponding portion of a corresponding training component and having a value depending on whether the corresponding portion of the training component is intact or damaged.

11. The method according to claim 10, wherein: Each labeled training mask (420) has a size equal to the mask (62) and is obtained by performing the following operations: - generating (303) a corresponding binary image (410), each pixel being equal to a first value ('0') if the corresponding portion of the training component is intact, or equal to a second value ('1') if the corresponding portion of the training component is damaged; as well as - generating a labeled training mask (420) based on the binary image (410).

12. The method according to any one of claims 9 to 11, wherein The first and second neural networks (40, 70) have respective parameters and have been trained by performing steps forming (602) a series of epochs of a plurality of batches of training multispectral images (125), and for each batch in each epoch: - applying (500) the first neural network (40) to each multispectral training image (125) of the batch to obtain a corresponding training mask (162), and calculating (502) a corresponding segmentation error contribution based on the training mask (162) and the corresponding labeled training mask (420); - for each multispectral training image (125) of the batch, applying (504) a second neural network (70) to a training data structure (165) based on the corresponding training mask (162) to generate a second-level training estimate (1000) indicating a probability that the corresponding training component (125) is damaged, and calculating (506) a decision error contribution as a difference between the second-level training estimate (1000) and the corresponding label; - calculating (508) the value of an error function according to the respective weights, the error function depending on the segmentation error contribution and the decision error contribution with respect to the batch; - updating (510) parameter values of the first neural network and the second neural network (40, 70) based on the calculated value of the error function; And wherein, during the series of epochs, the weight of the decision error contribution increases.

13. A processing system comprising means (16) configured to perform the method according to any one of claims 1 to 12.

14. A system comprising: - A processing system (16) according to claim 13; - an illumination device (12) configured to illuminate the component (1) with illumination light beams having, for each illumination light beam, a wavelength within a range in a corresponding spectral band; and An acquisition device (14) coupled to the processing system (16) and configured to acquire a preliminary image (20) of the component (1) in a corresponding spectral band, the multispectral image (25) being based on the preliminary image (20).

15. A computer program comprising instructions which, when executed by a computer (16), cause the computer (16) to perform the method according to any one of claims 1 to 12.

16. A computer medium readable by a computer (16) having stored thereon a computer program according to claim 15.

Images