Method of automatically inspecting surface of physical object, automated inspection system, method of manufacturing aircraft component, system, computer program product and storage device

Abstract

The invention relates to a method for automatically inspecting a surface (AS) of a physical object, such as a surface (AS) of an aircraft (100), an automated inspection system, a method for manufacturing an aircraft component (100), a system, a computer program product and a storage device. The method for automatic inspection comprises the step of blurring said surface represented in the form of a depth map image, followed by applying three successive derivative operations to said image. The invention also relates to a system configured to perform the method. Advantageously, it is thus possible to automatically detect planarity defects in the surface (AS) of the aircraft (100) with a level of accuracy that has not yet been achieved.

Description

Technical Field

[0001] The present invention relates to an improved method for automatically inspecting the surface of an object (e.g., the surface of an aircraft component) based on information obtained from a surface inspection device, and a system configured to perform such a method. Background Technology

[0002] Automated inspection and quality control methods used in industry typically result in significant time savings during the product manufacturing phase. This is particularly advantageous for equipment that requires a large number of parts and manufacturing stages, for which increased productivity is often sought. Inspection of aircraft surfaces is common during manufacturing or maintenance operations, requiring automated devices for inspecting and detecting potential surface defects. These devices must provide a level of perception and detection at least as high as that of the human eye, if not higher, to detect very small surface defects during manufacturing or maintenance operations. Surface inspection is also used during regularly repetitive maintenance operations.

[0003] This situation can be improved. Summary of the Invention

[0004] One object of the present invention is to provide an apparatus for automatically inspecting the surface of an object, such as an aircraft surface, which is capable of detecting surface defects with a depth of up to one-third of the resolution of the surface inspection equipment (or acquisition system) used.

[0005] Therefore, a method for automatically inspecting the condition of a solid surface is proposed, the method comprising:

[0006] - Obtain first information, which represents the shape of the surface of the entity object, in the form of a distance map between the surface and the sensor of the distance measuring device.

[0007] - Second information is obtained by applying a fuzzy filter to the first information.

[0008] - The third information is obtained by performing a first derivative operation on the second information using the first derivative operator.

[0009] - The fourth information is obtained by performing a second derivative operation on the third information using the second derivative operator.

[0010] The method also includes:

[0011] - The fifth information is obtained by performing a third derivative operation on the fourth information using the third derivative operator, and then...

[0012] - Based on the fifth piece of information obtained, a defect detection module is used to detect defects in the surface.

[0013] The method according to the invention may also include the following optional features, considered individually or in combination:

[0014] - Automated inspection methods also include steps to notify the presence of defects detected on the surface.

[0015] - An automated inspection method is included in a method for manufacturing aircraft parts, the manufacturing method further including a step of modifying the surface based on at least one piece of information representing a defect detected by the automated inspection method.

[0016] - The fuzzy filter used by the method is determined based on at least one characteristic of the distance measuring device (sensor) used to obtain the first information.

[0017] - The derivative operator used by the checking method makes:

[0018] The first derivative operator is a gradient type operator, the second derivative operator is a Hessian matrix type operator, and the third derivative operator is a non-planar gradient type operator.

[0019] Another object of the present invention is an automated inspection system for inspecting the condition of the surface of a physical object, the system comprising an electronic circuit system configured to:

[0020] - Obtain first information, which represents the shape of the surface of the entity object, in the form of a distance map between the surface and the sensor of the distance measuring device.

[0021] - Second information is obtained by applying a fuzzy filter to the first information.

[0022] - The third information is obtained by performing a derivative operation on the second information using the first derivative operator.

[0023] - The fourth information is obtained by performing a second derivative operation on the third information using the second derivative operator.

[0024] The method also includes:

[0025] - The fifth information is obtained by performing a third derivative operation on the fourth information using the third derivative operator, and then...

[0026] - Based on the fifth piece of information obtained, a defect detection module is used to detect defects in the surface.

[0027] The system according to the invention may also have the following optional features, considered individually or in combination:

[0028] - The automated surface condition inspection system also includes an electronic circuit system configured to perform steps that notify of the presence of defects detected in the surface.

[0029] - An automated inspection system is included in the aircraft component manufacturing system, which also includes means for performing steps to modify the surface based on at least one piece of information indicating a defect in the surface.

[0030] - At least one characteristic of the fuzzy filter used by the automated inspection system is determined based on at least one characteristic of the distance measuring device used to obtain the first information.

[0031] - The derivative operator used by the inspection system makes:

[0032] The first derivative operator is a gradient type operator, the second derivative operator is a Hessian matrix type operator, and the third derivative operator is a non-planar gradient type operator.

[0033] Another object of the present invention is a computer program product comprising program code instructions for performing the steps of the method as previously described when the instructions are executed by a processor of an automated inspection system for the surface of a physical object.

[0034] Finally, the present invention relates to a storage device comprising a computer program product as described above. Attached Figure Description

[0035] Figure 1 An automated inspection system for inspecting the condition of the surface of a physical object is illustrated schematically according to one embodiment.

[0036] Figure 2 The illustration shows a device that can be used. Figure 1 The system shows the surface of the aircraft being inspected.

[0037] Figure 3 A depth (or distance) map is a schematic representation of the surface of an entity object, taken as a set of information and used for... Figure 1 The input data for the automated inspection system has already been shown in the figure;

[0038] Figure 4 This illustrates an embodiment of... Figure 1 The flowchart shown above illustrates the steps of an automated inspection method for inspecting the condition of the surface of a physical object, executed within an automated inspection system.

[0039] Figure 5 It shows about Figure 4Details of the overall derivative steps of the described method; and

[0040] Figure 6 This illustrates, according to one embodiment, the component being configured to perform regarding Figure 4 A diagram illustrating an example of the architecture of the data processing unit of the electronic circuit system described in the method. Detailed Implementation

[0041] Figure 1 This describes an automated inspection system 1 for inspecting the condition of a surface AS of a physical object, according to one embodiment. The automated inspection system 1 includes a sensor device 10 connected to a processing unit 12 via a communication link 11, which, according to one embodiment, is wired. According to a variant, the communication link 11 is configured to operate wirelessly. According to one embodiment, the sensor device 10 is a camera or scanner operating as a distance sensor, configured to deliver a set of “3D point cloud” data that then represents the surface AS being inspected, or more precisely, the surface condition of the surface AS being inspected. The distance sensor device 10 measures the distance between its included reference points and a plurality of points on the surface of the physical object. According to one embodiment, each point in the 3D point cloud is instantiated in the form of an x-coordinate and y-coordinate pair, which is considered in conjunction with a distance value S(x, y) relative to the camera device, which is positioned with reference to the normal of a plane tangent to the surface AS being inspected in the measurement field of the sensor 10 (e.g., the camera device or scanner). For each inspected area of ​​surface AS, a 3D point cloud S(x, y) can be transmitted from sensor device 10 to processing unit 12 via communication link 11. This 3D point cloud S(x, y) constitutes initial information representing the shape of the inspected surface AS (hereinafter also referred to as the "surface condition" of the inspected surface AS). In other words, the 3D point cloud S(x, y) constitutes a depth map that determines a set of distances to points observed and inspected on the surface AS within the measurement field of sensor device 10. Processing unit 12 is configured to perform continuous processing operations on this initial information to detect surface defects present in the inspected surface AS and, if necessary, characterize such surface defects, such as dents and protrusions (impacts, scratches, indentations, depressions, etc.) of various shapes and sizes.

[0042] Figure 2 An example of the surface AS of the aircraft 100 being inspected is shown. According to... Figure 2 In the example described, surface AS is the surface of a fuselage component of aircraft 100. Obviously, this example is not limiting, and the automated inspection system 1 can be useful for inspecting many surfaces of an aircraft, especially the surfaces including the fuselage and wings.

[0043] Figure 3 A data structure containing the aforementioned first information, stored in the form of a matrix S of values, is schematically illustrated. Each element S(X, Y) in this data structure is determined and stored as the X and Y coordinates of a reference plane. For example, element S(X1, Y1) represents the distance between sensor device 10 and a point on surface AS, the location of which is determined spatially by coordinates X1 and Y1. According to one embodiment, the matrix S of values ​​also constitutes an image of distances measured for different points in matrix S. For example, the brighter the point in the image, the greater the distance between the corresponding point on the inspected surface AS and the distance measuring sensor of sensor device 10, and vice versa. The term "corresponding point on the inspected surface" here refers to the point on the inspected surface AS represented by a given point in image (or matrix) S. Thus, the structuring of the acquired initial information constitutes a 3D-to-2D transformation of the inspected surface AS to image S, which can be processed by processing unit 12 for the purpose of detecting the presence of any defects on the inspected surface AS.

[0044] Figure 4 The steps of an automated inspection method performed by and within an automated inspection system 1 according to an embodiment of the present invention are described. Step S0 is an initialization step, at which point all circuits and components of the automated inspection system 1 are powered on, normally supplied with energy, configured, and operated. Specifically, sensor device 10 is positioned on a support or by an operator facing the area of ​​the surface AS to be inspected, and initial information representing the surface condition of the surface AS is transmitted from sensor device 10 to processing unit 12. In step S1, a blurring operation is performed on a matrix (or image) S, which is instantiated using the initial information obtained by processing unit 12. According to one embodiment, the blurring operation performs a blurring of sufficient size to smooth the surface of the object and noise of the measuring tool, but does not remove the defects being sought (e.g., a Gaussian blur with a 2 mm sigma). The matrix or image generated by this blurring constitutes new information representing the surface AS in the form of a blurred image matrix. Next, step S2 involves cleverly performing a triple derivative on the blurred image matrix applied to the information obtained from the blurring. The term "triple derivative" here refers to three consecutive derivative operations applied in a clever manner to the image matrix representing the examined surface AS (or a region or portion of that surface) using three different derivative operators. (The rest of the text is incomplete and appears to be a fragment from a different document.) Figure 5Further details of these operators are described below. Finally, in step S3, the defect detection module is used to perform the detection of one or more possible defects present in the inspected surface AS. According to one embodiment, the defect detection module is implemented by the processing unit 12. According to one embodiment, the defect detection module is based on a conventional segmentation approach, which may use a blob detection algorithm such as a Gaussian difference filter, or a partitioning method such as a k-means algorithm or a watershed algorithm. According to another embodiment, the defect detection module is a neural network (NN) trained to perform segmentation and / or classification tasks. For example, this could be the "YOLO" (You Only Look Once) algorithm or an algorithm from the ResNet (Residual Network) family capable of performing detection and classification simultaneously, or a U-Net segmentation algorithm.

[0045] The selected detection algorithm operates on the image matrix generated by the three consecutive derivatives after blurring, and provides information that enables the identification and localization of any surface defects. According to one implementation, this highlighting is achieved by instantiating bounding boxes, each bounding box being associated with the probability that a defect exists within the region defined by the bounding boxes of the image matrix generated by the three derivatives. Then, a notification step S4 is performed after step S3. Figure 4 During the process (not shown), location information for one or more possible defects detected in the inspected AS surface may be delivered, where applicable, in order to control or organize changes to manufacturing methods and / or maintenance operations.

[0046] Figure 5 Details of step S2, which involves the derivative according to a non-limiting implementation, are provided.

[0047] According to this implementation, step S2 includes three consecutive derivative operations using three derivative operators respectively.

[0048] In the first step S20 (the first sub-step of S2) following step S1, the gradient of the surface being inspected is calculated and expressed as:

[0049]

[0050] Where S(x, y) is the blurred image matrix obtained at the end of the blurring step S1.

[0051] In fact, the first-order differential operator for a surface is the gradient. The gradient allows us to determine, at every point on the surface, a vector quantity that defines the direction and magnitude of the steepest slope at a given point on the surface.

[0052] Therefore, a two-dimensional representation of the surface is obtained in the form of a scalar field corresponding to the norm of the gradient. This representation advantageously provides information about the slope and inclination at each point on the surface being analyzed.

[0053] In the second step S21 (the second sub-step of S2) following step S20, the Hessian matrix is ​​calculated, and it is represented as:

[0054]

[0055] in,

[0056] as well as ,

[0057] This allows local variations in planarity to be determined:

[0058]

[0059] The Hessian matrix (also known as the second derivative matrix) is the equivalent of the gradient in second-order differential geometry. The Hessian matrix is ​​a matrix quantity that provides information about the curvature of the surface at every point on the surface being analyzed.

[0060] Regarding the eigenvalues ​​of the Hessian matrix:

[0061] The eigenvalues ​​of H are called principal curvatures and correspond to the intensity of curvature along the principal directions. Invariant under rotation, these eigenvalues ​​are real values ​​and are expressed as... and , making .

[0062] These eigenvalues ​​of H are calculated as follows:

[0063]

[0064] Where K is either K1 or K2, depending on the operator + or -.

[0065] Regarding the principal direction or second-order canonical coordinates:

[0066] The eigenvectors of H are called principal directions and correspond to the directions of principal curvatures, which are always orthogonal.

[0067] These principal directions are called canonical coordinates. and , making Corresponding to the direction of maximum curvature, and The direction corresponding to the minimum curvature.

[0068] These principal directions are calculated as follows:

[0069] as well as

[0070] in, And among them

[0071] Then the partial derivatives can be determined and used according to the second-order canonical coordinates.

[0072] According to the variation, multiple two-dimensional projections of the Hessian matrix can be obtained, and these projections can be used in the form of scalar fields. Note that in this case, by definition,

[0073] as well as

[0074] According to one implementation, the Gaussian curvature projection G is determined based on the Hessian matrix, which corresponds to the determinant of the Hessian matrix and provides an intrinsic measure of curvature at each point on the surface:

[0075]

[0076] According to one implementation, the mean curvature projection HM is determined based on the Hessian matrix. The mean curvature projection HM corresponds to the average value of the principal curvatures or eigenvalues ​​of the Hessian matrix, and represents the average curvature of the surface at each point.

[0077]

[0078] According to one implementation, a Laplace projection is determined based on the Hessian matrix. The Laplace projection corresponds to the trace of the Hessian matrix (which can also be directly calculated as the divergence of the gradient). The Laplace projection also provides information about the local curvature at each point.

[0079]

[0080] According to one implementation, the planarity deviation projection is determined based on the Hessian matrix. The planarity deviation projection corresponds to the sum of squares of the principal curvatures or eigenvalues ​​of the Hessian matrix, representing the degree of local "nonplanarity" at each point on the surface:

[0081]

[0082] According to one implementation, a projection based on the curvature degree C is determined according to the Hessian matrix, which corresponds to the square root of the planarity deviation:

[0083]

[0084] According to one implementation, a projection based on the shape angle F is determined according to the Hessian matrix. This projection corresponds to the arctangent of the ratio of the principal curvatures and provides information about the local shape of the surface at each point.

[0085]

[0086] Finally, in the third step S22 (the third sub-step of S2) following step S21, a so-called "third" derivative is performed using a new operator that defines the local rate of change of the previously determined planarity, in order to highlight defects (including very small-scale defects) for example by inserting the obtained information into the module used to detect potential defects during step S3 after step S22.

[0087]

[0088] There is no third-order differential operator that can be expressed in a simple form. According to one implementation, it is proposed to apply the first-order derivative operator (gradient) to one of the scalar fields obtained by the second derivative (e.g., the calculation of the gradient of the plane deviation).

[0089] Then, a two-dimensional representation is obtained in the form of a scalar field corresponding to the norm of the gradient. This representation provides information about the rate of change of the curvature or planarity of the surface at each point.

[0090] According to some variations, many other combinations can be performed to obtain third-order differential information (e.g., applying the gradient operator and then calculating the associated scalar field three times consecutively, calculating the Hessian matrix of the scalar field obtained after the first derivative, etc.). All of these variations advantageously allow information related to the rate of change of surface curvature to be highlighted.

[0091] Figure 6 An example of the internal architecture of the processing unit 12 of the automated inspection system 1, which is configured to perform the above-described methods, is illustrated schematically.

[0092] according to Figure 6The hardware architecture shown in the figure includes a processing unit 12 connected via a communication bus 129, comprising: a processor (PROC) or CPU (Central Processing Unit) 121; RAM (Random Access Memory) 122; Read-Only Memory (ROM) 123; a storage unit such as a hard disk (or a storage medium reader such as a Secure Digital Card (SD) card reader (STCK)) 124; and a communication interface module (INTER) 125 that enables the processing unit 12 to communicate with a remote device, such as a sensor device 10 configured to perform surface analysis by measuring distance or other remote equipment (e.g., equipment at an aircraft production site or aircraft maintenance site).

[0093] The processor 121 of the processing unit 12 is capable of executing instructions loaded into RAM 122 from ROM 123, external memory (not shown), storage medium (e.g., SD card), or communication network. When the processing unit 12 is powered on, the processor 121 is capable of reading instructions from RAM 122 and executing them. These instructions form a computer program that enables the processor 121 of the processing unit 12 to implement all or part of the automated inspection method as previously described.

[0094] For example, all or part of such an automated surface inspection method for inspecting aircraft surfaces can then be implemented in software by executing a set of instructions by a programmable machine such as a DSP (Digital Signal Processor) or microcontroller, or in hardware by a dedicated machine or component such as a FPGA (Field Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit). Typically, processing unit 12 includes an electronic circuitry system configured to implement an automated inspection method for the surface of a physical object. Of course, processing unit 12 also includes all the elements typically present in a system comprising a control unit and its peripherals, such as power supply circuitry, power supply monitoring circuitry, one or more clock circuits, reset circuitry, input / output ports, interrupt inputs, bus drivers, and this list is not exhaustive.

Claims

1. A method for automatically inspecting the condition of the surface (AS) of a physical object (100), the method comprising: - Obtain (S0) first information, the first information representing the shape of the surface (AS) of the entity object, the first information being in the form of a distance map between the surface (AS) and the sensors of the measuring device (10). - Second information is obtained by applying a fuzzy filter to the first information, (S1) - The third information (S2, S20) is obtained by performing a first derivative operation on the second information using the first derivative operator. - The fourth information (S2, S21) is obtained by performing a second derivative operation on the third information using the second derivative operator. Furthermore, the method is characterized in that it further includes: - The fifth information (S2, S22) is obtained by performing a third derivative operation on the fourth information using the third derivative operator, and then... - Based on the obtained fifth information, a defect detection module is used to detect defects in the surface (AS).

2. The automated inspection method according to claim 1 further includes the step of notifying (S4) the presence of the defect detected in the surface (AS).

3. A method for manufacturing an aircraft component (100), the method comprising the automated inspection method according to claim 2 and the step of modifying the surface (AS) based on at least one piece of information representing the defect.

4. The method according to any one of claims 1 to 3, wherein, The fuzz filter is determined based on at least one characteristic of the distance measuring device (10).

5. The method according to any one of claims 1 to 4, wherein: The first derivative operator is a gradient operator. The second derivative operator is a Hessian matrix type operator, and The third derivative operator is a non-planar gradient type operator.

6. An automated inspection system (1) for inspecting the condition of a surface (AS) of a physical object (100), the system comprising an electronic circuit system configured to: - Obtain (S0) first information, the first information representing the shape of the surface of the entity object, the first information being in the form of a distance map between the surface and the sensor of the measuring device. - Second information is obtained by applying a fuzzy filter to the first information, (S1) - The third information (S2, S20) is obtained by performing a derivative operation on the second information using the first derivative operator. - The fourth information (S2, S21) is obtained by performing a second derivative operation on the third information using the second derivative operator. Furthermore, the system is characterized in that it further includes an electronic circuit system configured to: - The fifth information (S2, S22) is obtained by performing a third derivative operation on the fourth information using the third derivative operator, and then... - Based on the obtained fifth information, the defect detection module is used to detect (S3) defects in the surface (AS).

7. The automated inspection system (1) for inspecting the condition of a surface (AS) according to claim 6 further includes an electronic circuit system configured to perform a notification step (S4) of the presence of the defect detected in the surface (AS).

8. A system for manufacturing an aircraft component (100), the system comprising an automated inspection system (1) according to claim 7 and means for performing a step of modifying the surface (AS) based on at least one piece of information representing the defect.

9. The system according to any one of claims 6 to 8, wherein, The fuzz filter is determined based on at least one characteristic of the distance measuring device (10).

10. A computer program product comprising program code instructions that, when executed by a processor of an automated inspection system for the surface of a physical object, perform the steps of the method according to any one of claims 1 to 5.

11. A storage device comprising the computer program product according to claim 10.

Images