Defect detection method based on deep learning and multi-modal images and related device

Abstract

The application provides a kind of nondestructive testing method based on deep learning and related device.It relates to image processing technical field.The nondestructive testing method includes: obtaining fixed image, floating image and fixed image registration after registration image and registration image and fixed image fusion after fusion image.And, establish the multi-modal detection dataset of fixed image, registration image and fusion image.Construct multi-modal target detection network, and train multi-modal target detection network by multi-modal detection dataset, wherein multi-modal target detection network includes feature extraction module, feature fusion module and detection module.Input multi-modal detection dataset into the multi-modal target detection network after training, to output result as defect detection result.The application uses multiple images as samples, trains multi-modal target detection network, and improves the analysis accuracy of nondestructive testing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a defect detection method and related apparatus based on deep learning. Background Technology

[0002] Non-destructive testing (NDT) is a technique that uses ultrasonic, radiographic, and infrared technologies to detect defects in materials, parts, and equipment without damaging or affecting their performance. Currently, the most common NDT method involves irradiating industrial parts with X-rays and using X-ray images to detect defects. However, X-rays have low sensitivity to some materials, making it difficult to accurately detect residual substances belonging to that material within industrial parts. Conversely, X-rays have high sensitivity to metals, enabling high-precision visualization of structural defects in metallic industrial parts.

[0003] Neutron rays are highly sensitive to elements with light atomic numbers and can detect residual substances that are not detected by X-rays at higher precision. However, neutron rays are less sensitive to metals, resulting in poor image quality, unclear structures, and a tendency to produce large differences in pixel values ​​in local areas. This makes it difficult to detect structural defects in industrial parts made of metal with high precision, and areas with large differences in pixel values ​​are easily marked as defects.

[0004] Taking aero-engine turbine blades as an example, aero-engine turbine blades are cast using ceramic cores. The channels of the turbine blades are complex, and the defects are due to the lightweight material. X-ray images can clearly show the structural defects of the turbine blades, such as cracks and voids, but they cannot show the residual materials on the turbine blades with high precision. Neutron images can clearly show the residual materials generated during the casting process of the turbine blades, but they cannot show the structural defects of the turbine blades with high precision.

[0005] Therefore, there is an urgent need for a non-destructive testing technology based on multimodal image acquisition that can combine the advantages of different image acquisition methods, in order to improve the defect detection rate and reduce the missed detection rate of non-destructive testing. Summary of the Invention

[0006] In view of the above problems, the present invention is proposed to provide a defect detection method and related apparatus based on deep learning and multimodal images that overcomes or at least partially solves the above problems. It can solve the problem that existing single-modal images cannot fully and accurately display sample defects, thereby improving the accuracy of defect detection and analysis.

[0007] Specifically, the present invention provides a defect detection method based on deep learning and multimodal images, characterized in that it includes:

[0008] A fixed image, a floating image, a registered image after registration with the fixed image, and a fused image after merging the registered image and the fixed image are obtained; and a multimodal detection dataset composed of the fixed image, the registered image, and the fused image is established.

[0009] A multimodal object detection network is constructed and trained using the multimodal detection dataset. The multimodal object detection network includes a feature extraction module, a feature fusion module, and a detection module.

[0010] The multimodal detection dataset is input into the trained multimodal target detection network, and the output result is used as the defect detection result.

[0011] Optionally, training the multimodal target detection network using the multimodal detection dataset includes:

[0012] The feature extraction module extracts image features from the fixed image, the registered image, and the fused image, respectively, and denoted as fixed image features, registered image features, and fused image features.

[0013] The feature fusion module concatenates the fused image features with the fixed image features and adds them to the registration image to obtain a fused registration image; and concatenates the fused image features with the registration image features and adds them to the fixed image to obtain a fused fixed image.

[0014] The detection module is trained based on the fused registration image, the fused fixed image, and the fused image.

[0015] Optionally, the feature extraction module employs a channel attention mechanism, which assigns different extraction weights to different positions of the fixed image, the registered image, and the fused image to obtain fixed image features, registered image features, and fused image features.

[0016] Optionally, the defect detection method includes:

[0017] The multimodal detection dataset is subjected to data augmentation processing, which includes at least one of the following:

[0018] The fixed image, the registered image, and the fused image are reversed and translated at the same angle;

[0019] The same grayscale value change process is applied to the fixed image, the registered image, and the fused image;

[0020] The same noise processing is applied to the fixed image, the registered image, and the fused image.

[0021] Optionally, training the multimodal target detection network using the multimodal detection dataset includes:

[0022] Obtain the multimodal detection dataset containing at least two types of label data, and obtain the training set and test set through the multimodal detection dataset;

[0023] Based on the multimodal detection dataset, the multimodal target detection network is trained using mAP as the evaluation metric to obtain the optimal detection weights;

[0024] The multimodal target detection network is configured according to the optimal detection weights to obtain the trained multimodal target detection network.

[0025] Optionally, the feature extraction module employs a parallel feature extraction network comprising three sub-networks, wherein the three sub-networks extract image features from the fixed image, the registered image, and the fused image respectively and simultaneously.

[0026] The feature fusion module employs a cross-modal feature fusion network embedded in the parallel feature extraction network; and

[0027] The feature extraction module extracts images from the fused registered image, the fused fixed image, and the fused image after being fused by the feature fusion module.

[0028] Optionally, the image pairs of the fixed image and the floating image are composed of any one of the following:

[0029] Neutron images and X-ray images;

[0030] Infrared images and RGB images.

[0031] Optionally, the fixed image is an X-ray image, and the floating image is a neutron image; and

[0032] The training of the multimodal object detection network using the multimodal detection dataset includes:

[0033] Defect markers are expressed using the label data on the registered image and the fused image, and structural defect markers are expressed using the label data on the fixed image, so that the defect markers and the defect markers are displayed on the output structure.

[0034] Specifically, the present invention also provides a defect detection device based on deep learning and multimodal images, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the defect detection method described above is implemented.

[0035] Specifically, the present invention also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement any of the defect detection methods described above.

[0036] In the deep learning-based defect detection method of this invention, multiple images—including fixed images, registered images, and fused images—are combined as samples to form a multimodal detection dataset, which is then input into a multimodal object detection network. The detection weights of the multimodal object detection network are trained and updated using mPA as the evaluation metric. Training the multimodal object detection network with multiple images as samples improves the analytical accuracy of defect detection.

[0037] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0038] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0039] Figure 1 This is a flowchart of a defect detection method according to an embodiment of the present invention;

[0040] Figure 2 This is a flowchart of a feature fusion module of a multimodal target detection network according to an embodiment of the present invention;

[0041] Figure 3 This is a flowchart of a multimodal target detection network according to an embodiment of the present invention. Detailed Implementation

[0042] The following reference Figures 1 to 3This invention describes a deep learning-based defect detection method and related apparatus according to embodiments of the present invention. In this description, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of that feature, that is, include one or more of that feature. In the description of the present invention, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. When a feature "includes or contains" one or more of the features it encompasses, unless otherwise specifically stated, this indicates that other features are not excluded and may be further included.

[0043] Unless otherwise expressly specified and limited, the terms "set up," "install," "connect," "link," "fix," and "couple" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art should be able to understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0044] Furthermore, in the description of this embodiment, "above" or "below" the second feature can include direct contact between the first and second features, or it can include contact between the first and second features through another feature between them. That is, in the description of this embodiment, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," or "below" of the second feature can mean the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0045] In the description of this embodiment, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0046] Figure 1 This is a flowchart of a defect detection method according to an embodiment of the present invention, such as... Figure 1 As shown, and refer to Figure 3 This invention provides a defect detection method based on deep learning and multimodal images, comprising:

[0047] The system acquires a fixed image, a floating image, a registered image after registration with the fixed image, and a fused image after fusing the registered and fixed images. Furthermore, it establishes a multimodal detection dataset composed of the fixed image, the registered image, and the fused image.

[0048] A multimodal object detection network is constructed and trained using a multimodal detection dataset. The multimodal object detection network includes a feature extraction module, a feature fusion module, and a detection module.

[0049] The multimodal detection dataset is input into the trained multimodal target detection network, and the output results are used as the defect detection results.

[0050] Specifically, the fixed image is an X-ray image, and the floating image is a neutron image. The registration image is the image obtained by registering the neutron image with the X-ray image as a reference.

[0051] Multiple images—fixed, registered, and fused—are used as samples to form a multimodal detection dataset, which is then input into a multimodal object detection network for training. The detection weights of the multimodal object detection network are trained and updated using mPA as the evaluation metric. Compared to previous single-modal object detection networks, the multimodal object detection network of this invention introduces three modalities for defect detection: X-Ray images and registered neutron images as the primary modality, and the fused image of the X-Ray and registered neutron images as the complementary modality. This results in more reliable and accurate detection, improving the defect detection rate and reducing the false negative rate.

[0052] In some embodiments of the present invention, the defect detection method further includes configuring a multimodal detection environment before acquiring an image.

[0053] In some embodiments of the present invention, the step of obtaining a fixed image, a floating image, and a registered image after registration includes registering the fixed image and the floating image to obtain a registered image, aligning the registered image and the fixed image spatially, and replacing the original floating image with the registered image. This setup facilitates the extraction of image features from the fixed image, the registered image, and the fused image. The step of registering the fixed image and the floating image further includes:

[0054] Data acquisition: Illuminate the same object with different imaging devices and at different imaging times to acquire initial images, namely the original fixed image and the original floating image;

[0055] Data preprocessing: The images are denoised. The processed original floating image and the original fixed image are then preliminarily affine aligned using an azimuthal transformation network. Afterwards, the pixels of the processed image groups are normalized. The normalized original fixed image is transformed into a fixed image (X), and the original floating image is transformed into a floating image (N).

[0056] Flow field calculation: X and N are concatenated on the channel as the two inputs of the network. The concatenated Tensor is input into the network. The image similarity between X and N is used as the metric. The network weight information is trained and updated through optimization algorithms. The flow field and transformation parameter T between X and N are calculated through operations such as convolution, downsampling and upsampling.

[0057] Spatial transformation: The flow field and N are used as inputs to the spatial transformation network. The transformation parameter T is applied to the floating image N through the interpolation function to obtain the final registration result R.

[0058] Backpropagation: After completing one registration operation, the image similarity and the regularization value of the flow field are used as the loss function to update the weight parameters and backpropagate. Then, the above steps are repeated until the network converges.

[0059] In some embodiments of the present invention, the defect detection method further includes:

[0060] Data augmentation processing is performed on the multimodal detection dataset, including at least one of the following: applying the same angle of inversion and translation to the fixed image, registered image, and fused image; applying the same grayscale value change processing to the fixed image, registered image, and fused image; or adding the same noise processing to the fixed image, registered image, and fused image.

[0061] Preferably, the data enhancement processing includes: performing the same angle of inversion and translation processing on the fixed image, the registered image, and the fused image; performing the same grayscale value change processing on the fixed image, the registered image, and the fused image; and adding the same noise processing to the fixed image, the registered image, and the fused image.

[0062] Specifically, after establishing a multimodal detection dataset consisting of a fixed image, a registered image, and a fused image, data augmentation processing is performed on the multimodal dataset. During data augmentation, the fixed image, registered image, and fused image are first subjected to the same angle of inversion and translation. Then, the same grayscale value changes are performed on the fixed image, registered image, and fused image. Finally, the same noise is added to the fixed image, registered image, and fused image. This setup makes the fixed image, registered image, and fused image easier to recognize.

[0063] In some embodiments of the present invention, the image pair of the fixed image and the floating image is composed of any one of the following: a neutron image and an X-ray image; an infrared image and an RGB image. Preferably, the image pair of the fixed image and the floating image is composed of a neutron image and an X-ray image.

[0064] In some embodiments of the present invention, the step of training a multimodal target detection network using a multimodal detection dataset includes: representing defect markers with label data on registered and fused images, and representing structural defect markers with label data on a fixed image, so that defect markers and structural defects are displayed on the output structure. Specifically, the label data representing defect markers is 0, and the label data representing structural defects is 1. That is, when labeling data on the fixed image, registered image, and fused image, the graphic markers that appear as defects in the registered and fused images are labeled as 0, and the graphic markers that appear as structural defects in the X-ray image are labeled as 1.

[0065] In some embodiments of the present invention, the step of training the multimodal object detection network using a multimodal detection dataset further includes: obtaining a multimodal detection dataset containing at least two classes of labeled data, and obtaining a training set and a test set from the multimodal detection dataset. Preferably, the multimodal detection dataset obtained includes data labels containing labels expressing defective features and data labels containing labels expressing structural defects.

[0066] like Figure 2 As shown, and refer to Figure 3 In some embodiments of the present invention, the step of training the multimodal target detection network using a multimodal detection dataset further includes: the feature extraction module extracting image features from the fixed image, the registered image, and the fused image, respectively, and denoting them as fixed image features, registered image features, and fused image features.

[0067] The feature fusion module concatenates the features of the fused image with those of the fixed image and adds them to the registered image to obtain a fused registered image. It then concatenates the features of the fused image with those of the registered image and adds them to the fixed image to obtain a fused fixed image. The detection module is trained using the fused registered image, the fused fixed image, and the fused image. Specifically, the fixed image, the registered image, and the fused image from the multimodal data detection set are input into the feature fusion module. The resulting fused fixed image, fused registered image, and fused image are then used as the training set to train the multimodal target detection network. By fusing and concatenating the same features from the fixed image, the registered image, and the fused image, the image features in these images are enhanced and improved, yielding more reliable information. This improves the detection accuracy of defects in industrial parts under complex environments, avoids false detections, and reduces the false detection rate.

[0068] like Figure 2 As shown, in some embodiments of the present invention, the feature extraction module adopts a channel attention mechanism, which assigns different extraction weights to different positions of the fixed image, the registered image, and the fused image to obtain fixed image features, registered image features, and fused image features.

[0069] Specifically, the feature fusion module uses the fixed image and the registered image as the primary modality, and the fused image as the supplementary modality. The channel attention mechanism assigns different weights to different locations in the image by compressing and activating the features of the fixed image, the registered image, and the fused image.

[0070] Among them, the fixed image is F x The registered image is F n The merged image is F f ; Fixed image features are F xd The registered image features are F nd The image features are fused into F fd ; Fuse fixed images into F x ′, the fused and registered image is F n The fused image after processing by the feature fusion module is F. f The working process of the feature fusion module can be described as follows:

[0071]

[0072]

[0073] F f '=se(F f )

[0074] like Figure 3 As shown, in some embodiments of the present invention,

[0075] The feature extraction module employs a parallel feature extraction network comprising three sub-networks, which extract image features from the fixed image, registered image, and fused image simultaneously and separately. The feature fusion module uses a cross-modal feature fusion network embedded within the parallel feature extraction network. Furthermore, the feature extraction module extracts image features from the fused registered image, fused fixed image, and fused image after fusion by the feature fusion module. Specifically, the parallel feature extraction network is designed based on the feature extraction network in the electrostatic target detection network, and is an extension of the feature extraction network. After the cross-modal feature fusion network is embedded in the parallel feature extraction network, the multimodal target detection network remains unchanged.

[0076] In some embodiments of the present invention, a multimodal object detection network is trained using mAP as an evaluation metric based on a multimodal detection dataset to obtain optimal detection weights. Specifically, mAP stands for Mean Accuracy.

[0077] Configure the multimodal object detection network according to the optimal detection weights to obtain the trained multimodal object detection network.

[0078] Specifically, the multimodal object detection network is designed based on the YOLOv5 detection architecture. The fused fixed image, fused registered image, and fused image from the training set are used as input. A parallel feature extraction network extracts image features from these three images. Then, a cross-modal feature fusion module concatenates the fused image features with the fused fixed image features and adds them to the fused registered image to obtain a new fused registered image. The concatenation of these features with the fused registered image features is then added to the fused fixed image to obtain a new fused fixed image. Next, the parallel feature extraction network extracts image features from the new fused fixed image, the new fused registered image, and the fused image. Simultaneously, the new fused fixed image, the new fused registered image, and the fused image are concatenated and fused to obtain a composite image, which is then included in the test set. The test image is input into the detection module, using mAP as the evaluation metric. The detection module compares the composite image with the mAP.

[0079] The above operation is repeated at least three times to train the multimodal object detection network and obtain the optimal detection weights. Preferably, the above operation is repeated three times to obtain three composite images. The three detection images are included in the test set input detection module. The detection module outputs the scores obtained by comparing the three composite images with mAP. The composite image with the highest score is regarded as the optimal detection weight and the optimal detection weight is saved.

[0080] In some embodiments of the present invention, the step of inputting the multimodal detection dataset into the trained multimodal target detection network and using the output result as the defect detection result further includes:

[0081] The file containing the optimal detection weights is downloaded into the trained multimodal object detection network. X-ray images of the industrial part, neutron images registered with the X-ray images, and the fused image of the two are acquired. These three images are then input into the multimodal object detection network to detect defects in the industrial part and determine whether defects exist. The detection results are divided into two parts: defects and structural defects.

[0082] In some embodiments of the present invention, the multimodal object detection network, both during and after training, utilizes the overall loss function of a multimodal defect detection algorithm to detect images of industrial parts. The overall loss function of the multimodal defect detection algorithm is defined as follows:

[0083] L total =L cls +L box +L conf

[0084] Among them, L cls For classification loss, L box For location regression loss, L conf This represents the confidence loss.

[0085] The classification loss function is defined as shown in the formula:

[0086]

[0087] Where, p i (c) represents the probability that the true sample is C. This represents the probability that the network predicts the sample to be C.

[0088] The location regression loss function is defined as shown in the formula:

[0089]

[0090] The confidence loss function is defined as shown in the formula:

[0091]

[0092] The confidence loss uses the squared error loss, c i and This represents the true confidence level and the confidence level of the network prediction.

[0093] Embodiments of the present invention also provide a defect detection device based on deep learning and multimodal images, which includes a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the defect detection method in any of the above embodiments is implemented.

[0094] This invention also provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the defect detection method in any of the above embodiments.

[0095] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A defect detection method based on deep learning and multimodal images, characterized in that, include: A fixed image, a floating image, a registered image after registration with the fixed image, and a fused image after merging the registered image and the fixed image are obtained; and a multimodal detection dataset composed of the fixed image, the registered image, and the fused image is established. A multimodal object detection network is constructed and trained using the multimodal detection dataset. The multimodal object detection network includes a feature extraction module, a feature fusion module, and a detection module. The multimodal detection dataset is input into the trained multimodal target detection network, and the output result is used as the defect detection result. The step of training the multimodal target detection network using the multimodal detection dataset includes: The feature extraction module extracts image features from the fixed image, the registered image, and the fused image, respectively, and denoted as fixed image features, registered image features, and fused image features. The feature fusion module concatenates the fused image features with the fixed image features and adds them to the registration image to obtain a fused registration image; and concatenates the fused image features with the registration image features and adds them to the fixed image to obtain a fused fixed image. The detection module is trained based on the fused registration image, the fused fixed image, and the fused image; The feature extraction module employs a parallel feature extraction network comprising three sub-networks, which extract image features from the fixed image, the registered image, and the fused image, respectively and simultaneously. The feature fusion module employs a cross-modal feature fusion network embedded in the parallel feature extraction network; and The feature extraction module extracts images from the fused registered image, the fused fixed image, and the fused image after being fused by the feature fusion module.

2. The defect detection method according to claim 1, characterized in that, The feature extraction module employs a channel attention mechanism, which assigns different extraction weights to different positions of the fixed image, the registered image, and the fused image to obtain fixed image features, registered image features, and fused image features.

3. The defect detection method according to claim 1, characterized in that, The defect detection method further includes: The multimodal detection dataset is subjected to data augmentation processing, which includes at least one of the following: The fixed image, the registered image, and the fused image are reversed and translated at the same angle; The same grayscale value change process is applied to the fixed image, the registered image, and the fused image; The same noise processing is applied to the fixed image, the registered image, and the fused image.

4. The defect detection method according to claim 1, characterized in that, The training of the multimodal object detection network using the multimodal detection dataset includes: Obtain the multimodal detection dataset containing at least two types of label data, and obtain the training set and test set through the multimodal detection dataset; Based on the multimodal detection dataset, the multimodal target detection network is trained using mAP as the evaluation metric to obtain the optimal detection weights; The multimodal target detection network is configured according to the optimal detection weights to obtain the trained multimodal target detection network.

5. The defect detection method according to claim 1, characterized in that, The image pairs of the fixed image and the floating image are formed into any one of the following groups: Neutron images and X-ray images; Infrared images and RGB images.

6. The defect detection method according to claim 4, characterized in that, The fixed image is an X-ray image, and the floating image is a neutron image; as well as The training of the multimodal object detection network using the multimodal detection dataset includes: Defect markers are expressed using the label data on the registered image and the fused image, and structural defect markers are expressed using the label data on the fixed image, so that the defect markers and the structural defect markers are displayed on the output result.

7. A defect detection device based on deep learning and multimodal images, characterized in that, It includes a processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement the defect detection method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the defect detection method as described in any one of claims 1-6.

Images