Non-destructive testing method and system based on multi-modal images

Abstract

The application provides a kind of nondestructive testing method and system based on multi-modal image, comprising: synchronously collecting object surface image and ray detection image, surface image is composed of surface optical image and depth image;Surface image and ray detection image are registered;Surface optical image and depth image are identified using surface image defect automatic identification method, image with defect feature is selected out and defect position is framed out;Ray detection image is identified using ray image defect identification model, image with defect feature is selected out and defect position is framed out;The position of ray image defect identification positioning result and surface image feature identification result is compared, to determine whether defect feature belongs to internal defect.The application can simultaneously collect object ray detection image and surface image, so as to effectively distinguish object internal structure information and surface structure information, greatly improve defect identification algorithm identification accuracy.

Description

Technical Field

[0001] This invention relates to the technical field of nondestructive testing, and more specifically, to a nondestructive testing method and system based on multimodal images. Background Technology

[0002] X-ray inspection is a non-destructive testing technique that utilizes the attenuation properties of X-rays within an object and their interaction with the imaging medium to detect internal defects. X-ray inspection offers advantages such as intuitive results and high reliability, and is widely used in the internal quality inspection of key products in fields such as aviation, aerospace, and special equipment. Traditional X-ray inspection uses film for imaging, and the results are recorded as simulated images on film. With the development of electronic information technology, digital X-ray inspection technology, which uses radiation detectors for imaging, has become widely used. Simultaneously, with the development of artificial intelligence technology, research on digital X-ray inspection image defect recognition technology based on automatic image recognition algorithms is extensive both domestically and internationally. Traditional defect recognition algorithms only identify internal defects based on X-ray images. Since a X-ray image is a two-dimensional projection of the object along the ray direction onto the detector plane, it simultaneously contains both surface and internal structural information. Therefore, when using recognition algorithms to identify internal defects, the recognition results are easily affected by surface structural features.

[0003] Radiographic inspection is a primary method for inspecting the internal quality of welds. Since a radiographic image is a two-dimensional superimposed projection of the inspected object along the ray direction, a weld radiographic image contains all the surface and internal morphological features of the weld. Current research on defect recognition using radiographic images, both domestically and internationally, only trains on the radiographic images for recognition, failing to distinguish between surface features and internal defects. This leads to misjudgments, reduces defect recognition accuracy, and thus restricts engineering applications.

[0004] Therefore, a new technical solution is needed to improve the above-mentioned technical problems. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a non-destructive testing method and system based on multimodal images.

[0006] According to the present invention, a non-destructive testing method based on multimodal images is provided, the method comprising the following steps:

[0007] Step S1: Simultaneously acquire images of the object surface and ray detection images;

[0008] Step S2: Register the surface image and the X-ray inspection image;

[0009] Step S3: Use the surface image defect automatic recognition method to identify the surface optical image and depth image, select the images with defect features and select the defect location;

[0010] Step S4: Use the X-ray image defect recognition model to identify the X-ray inspection images, select images with defect features, and outline the defect locations;

[0011] Step S5: Compare the location results of the X-ray image defect identification and the surface image feature identification to determine whether the defect features belong to internal defects.

[0012] Preferably, in step S1:

[0013] The synchronous acquisition refers to the process of taking images of the X-ray incident and exit surfaces of an object at any location when it is irradiated with X-rays and imaged with a radiation detector. The surface area captured by the camera should be inclusive of the X-ray incident and exit surfaces, and the effective detection area is determined based on the smaller area of ​​the image.

[0014] Preferably, in step S2:

[0015] The ray image, surface depth image, and optical image are annotated with detail alignment. Then, the designed registration network is input to calculate the optical flow maps of the surface depth image and the optical image respectively. The optical flow map represents the offset between the pixel position of the corresponding depth image and optical image and the pixel position of the ray image. The surface depth image and optical image matrix are added to the optical flow map matrix to obtain the registered surface depth image and optical image.

[0016] Preferably, step S3 includes the following steps:

[0017] Step S3.1: Use the sliding window method to crop the surface optical image and depth image. The size of the sliding window is W*W, and the step size of the sliding window is S.

[0018] Step S3.2: Input the cropped surface optical image and depth image into the feature extraction module respectively to extract image features; the feature extraction module consists of a convolutional neural network, and the optical feature map output by the feature extraction module is represented as follows. Where h1, w1, and c1 represent the height, width, and number of channels of the optical feature map, respectively; the depth feature map output by the feature extraction module is represented as... Where h2, w2, and c2 represent the height, width, and number of channels of the depth feature map, respectively;

[0019] Step S3.3: Input the cropped surface optical image and depth image into the feature importance calculation module respectively to calculate the weights of the extracted features; the optical feature weights output by the feature importance model are expressed as follows: The deep feature weights output by the feature importance model are represented as follows:

[0020] Step S3.4: Transfer the optical feature map F g With optical feature weight W g Multiply to obtain the weighted optical feature map. This feature Figure 1 There are a total of h1×w1 pixels, and each pixel corresponds to a surface optical image descriptor. Ultimately, all surface optical image descriptors are composed of a set. Indicate; the depth feature map F s With depth feature weights W s Multiply to obtain the weighted depth feature map. This feature Figure 1 There are a total of h2×w2 pixels, and each pixel corresponds to a depth image descriptor. Ultimately, all depth optical image descriptors are composed of a set. express;

[0021] Step S3.5: Input the surface optical image bounding box regions from the training set into the feature extraction module trained on the large dataset to obtain the initial feature map. Each feature... Figure 1 There are a total of h1×w1 pixels, each corresponding to an initial feature descriptor of the surface optical image. All initial feature descriptors of the surface optical image are clustered into N1 clusters. The cluster center of each cluster serves as a sub-feature in sub-feature group 1. The initial feature descriptor subset of the k-th cluster is represented as: Calculate the distance between the initial feature descriptors in this set, a p With a q The distance is calculated as follows:

[0022]

[0023] Among them, [a p ] d Characteristic descriptor a p The d-th dimension of the data;

[0024] Sub-feature C 1k Characteristic coefficients φ k The initial value is calculated as follows:

[0025]

[0026] Computational surface optical image descriptor With sub-feature C 1k The distance is calculated using the following formula:

[0027]

[0028] Surface optical image descriptor Normalize the distances to the N1 sub-features:

[0029]

[0030] Calculate the normalized distance between all surface optical image descriptors and all sub-features; the sum of the normalized distances between all surface optical image descriptors and the k-th sub-feature is used as the output e of the k-th encoder in encoder 1. k ,Right now:

[0031]

[0032] The final output of encoder 1 is

[0033] Step S3.6: Input the depth image bounding box regions from the training set into the feature extraction module trained on the large dataset to obtain depth image feature descriptors; cluster all initial depth feature descriptors to obtain N2 clusters, resulting in N2 sub-features, forming sub-feature group 2; calculate the normalized distance between all depth image descriptors and all sub-features in sub-feature group 2; the sum of the normalized distances between all depth image descriptors and the l-th sub-feature is used as the output of the l-th encoder in encoder 2. l The final output of encoder 2 is

[0034] Step S3.7: Connect the outputs of encoder 1 and encoder 2 to obtain... Input E into the classifier, and the output will be the defect category in the image;

[0035] Step S3.8: The final automatic surface image defect recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The automatic surface image defect recognition network is trained until convergence. The optical image and depth image of the surface to be detected are cropped using a sliding window and then input into the automatic surface image defect recognition network to achieve defect recognition. If the defect category is positive, the corresponding sliding window position is the surface defect position.

[0036] Preferably, step S4 includes the following steps:

[0037] Step S4.1: Use the sliding window method to crop the depth image, input the cropped ray image into the feature extraction module to extract image features; input the cropped ray image into the feature importance calculation module to calculate the weight of the extracted features; multiply the ray feature map with the ray feature weights to obtain the weighted ray feature map, where each pixel of the ray feature map corresponds to a ray image descriptor;

[0038] Step S4.2: Input the ray image bounding box regions from the training set into the feature extraction module trained on the large dataset to obtain initial feature maps. Each pixel in the feature map corresponds to an initial feature descriptor of the ray image. Cluster all the initial feature descriptors of the ray images, resulting in N3 clusters and N3 sub-features, forming sub-feature group 3. Calculate the normalized distance between all ray image descriptors and all sub-features in sub-feature group 3. The sum of the normalized distances between all depth image descriptors and the r-th sub-feature is used as the output t of the r-th encoder in encoder 3. l The final output of encoder 3 is Input E3 into the classifier and output the defect category in the ray image; if the defect category is positive, the corresponding sliding window position is the ray defect position;

[0039] Step S4.3: The final X-ray image defect automatic recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The X-ray image defect automatic recognition network is trained until convergence. The X-ray image to be detected is cropped using a sliding window and then input into the X-ray image defect automatic recognition network to achieve defect recognition and localization.

[0040] The present invention also provides a non-destructive testing system based on multimodal images, the system comprising the following modules:

[0041] Module M1: Simultaneously acquires images of the object surface and ray detection images;

[0042] Module M2: Registers surface images and X-ray inspection images;

[0043] Module M3: Employs an automatic surface image defect recognition system to identify surface optical and depth images, select images with defect features, and outline the defect locations.

[0044] Module M4: Uses a X-ray image defect recognition model to identify X-ray inspection images, select images with defect features, and outline the defect locations;

[0045] Module M5: Compares the location results of defect identification in X-ray images with the location results of feature identification in surface images to determine whether the defect features belong to internal defects.

[0046] Preferably, in module M1:

[0047] The synchronous acquisition refers to the process of taking images of the X-ray incident and exit surfaces of an object at any location when it is irradiated with X-rays and imaged with a radiation detector. The surface area captured by the camera should be inclusive of the X-ray incident and exit surfaces, and the effective detection area is determined based on the smaller area of ​​the image.

[0048] Preferably, in module M2:

[0049] The ray image, surface depth image, and optical image are annotated with detail alignment. Then, the designed registration network is input to calculate the optical flow maps of the surface depth image and the optical image respectively. The optical flow map represents the offset between the pixel position of the corresponding depth image and optical image and the pixel position of the ray image. The surface depth image and optical image matrix are added to the optical flow map matrix to obtain the registered surface depth image and optical image.

[0050] Preferably, module M3 includes the following modules:

[0051] Module M3.1: Uses the sliding window method to crop surface optical images and depth images. The size of the sliding window is W*W, and the step size of the sliding window is S.

[0052] Module M3.2: Inputs the cropped surface optical image and depth image into the feature extraction module to extract image features; the feature extraction module consists of a convolutional neural network, and the optical feature map output by the feature extraction module is represented as follows. Where h1, w1, and c1 represent the height, width, and number of channels of the optical feature map, respectively; the depth feature map output by the feature extraction module is represented as... Where h2, w2, and c2 represent the height, width, and number of channels of the depth feature map, respectively;

[0053] Module M3.3: Input the cropped surface optical image and depth image into the feature importance calculation module to calculate the weights of the extracted features; the optical feature weights output by the feature importance model are expressed as follows: The deep feature weights output by the feature importance model are represented as follows:

[0054] Module M3.4: Transfers optical feature map F g With optical feature weight W g Multiply to obtain the weighted optical feature map. This feature Figure 1 There are a total of h1×w1 pixels, and each pixel corresponds to a surface optical image descriptor. Ultimately, all surface optical image descriptors are composed of a set. Indicate; the depth feature map F s With depth feature weights W s Multiply to obtain the weighted depth feature map. This feature Figure 1 There are a total of h2×w2 pixels, and each pixel corresponds to a depth image descriptor. Ultimately, all depth optical image descriptors are composed of a set. express;

[0055] Module M3.5: Inputs the surface optical image bounding box regions from the training set into the feature extraction module trained on a large dataset to obtain the initial feature map. Each feature... Figure 1 There are a total of h1×w1 pixels, each corresponding to an initial feature descriptor of the surface optical image. All initial feature descriptors of the surface optical image are clustered into N1 clusters. The cluster center of each cluster serves as a sub-feature in sub-feature group 1. The initial feature descriptor subset of the k-th cluster is represented as: Calculate the distance q between the initial feature descriptors in this set. p With a q The distance is calculated as follows:

[0056]

[0057] Among them, [a p ] d Characteristic descriptor a p The d-th dimension of the data;

[0058] Sub-feature C 1k Characteristic coefficients φ k The initial value is calculated as follows:

[0059]

[0060] Computational surface optical image descriptor With sub-feature C 1k The distance is calculated using the following formula:

[0061]

[0062] Surface optical image descriptor Normalize the distances to the N1 sub-features:

[0063]

[0064] Calculate the normalized distance between all surface optical image descriptors and all sub-features; the sum of the normalized distances between all surface optical image descriptors and the k-th sub-feature is used as the output e of the k-th encoder in encoder 1. k ,Right now:

[0065]

[0066] The final output of encoder 1 is

[0067] Module M3.6: Inputs the depth image bounding box regions from the training set into the feature extraction module trained on a large dataset to obtain depth image feature descriptors; clusters all initial depth feature descriptors into N2 clusters, resulting in N2 sub-features, forming sub-feature group 2; calculates the normalized distance between all depth image descriptors and all sub-features in sub-feature group 2; the sum of the normalized distances between all depth image descriptors and the l-th sub-feature is used as the output of the l-th encoder in encoder 2. l The final output of encoder 2 is

[0068] Module M3.7: Connects the outputs of encoder 1 and encoder 2 to obtain... Input E into the classifier, and the output will be the defect category in the image;

[0069] Module M3.8: The final automatic surface image defect recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The automatic surface image defect recognition network is trained until convergence. The optical image and depth image of the surface to be detected are cropped using a sliding window and then input into the automatic surface image defect recognition network to achieve defect recognition. If the defect category is positive, the corresponding sliding window position is the surface defect position.

[0070] Preferably, module M4 includes the following modules:

[0071] Module M4.1: The depth image is cropped using the sliding window method. The cropped ray image is then input into the feature extraction module to extract image features. The cropped ray image is then input into the feature importance calculation module to calculate the weights of the extracted features. The ray feature map is multiplied by the ray feature weights to obtain a weighted ray feature map. Each pixel in the ray feature map corresponds to a ray image descriptor.

[0072] Module M4.2: Inputs the ray image bounding box regions from the training set into the feature extraction module trained on a large dataset to obtain initial feature maps. Each pixel in the feature map corresponds to an initial feature descriptor of the ray image. Clustering all the initial feature descriptors of the ray images results in N3 clusters, yielding N3 sub-features, forming sub-feature group 3. Calculate the normalized distance between all ray image descriptors and all sub-features in sub-feature group 3. The sum of the normalized distances between all depth image descriptors and the r-th sub-feature is used as the output t of the r-th encoder in encoder 3. l The final output of encoder 3 is Input E3 into the classifier and output the defect category in the ray image; if the defect category is positive, the corresponding sliding window position is the ray defect position;

[0073] Module M4.3: The final X-ray image defect automatic recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The X-ray image defect automatic recognition network is trained until convergence. The X-ray image to be detected is cropped using a sliding window and then input into the X-ray image defect automatic recognition network to achieve defect recognition and localization.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. This invention can enhance the criteria for defect identification algorithms. By recognizing both ray images and surface images, it solves the problem of indistinguishable surface and internal features in a single ray image, thereby significantly improving the accuracy of internal defect identification.

[0076] 2. This invention simultaneously acquires surface images and ray detection images of an object, and uses an automatic image recognition algorithm to perform feature recognition on both surface and ray images. Finally, it comprehensively interprets the recognition results of the two images, which can effectively eliminate the influence of surface features on the identification of internal defects and significantly reduce the misjudgment rate of internal defect identification.

[0077] 3. This invention can simultaneously acquire X-ray inspection images and surface images of an object, thereby effectively distinguishing between internal structural information and surface structural information of the object, and greatly improving the accuracy of defect recognition algorithms. Attached Figure Description

[0078] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0079] Figure 1 This is a schematic diagram of the hardware composition of a non-destructive testing system based on multimodal images according to the present invention.

[0080] Figure 2This is a diagram showing the module composition of the automatic surface image defect identification method in the non-destructive testing device and method based on multimodal images of the present invention;

[0081] Figure 3 This is a diagram showing the module composition of the automatic X-ray image defect identification method in the non-destructive testing device and method based on multimodal images of the present invention;

[0082] Figure 4 This is a flowchart illustrating the principle of the present invention. Detailed Implementation

[0083] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0084] Example 1:

[0085] Reference Figure 4 According to the present invention, a non-destructive testing method based on multimodal images is provided, the method comprising the following steps:

[0086] Step S1: Simultaneously acquire surface images and X-ray detection images of the object; the simultaneous acquisition refers to taking surface images of the X-ray incident surface and the exit surface of the object at any location when it is irradiated with X-rays and imaged with a radiation detector. The surface area captured by the camera should have an inclusion relationship with the X-ray incident surface and the exit surface, and the effective detection area is determined based on the smaller area of ​​the image.

[0087] Step S2: Register the surface image and the ray detection image; perform detail alignment annotation on the ray image, surface depth image and optical image, and then input the designed registration network to calculate the optical flow map of the surface depth image and the optical image respectively. The optical flow map represents the offset between the pixel position of the corresponding depth image and optical image and the pixel position of the ray image. Add the optical flow map matrix to the surface depth image and optical image matrix to obtain the registered surface depth image and optical image.

[0088] Step S3: Use the surface image defect automatic recognition method to identify the surface optical image and depth image, select the images with defect features and select the defect location;

[0089] Step S3.1: Use the sliding window method to crop the surface optical image and depth image. The size of the sliding window is W*W, and the step size of the sliding window is S.

[0090] Step S3.2: Input the cropped surface optical image and depth image into the feature extraction module respectively to extract image features; the feature extraction module consists of a convolutional neural network, and the optical feature map output by the feature extraction module is represented as follows. Where h1, w1, and c1 represent the height, width, and number of channels of the optical feature map, respectively; the depth feature map output by the feature extraction module is represented as... Where h2, w2, and c2 represent the height, width, and number of channels of the depth feature map, respectively;

[0091] Step S3.3: Input the cropped surface optical image and depth image into the feature importance calculation module respectively to calculate the weights of the extracted features; the optical feature weights output by the feature importance model are expressed as follows: The deep feature weights output by the feature importance model are represented as follows:

[0092] Step S3.4: Transfer the optical feature map F g With optical feature weight W g Multiply to obtain the weighted optical feature map. This feature Figure 1 There are a total of h1×w1 pixels, and each pixel corresponds to a surface optical image descriptor. Ultimately, all surface optical image descriptors are composed of a set. Indicate; the depth feature map F s With depth feature weights W s Multiply to obtain the weighted depth feature map. This feature Figure 1 There are a total of h2×w2 pixels, and each pixel corresponds to a depth image descriptor. Ultimately, all depth optical image descriptors are composed of a set. express;

[0093] Step S3.5: Input the surface optical image bounding box regions from the training set into the feature extraction module trained on the large dataset to obtain the initial feature map. Each feature... Figure 1 There are a total of h1×w1 pixels, each corresponding to an initial feature descriptor of the surface optical image. All initial feature descriptors of the surface optical image are clustered into N1 clusters. The cluster center of each cluster serves as a sub-feature in sub-feature group 1. The initial feature descriptor subset of the k-th cluster is represented as: Calculate the distance between the initial feature descriptors in this set, a p With a q The distance is calculated as follows:

[0094]

[0095] Among them, [a p ] d Characteristic descriptor a p The d-th dimension of the data;

[0096] Sub-feature C 1k Characteristic coefficients φ k The initial value is calculated as follows:

[0097]

[0098] Computational surface optical image descriptor With sub-feature C 1k The distance is calculated using the following formula:

[0099]

[0100] Surface optical image descriptor Normalize the distances to the N1 sub-features:

[0101]

[0102] Calculate the normalized distance between all surface optical image descriptors and all sub-features; the sum of the normalized distances between all surface optical image descriptors and the k-th sub-feature is used as the output e of the k-th encoder in encoder 1. k ,Right now:

[0103]

[0104] The final output of encoder 1 is

[0105] Step S3.6: Input the depth image bounding box regions from the training set into the feature extraction module trained on the large dataset to obtain depth image feature descriptors; cluster all initial depth feature descriptors to obtain N2 clusters, resulting in N2 sub-features, forming sub-feature group 2; calculate the normalized distance between all depth image descriptors and all sub-features in sub-feature group 2; the sum of the normalized distances between all depth image descriptors and the l-th sub-feature is used as the output of the l-th encoder in encoder 2. ι The final output of encoder 2 is

[0106] Step S3.7: Connect the outputs of encoder 1 and encoder 2 to obtain... Input E into the classifier, and the output will be the defect category in the image;

[0107] Step S3.8: The final automatic surface image defect recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The automatic surface image defect recognition network is trained until convergence. The optical image and depth image of the surface to be detected are cropped using a sliding window and then input into the automatic surface image defect recognition network to achieve defect recognition. If the defect category is positive, the corresponding sliding window position is the surface defect position.

[0108] Step S4: Use the X-ray image defect recognition model to identify the X-ray inspection images, select images with defect features, and outline the defect locations;

[0109] Step S4.1: Use the sliding window method to crop the depth image, input the cropped ray image into the feature extraction module to extract image features; input the cropped ray image into the feature importance calculation module to calculate the weight of the extracted features; multiply the ray feature map with the ray feature weights to obtain the weighted ray feature map, where each pixel of the ray feature map corresponds to a ray image descriptor;

[0110] Step S4.2: Input the ray image bounding box regions from the training set into the feature extraction module trained on the large dataset to obtain initial feature maps. Each pixel in the feature map corresponds to an initial feature descriptor of the ray image. Cluster all the initial feature descriptors of the ray images, resulting in N3 clusters and N3 sub-features, forming sub-feature group 3. Calculate the normalized distance between all ray image descriptors and all sub-features in sub-feature group 3. The sum of the normalized distances between all depth image descriptors and the r-th sub-feature is used as the output t of the r-th encoder in encoder 3. l The final output of encoder 3 is Input E3 into the classifier and output the defect category in the ray image; if the defect category is positive, the corresponding sliding window position is the ray defect position;

[0111] Step S4.3: The final X-ray image defect automatic recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The X-ray image defect automatic recognition network is trained until convergence. The X-ray image to be detected is cropped using a sliding window and then input into the X-ray image defect automatic recognition network to achieve defect recognition and localization.

[0112] Step S5: Compare the location results of the X-ray image defect identification and the surface image feature identification to determine whether the defect features belong to internal defects.

[0113] The present invention also provides a nondestructive testing system based on multimodal images. The nondestructive testing system based on multimodal images can be implemented by executing the process steps of the nondestructive testing method based on multimodal images. That is, those skilled in the art can understand the nondestructive testing method based on multimodal images as a preferred embodiment of the nondestructive testing system based on multimodal images.

[0114] Example 2:

[0115] The present invention also provides a non-destructive testing system based on multimodal images, the system comprising the following modules:

[0116] Module M1: Synchronously acquires surface images and X-ray detection images of the object; the synchronous acquisition refers to taking images of the surface of the object being detected at any location when it is irradiated with X-rays and imaged with a radiation detector, with cameras taking images of the X-ray incident and exit surfaces respectively. The surface area captured by the camera should have an inclusion relationship with the X-ray incident and exit surfaces, and the effective detection area is determined based on the smaller area of ​​the image.

[0117] Module M2: Registers the surface image and the ray inspection image; performs detail alignment annotation on the ray image, surface depth image, and optical image; then inputs the designed registration network to calculate the optical flow maps of the surface depth image and the optical image respectively. The optical flow map represents the offset between the pixel positions of the corresponding depth image and optical image and the pixel positions of the ray image. Adding the optical flow map matrix to the surface depth image and optical image matrices gives the registered surface depth image and optical image.

[0118] Module M3: Employs an automatic surface image defect recognition system to identify surface optical and depth images, select images with defect features, and outline the defect locations.

[0119] Module M3.1: Uses the sliding window method to crop surface optical images and depth images. The size of the sliding window is W*W, and the step size of the sliding window is S.

[0120] Module M3.2: Inputs the cropped surface optical image and depth image into the feature extraction module to extract image features; the feature extraction module consists of a convolutional neural network, and the optical feature map output by the feature extraction module is represented as follows. Where h1, w1, and c1 represent the height, width, and number of channels of the optical feature map, respectively; the depth feature map output by the feature extraction module is represented as... Where h2, w2, and c2 represent the height, width, and number of channels of the depth feature map, respectively;

[0121] Module M3.3: Input the cropped surface optical image and depth image into the feature importance calculation module to calculate the weights of the extracted features; the optical feature weights output by the feature importance model are expressed as follows: The deep feature weights output by the feature importance model are represented as follows:

[0122] Module M3.4: Transfers optical feature map F g With optical feature weight W g Multiply to obtain the weighted optical feature map. This feature Figure 1 There are a total of h1×w1 pixels, and each pixel corresponds to a surface optical image descriptor. Ultimately, all surface optical image descriptors are composed of a set. Indicate; the depth feature map F s With depth feature weights W s Multiply to obtain the weighted depth feature map. This feature Figure 1 There are a total of h2×w2 pixels, and each pixel corresponds to a depth image descriptor. Ultimately, all depth optical image descriptors are composed of a set. express;

[0123] Module M3.5: Inputs the surface optical image bounding box regions from the training set into the feature extraction module trained on a large dataset to obtain the initial feature map. Each feature... Figure 1 There are a total of h1×w1 pixels, each corresponding to an initial feature descriptor of the surface optical image. All initial feature descriptors of the surface optical image are clustered into N1 clusters. The cluster center of each cluster serves as a sub-feature in sub-feature group 1. The initial feature descriptor subset of the k-th cluster is represented as: Calculate the distance between the initial feature descriptors in this set, a p With a q The distance is calculated as follows:

[0124]

[0125] Among them, [a p ] d Characteristic descriptor a p The d-th dimension of the data;

[0126] Sub-feature C 1k Characteristic coefficients φ k The initial value is calculated as follows:

[0127]

[0128] Computational surface optical image descriptor With sub-feature C 1k The distance is calculated using the following formula:

[0129]

[0130] Surface optical image descriptor Normalize the distances to the N1 sub-features:

[0131]

[0132] Calculate the normalized distance between all surface optical image descriptors and all sub-features; the sum of the normalized distances between all surface optical image descriptors and the k-th sub-feature is used as the output e of the k-th encoder in encoder 1. k ,Right now:

[0133]

[0134] The final output of encoder 1 is

[0135] Module M3.6: Inputs the depth image bounding box regions from the training set into the feature extraction module trained on a large dataset to obtain depth image feature descriptors; clusters all initial depth feature descriptors into N2 clusters, resulting in N2 sub-features, forming sub-feature group 2; calculates the normalized distance between all depth image descriptors and all sub-features in sub-feature group 2; the sum of the normalized distances between all depth image descriptors and the l-th sub-feature is used as the output of the l-th encoder in encoder 2. l The final output of encoder 2 is

[0136] Module M3.7: Connects the outputs of encoder 1 and encoder 2 to obtain... Input E into the classifier, and the output will be the defect category in the image;

[0137] Module M3.8: The final automatic surface image defect recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The automatic surface image defect recognition network is trained until convergence. The optical image and depth image of the surface to be detected are cropped using a sliding window and then input into the automatic surface image defect recognition network to achieve defect recognition. If the defect category is positive, the corresponding sliding window position is the surface defect position.

[0138] Module M4: Uses a X-ray image defect recognition model to identify X-ray inspection images, select images with defect features, and outline the defect locations;

[0139] Module M4.1: The depth image is cropped using the sliding window method. The cropped ray image is then input into the feature extraction module to extract image features. The cropped ray image is then input into the feature importance calculation module to calculate the weights of the extracted features. The ray feature map is multiplied by the ray feature weights to obtain a weighted ray feature map. Each pixel in the ray feature map corresponds to a ray image descriptor.

[0140] Module M4.2: Inputs the ray image bounding box regions from the training set into the feature extraction module trained on a large dataset to obtain initial feature maps. Each pixel in the feature map corresponds to an initial feature descriptor of the ray image. Clustering all the initial feature descriptors of the ray images results in N3 clusters, yielding N3 sub-features, forming sub-feature group 3. Calculate the normalized distance between all ray image descriptors and all sub-features in sub-feature group 3. The sum of the normalized distances between all depth image descriptors and the r-th sub-feature is used as the output t of the r-th encoder in encoder 3. l The final output of encoder 3 is Input E3 into the classifier and output the defect category in the ray image; if the defect category is positive, the corresponding sliding window position is the ray defect position;

[0141] Module M4.3: The final X-ray image defect automatic recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The X-ray image defect automatic recognition network is trained until convergence. The X-ray image to be detected is cropped using a sliding window and then input into the X-ray image defect automatic recognition network to achieve defect recognition and localization.

[0142] Module M5: Compares the location results of defect identification in X-ray images with the location results of feature identification in surface images to determine whether the defect features belong to internal defects.

[0143] Example 3:

[0144] The technical problem to be solved by this invention is to overcome the problem that existing X-ray inspection image defect recognition algorithms only use X-ray images as objects for defect detection, and the recognition results are easily affected by surface features and surface defects, resulting in low recognition accuracy.

[0145] To address the aforementioned technical problems, this invention proposes a non-destructive testing device and method based on multimodal images. This device can simultaneously acquire X-ray inspection images and surface images of an object, thereby effectively distinguishing between internal and surface structural information and significantly improving the accuracy of defect identification algorithms. The specific technical solution of this invention is as follows:

[0146] A non-destructive testing device and method based on multimodal images are disclosed. The hardware of the testing system consists of an X-ray machine, a radiation detector, a camera, a light source, a computer, and auxiliary equipment that is compatible with the above hardware. The software of the testing system is computer image recognition software with image defect recognition function.

[0147] A nondestructive testing device and method based on multimodal images, comprising the following steps:

[0148] (1) Simultaneously acquire object surface images and X-ray detection images;

[0149] (2) Register the surface image and the X-ray inspection image;

[0150] (3) The surface image defect automatic identification method is used to identify the surface optical image and depth image, select the image with defect features and select the defect location;

[0151] (4) Use the X-ray image defect recognition model to identify the X-ray inspection images, select images with defect features and outline the defect locations;

[0152] (5) Compare the location results of the X-ray image defect identification and the surface image feature identification to determine whether the defect features belong to internal defects.

[0153] Furthermore, a preferred method involves placing an X-ray machine and a radiation detector on opposite sides of the object being inspected during the detection process. Simultaneously, a camera and a light source are installed on the X-ray machine side and the radiation detector side, respectively. X-rays emitted by the X-ray machine pass through the object and illuminate the surface of the radiation detector, forming a X-ray image. The light source emits uniform visible light and structured light with spatial distribution characteristics. The light emitted by the light source is reflected from the object's surface and illuminates the camera, forming a surface image. The image acquired by the visible light emitted by the light source is a surface optical image, and the image acquired by the structured light emitted by the light source is a depth image.

[0154] Further, a preferred method is that the synchronous acquisition mentioned in step (1) refers to taking images of the X-ray incident surface and the exit surface of an object at any location of the object being detected by using a camera when it is irradiated with X-rays and imaged with a radiation detector. The surface area captured by the camera should have an inclusion relationship with the X-ray incident surface and the exit surface, and the effective detection area is determined based on the smaller area of ​​the image.

[0155] Furthermore, a preferred method is that the registration of the surface image and the ray detection image in step (2) is performed as follows:

[0156] The ray image, surface depth image, and optical image are annotated with detail alignment. Then, the designed registration network is input to calculate the optical flow maps of the surface depth image and the optical image, respectively. The optical flow map represents the offset between the pixel positions of the corresponding depth image and optical image and the pixel positions of the ray image. Adding the optical flow map matrix to the surface depth image and optical image matrices yields the registered surface depth image and optical image.

[0157] Furthermore, a preferred method is that, in step (4), comparing the location of the defect identification and positioning result of the X-ray image with the location of the surface feature identification result means that, for a X-ray image containing defect features, if the defect features are also identified in the surface image at the corresponding location, then the defect is determined to be a surface defect of the object; if the defect features are not identified in the surface image at the corresponding location, then the defect features are determined to be an internal defect of the object.

[0158] like Figure 1 As shown, a preferred embodiment of the present invention provides a weld internal defect detection and identification system, the hardware of which mainly includes: an X-ray machine 1, a radiation detector 2, a camera 3, a camera 4, and a welded component 5. The X-ray machine 1 emits X-rays when turned on. X-rays can penetrate the welded component. Depending on the thickness, density, etc., of the object along the ray propagation path, uniformly intensified rays will experience varying degrees of intensity attenuation after passing through the welded component. By detecting the intensity changes of the rays penetrating the welded component, structural information of the welded component along the ray direction can be obtained. The radiation detector 2 is an electronic device that uses X-rays for imaging. When X-rays of different intensities irradiate the detector surface, the detector converts the radiation into a digital image. This image reflects the radiation intensity of different areas irradiated on the detector surface, thereby obtaining structural information of the welded component's interior and surface. This image can be used to determine whether defects exist in the welded component. The camera 3 and the X-ray machine 1 are placed on the incident side of the X-ray irradiation of the welded component. The camera 3 can photograph the surface of the welded component, thereby recording the structural information of the welded component surface on the incident side of the X-ray. The camera 4 and radiation detector 2 are placed on the X-ray emission side. The camera 4 can take pictures of the surface of the welded part, thereby recording the structural information of the surface of the welded part on the X-ray emission side.

[0159] During the inspection, X-ray machine 1 emits X-rays, and the radiation detector converts the X-rays that pass through the weldment and strike the detector surface to obtain a X-ray image. Simultaneously, cameras 3 and 4 simultaneously capture images of the weldment surface in the X-ray irradiation area, obtaining optical and depth images of the weldment surface, thus acquiring a multimodal image of a single location. Since the X-ray image is a two-dimensional projection of the object along the X-ray propagation direction onto the detector plane, it contains both surface and internal structural information. Cameras 3 and 4 respectively acquire optical and depth images of the weldment's X-ray incident and exit sides, thereby recording the structural information of the incident and exit surfaces along the X-ray propagation direction. Through this inspection system, three types of images containing the internal and external structural features of the weldment can be obtained, effectively distinguishing between internal defects and surface structural features contained in the X-ray images of the weldment.

[0160] like Figure 2 As shown, the automatic surface image defect recognition method of the present invention includes a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. Further, the steps of the automatic surface image defect recognition method are as follows:

[0161] (a) The surface optical image and depth image are cropped using the sliding window method. The size of the sliding window is W*W and the step size of the sliding window is S.

[0162] (b) The cropped surface optical image and depth image are input into the feature extraction module to extract image features. The feature extraction module consists of a convolutional neural network, and the optical feature map output by the feature extraction module is represented as follows: Where h1, w1, and c1 represent the height, width, and number of channels of the optical feature map, respectively. The depth feature map output by the feature extraction module is represented as follows: Where h2, w2, and c2 represent the height, width, and number of channels of the depth feature map, respectively.

[0163] (c) Input the cropped surface optical image and depth image into the feature importance calculation module to calculate the weights of the extracted features. The optical feature weights output by the feature importance model are expressed as follows: The deep feature weights output by the feature importance model are represented as follows:

[0164] (d) Optical feature map F g With optical feature weight W g Multiply to obtain the weighted optical feature map. This feature Figure 1 There are a total of h1×w1 pixels, and each pixel corresponds to a surface optical image descriptor. Ultimately, all surface optical image descriptors are composed of a set. express.

[0165] The depth feature map F s With depth feature weights W s Multiply to obtain the weighted depth feature map. This feature Figure 1 There are a total of h2×w2 pixels, and each pixel corresponds to a depth image descriptor. Ultimately, all depth optical image descriptors are composed of a set. express.

[0166] (e) Input the surface optical image bounding box regions from the training set into the feature extraction module trained on a large dataset to obtain the initial feature map, where each feature map is... Figure 1 There are h1×w1 pixels, each corresponding to an initial feature descriptor of the surface optical image. All initial feature descriptors are clustered into N1 clusters. The cluster center of each cluster is used as a sub-feature in sub-feature group 1. The initial feature descriptor subset of the k-th cluster is represented as: Calculate the distance between the initial feature descriptors in this set, a p With a q The distance is calculated as follows:

[0167]

[0168] Among them, [a p ] d Characteristic descriptor a p The d-th dimension of the data.

[0169] Sub-feature C 1k Characteristic coefficients φ k The initial value is calculated as follows:

[0170]

[0171] Computational surface optical image descriptor With sub-feature C 1k The distance is calculated using the following formula:

[0172]

[0173] Surface optical image descriptor Normalize the distances to the N1 sub-features:

[0174]

[0175] Calculate the normalized distance between all surface optical image descriptors and all sub-features. The sum of the normalized distances between all surface optical image descriptors and the k-th sub-feature is taken as the output e of the k-th encoder in encoder 1. k ,Right now:

[0176]

[0177] The final output of encoder 1 is

[0178] (f) Similar to step (e), input the depth image bounding box regions from the training set into the feature extraction module trained on the large dataset to obtain depth image feature descriptors. Cluster all initial depth feature descriptors, resulting in N2 clusters and N2 sub-features, forming sub-feature group 2. Calculate the normalized distance between all depth image descriptors and all sub-features in sub-feature group 2. The sum of the normalized distances between all depth image descriptors and the l-th sub-feature is used as the output o of the l-th encoder in encoder 2. l The final output of encoder 2 is

[0179] (g) Connect the outputs of encoder 1 and encoder 2 to obtain... Input E into the classifier, and it will output the defect category in the image.

[0180] (h) The final automatic surface image defect recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The network is trained until convergence. The optical and depth images of the surface to be detected are then cropped using a sliding window and input into the network to achieve defect recognition. If the defect category is positive, the corresponding sliding window position is the surface defect location.

[0181] like Figure 3 As shown, the automatic defect identification method for X-ray images in this invention includes a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. Further, the specific steps of the automatic defect identification method for X-ray images are as follows:

[0182] The depth image is cropped using a sliding window method. The cropped ray image is then input into the feature extraction module to extract image features. The cropped ray image is then input into the feature importance calculation module to calculate the weights of the extracted features. The ray feature map is multiplied by the ray feature weights to obtain a weighted ray feature map, where each pixel corresponds to a ray image descriptor.

[0183] The ray image bounding box regions from the training set are input into a feature extraction module trained on a large dataset to obtain initial feature maps. Each pixel in the feature map corresponds to an initial feature descriptor for the ray image. All initial feature descriptors are clustered into N3 clusters, resulting in N3 sub-features, forming sub-feature group 3. The normalized distance between all ray image descriptors and all sub-features in sub-feature group 3 is calculated. The sum of the normalized distances between all depth image descriptors and the r-th sub-feature is used as the output t of the r-th encoder in encoder 3. l The final output of encoder 3 is Input E3 into the classifier, and it outputs the defect category in the ray image. If the defect category is positive, the corresponding sliding window position is the ray defect location.

[0184] The final X-ray image defect automatic recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The network is trained until convergence. The X-ray image to be detected is then cropped using a sliding window and input into the network to achieve defect recognition and localization.

[0185] Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0186] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0187] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A non-destructive testing method based on multimodal images, characterized in that, The method includes the following steps: Step S1: Simultaneously acquire images of the object surface and ray detection images; Step S2: Register the surface image and the X-ray inspection image; Step S3: Use the surface image defect automatic recognition method to identify the surface optical image and depth image, select the images with defect features and select the defect location; Step S4: Use the X-ray image defect recognition model to identify the X-ray inspection images, select images with defect features, and outline the defect locations; Step S5: Compare the location results of the X-ray image defect identification with the surface image feature identification to determine whether the defect features belong to internal defects; Step S4 includes the following steps: Step S4.1: Use the sliding window method to crop the ray image, input the cropped ray image into the feature extraction module to extract image features; input the cropped ray image into the feature importance calculation module to calculate the weight of the extracted features; multiply the ray feature map with the ray feature weights to obtain the weighted ray feature map, where each pixel of the ray feature map corresponds to a ray image descriptor; Step S4.2: Input the ray image bounding box regions from the training set into the feature extraction module trained on the large dataset to obtain initial feature maps. Each pixel in the feature map corresponds to an initial feature descriptor of the ray image. Cluster all the initial feature descriptors of the ray images, resulting in a total of [number missing] clusters. Clusters, resulting in The r-th sub-feature is composed of 3 sub-feature groups; the normalized distance between all ray image descriptors and all sub-features in sub-feature group 3 is calculated; the sum of the normalized distances between all ray image descriptors and the r-th sub-feature is used as the output of the r-th encoder in encoder 3. The final output of encoder 3 is ;Will Input the classifier, output the defect category in the ray image; If the defect type is positive, the corresponding sliding window position is the ray defect position; Step S4.3: The final X-ray image defect automatic recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The X-ray image defect automatic recognition network is trained until convergence. The X-ray image to be detected is cropped using a sliding window and then input into the X-ray image defect automatic recognition network to achieve defect recognition and localization.

2. The non-destructive testing method based on multimodal images according to claim 1, characterized in that, In step S1: The synchronous acquisition refers to the process of taking images of the X-ray incident and exit surfaces of an object at any location when it is irradiated with X-rays and imaged with a radiation detector. The surface area captured by the camera should be inclusive of the X-ray incident and exit surfaces, and the effective detection area is determined based on the smaller area of ​​the image.

3. The non-destructive testing method based on multimodal images according to claim 1, characterized in that, In step S2: The ray image, surface depth image, and optical image are annotated with detail alignment. Then, the designed registration network is input to calculate the optical flow maps of the surface depth image and the optical image respectively. The optical flow map represents the offset between the pixel position of the corresponding depth image and optical image and the pixel position of the ray image. The surface depth image and optical image matrix are added to the optical flow map matrix to obtain the registered surface depth image and optical image.

4. The non-destructive testing method based on multimodal images according to claim 1, characterized in that, Step S3 includes the following steps: Step S3.1: Use the sliding window method to crop the surface optical image and depth image. The size of the sliding window is W*W, and the step size of the sliding window is S. Step S3.2: Input the cropped surface optical image and depth image into the feature extraction module to extract image features; The feature extraction module consists of a convolutional neural network, and the optical feature map output by the feature extraction module is represented as follows: ,in , , These represent the height, width, and number of channels of the optical feature map, respectively; the depth feature map output by the feature extraction module is represented as... ,in , , These represent the height, width, and number of channels of the depth feature map, respectively. Step S3.3: Input the cropped surface optical image and depth image into the feature importance calculation module to calculate the weights of the extracted features; The optical feature weights output by the feature importance model are expressed as follows: The deep feature weights output by the feature importance model are expressed as follows: ; Step S3.4: Transfer the optical feature map With optical feature weights Multiply to obtain the weighted optical feature map. The feature map has a total of Each pixel corresponds to a surface optical image descriptor. , Ultimately, all surface optical image descriptors are composed of a set. Indicates; the depth feature map With depth feature weights Multiply to obtain the weighted depth feature map. The feature map has a total of There are 100 pixels, and each pixel corresponds to a depth image descriptor. j Ultimately, all depth optical image descriptors are composed of a set. express; Step S3.5: Input the surface optical image bounding box regions from the training set into the feature extraction module trained on the large dataset to obtain initial feature maps. Each feature map has a total of There are 100 pixels, each corresponding to an initial feature descriptor of the surface optical image; all initial feature descriptors of the surface optical image are clustered, resulting in a total of 10 clusters. There are 1 cluster; the cluster center of each cluster serves as a sub-feature in sub-feature group 1. , The initial feature descriptor subset of the k-th cluster is represented as: ; Calculate the distances between the initial feature descriptors in this set, and The distance is calculated as follows: , in, Representing feature descriptors The Dimensional data; Sub-features Characteristic coefficients The initial value is calculated as follows: Computational surface optical image descriptor AND characteristics The distance is calculated using the following formula: ； Surface optical image descriptor and Normalize the distance between individual features: ； Calculate the normalized distance between all surface optical image descriptors and all sub-features; the sum of the normalized distances between all surface optical image descriptors and the k-th sub-feature is used as the output of the k-th encoder in encoder 1. ,Right now: ； The final output of encoder 1 is ; Step S3.6: Input the depth image bounding box regions from the training set into the feature extraction module trained on the large dataset to obtain depth image feature descriptors; cluster all initial depth feature descriptors to obtain a total of [number missing] clusters. Clusters, resulting in Sub-features are grouped into sub-feature group 2; the normalized distance between all depth image descriptors and all sub-features in sub-feature group 2 is calculated; the sum of the normalized distances between all depth image descriptors and the l-th sub-feature is used as the output of the l-th encoder in encoder 2. The final output of encoder 2 is ; Step S3.7: Connect the outputs of encoder 1 and encoder 2 to obtain... Input E into the classifier, and output the defect category in the image; Step S3.8: The final automatic surface image defect recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The automatic surface image defect recognition network is trained until convergence. The optical image and depth image of the surface to be detected are cropped using a sliding window and then input into the automatic surface image defect recognition network to achieve defect recognition. If the defect category is positive, the corresponding sliding window position is the surface defect position.

5. A non-destructive testing system based on multimodal images, characterized in that, The system includes the following modules: Module M1: Simultaneously acquires images of the object surface and ray detection images; Module M2: Registers surface images and X-ray inspection images; Module M3: Employs an automatic surface image defect recognition system to identify surface optical and depth images, select images with defect features, and outline the defect locations. Module M4: Uses a X-ray image defect recognition model to identify X-ray inspection images, select images with defect features, and outline the defect locations; Module M5: Compares the location of defects identified in X-ray images with the location of features identified in surface images to determine whether the defect features are internal defects; Module M4 includes the following modules: Module M4.1: The ray image is cropped using the sliding window method. The cropped ray image is then input into the feature extraction module to extract image features. The cropped ray image is then input into the feature importance calculation module to calculate the weights of the extracted features. The ray feature map is multiplied by the ray feature weights to obtain a weighted ray feature map. Each pixel in the ray feature map corresponds to a ray image descriptor. Module M4.2: Inputs the ray image bounding box regions from the training set into the feature extraction module trained on a large dataset to obtain initial feature maps. Each pixel in the feature map corresponds to an initial feature descriptor for the ray image. All initial feature descriptors for the ray images are clustered, resulting in a total of [number missing] clusters. Clusters, resulting in The r-th sub-feature is composed of 3 sub-feature groups; the normalized distance between all ray image descriptors and all sub-features in sub-feature group 3 is calculated; the sum of the normalized distances between all ray image descriptors and the r-th sub-feature is used as the output of the r-th encoder in encoder 3. The final output of encoder 3 is ;Will Input the classifier, output the defect category in the ray image; If the defect type is positive, the corresponding sliding window position is the ray defect position; Module M4.3: The final X-ray image defect automatic recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The X-ray image defect automatic recognition network is trained until convergence. The X-ray image to be detected is cropped using a sliding window and then input into the X-ray image defect automatic recognition network to achieve defect recognition and localization.

6. The nondestructive testing system based on multimodal images according to claim 5, characterized in that, In module M1: The synchronous acquisition refers to the process of taking images of the X-ray incident and exit surfaces of an object at any location when it is irradiated with X-rays and imaged with a radiation detector. The surface area captured by the camera should be inclusive of the X-ray incident and exit surfaces, and the effective detection area is determined based on the smaller area of ​​the image.

7. The nondestructive testing system based on multimodal images according to claim 5, characterized in that, In module M2: The ray image, surface depth image, and optical image are annotated with detail alignment. Then, the designed registration network is input to calculate the optical flow maps of the surface depth image and the optical image respectively. The optical flow map represents the offset between the pixel position of the corresponding depth image and optical image and the pixel position of the ray image. The surface depth image and optical image matrix are added to the optical flow map matrix to obtain the registered surface depth image and optical image.

8. The nondestructive testing system based on multimodal images according to claim 5, characterized in that, Module M3 includes the following modules: Module M3.1: Uses the sliding window method to crop surface optical images and depth images. The size of the sliding window is W*W, and the step size of the sliding window is S. Module M3.2: Input the cropped surface optical image and depth image into the feature extraction module to extract image features; The feature extraction module consists of a convolutional neural network, and the optical feature map output by the feature extraction module is represented as follows: ,in , , These represent the height, width, and number of channels of the optical feature map, respectively; the depth feature map output by the feature extraction module is represented as... ,in , , These represent the height, width, and number of channels of the depth feature map, respectively. Module M3.3: Input the cropped surface optical image and depth image into the feature importance calculation module to calculate the weights of the extracted features; The optical feature weights output by the feature importance model are expressed as follows: The deep feature weights output by the feature importance model are expressed as follows: ; Module M3.4: Converts optical feature maps With optical feature weights Multiply to obtain the weighted optical feature map. The feature map has a total of Each pixel corresponds to a surface optical image descriptor. , Ultimately, all surface optical image descriptors are composed of a set. Indicates; the depth feature map With depth feature weights Multiply to obtain the weighted depth feature map. The feature map has a total of There are 100 pixels, and each pixel corresponds to a depth image descriptor. j Ultimately, all depth optical image descriptors are composed of a set. express; Module M3.5: Inputs the surface optical image bounding box regions from the training set into the feature extraction module trained on a large dataset to obtain initial feature maps. Each feature map has a total of There are 100 pixels, each corresponding to an initial feature descriptor of the surface optical image; all initial feature descriptors of the surface optical image are clustered, resulting in a total of 10 clusters. There are 1 cluster; the cluster center of each cluster serves as a sub-feature in sub-feature group 1. , The initial feature descriptor subset of the k-th cluster is represented as: ; Calculate the distances between the initial feature descriptors in this set, and The distance is calculated as follows: , in, Representing feature descriptors The Dimensional data; Sub-features Characteristic coefficients The initial value is calculated as follows: Computational surface optical image descriptor AND characteristics The distance is calculated using the following formula: ； Surface optical image descriptor and Normalize the distance between individual features: ； Calculate the normalized distance between all surface optical image descriptors and all sub-features; the sum of the normalized distances between all surface optical image descriptors and the k-th sub-feature is used as the output of the k-th encoder in encoder 1. ,Right now: ； The final output of encoder 1 is ; Module M3.6: Inputs the depth image bounding box regions from the training set into the feature extraction module trained on a large dataset to obtain depth image feature descriptors; clusters all initial depth feature descriptors, resulting in a total of [number missing] clusters. Clusters, resulting in Sub-features are grouped into sub-feature group 2; the normalized distance between all depth image descriptors and all sub-features in sub-feature group 2 is calculated; the sum of the normalized distances between all depth image descriptors and the l-th sub-feature is used as the output of the l-th encoder in encoder 2. The final output of encoder 2 is ; Module M3.7: Connects the outputs of encoder 1 and encoder 2 to obtain... Input E into the classifier, and output the defect category in the image; Module M3.8: The final automatic surface image defect recognition network consists of a feature extraction module, a feature importance calculation module, sub-feature groups, an encoder, and a classifier. The automatic surface image defect recognition network is trained until convergence. The optical image and depth image of the surface to be detected are cropped using a sliding window and then input into the automatic surface image defect recognition network to achieve defect recognition. If the defect category is positive, the corresponding sliding window position is the surface defect position.

Images