A multi-processor nondestructive defect detection method based on multi-modal information fusion

Abstract

This invention relates to the field of artificial intelligence and industrial nondestructive testing, and discloses a multi-processor nondestructive defect detection method based on multimodal information fusion. The method includes: spatial registration of CT data and infrared thermal imaging data; dividing the registered data into multiple image blocks and extracting primary features; screening based on D-S evidence theory to identify suspicious image blocks; inputting the CT image blocks and infrared image blocks corresponding to the suspicious image blocks into a dual-stream convolutional neural network to output a set of suspicious image blocks; marking the connected regions of the output set, merging adjacent image blocks of the same category, calculating defect parameters, filtering out noise regions, and generating the final detection result. This invention can achieve accurate detection of various internal defects such as pores and cracks by fully utilizing the internal structural information provided by CT and the thermal diffusion anomaly information provided by infrared without damaging the multi-processor workpiece, and effectively handles sensor information conflicts and uncertainties.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and industrial nondestructive testing, specifically to a multiprocessor nondestructive defect detection method based on multimodal information fusion. Background Technology

[0002] With the continuous advancement of integrated circuit manufacturing processes, multi-processor chips (such as multi-core CPUs, GPUs, and system-on-a-chip) are becoming increasingly integrated and their internal structures increasingly complex. These chips may develop various internal defects during manufacturing, such as voids in through-silicon vias (TSVs), cracks in microbump solder joints, delamination between the chip and substrate, and microcracks caused by material thermal mismatch. These defects are often hidden inside the package and cannot be detected by conventional optical inspection, yet they can severely affect the chip's electrical performance, heat dissipation characteristics, and long-term reliability, and may even lead to system failure. Therefore, high-precision, high-reliability non-destructive testing of multi-processor chips has become a critical quality control step in the semiconductor manufacturing industry.

[0003] Currently, the main non-destructive testing methods for internal defects in chips include industrial computed tomography (CT) and infrared thermal imaging technology.

[0004] Industrial CT technology is currently the mainstream method for detecting volumetric defects within materials. It uses X-rays to penetrate the workpiece and reconstructs its three-dimensional structure using projection data from different angles, providing a clear view of the shape, size, and spatial location of defects. For defects with significant density differences, such as pores and inclusions, in multi-processor systems, industrial CT offers high detection accuracy and spatial resolution. However, industrial CT has inherent limitations in practical applications: First, it is relatively insensitive to area defects (such as tightly closed microcracks and unfused interfaces). When the crack opening width is smaller than the CT system's maximum resolution, the difference in X-ray attenuation is insufficient to create effective contrast, easily leading to missed detections. Second, CT inspections are typically time-consuming and expensive, and for workpieces with complex structures, the scanning efficiency is insufficient to meet the demands of rapid online screening. Furthermore, CT inspections rely primarily on density comparison, often failing to effectively detect early damage caused by changes in thermal conductivity or poor interfacial bonding.

[0005] Infrared thermal imaging, as a non-contact and rapid thermal detection method, has been applied in microelectronic packaging inspection in recent years. Its principle is to induce temperature changes in the chip through external thermal excitation (such as pulsed laser, hot air, or electrothermal excitation), and then use an infrared camera to record the surface temperature field distribution and heat wave conduction process. When voids, delamination, or cracks exist internally, the thermal conductivity of these defective areas changes, thus affecting the heat diffusion path and forming localized temperature anomalies (hot spots) on the chip surface. Infrared thermal imaging has high sensitivity to near-surface interface delamination and areas with abnormal thermal conductivity, and it offers fast detection speed and large detection area. However, its inherent limitations include: the detection depth is limited by the thermal diffusion length, making it difficult to accurately quantify deep defects; and the results are easily affected by factors such as uneven emissivity of the chip surface, background thermal noise, and uneven excitation, requiring improvement in repeatability and reliability.

[0006] To overcome the limitations of single detection technologies, the industry has attempted to combine industrial CT technology with infrared thermal imaging. However, existing combinations are mostly simple tandem detection methods, where the chip is scanned separately by CT and then infrared thermally imaged, with the results manually compared. This "data patchwork" rather than "information fusion" approach fails to fully leverage the complementary advantages of the two technologies and struggles to address the differences in spatial coordinates, physical meaning, and resolution between heterogeneous data. Particularly for high-density, multi-layered structures like multiprocessor chips, a systematic solution remains lacking for how to organically fuse the three-dimensional geometric information provided by CT with the thermal diffusion anomaly information provided by infrared imaging to achieve precise defect localization and qualitative and quantitative analysis.

[0007] In summary, existing nondestructive testing methods still have shortcomings in terms of reliability, efficiency, and comprehensive identification of complex defects. There is an urgent need for a testing method that can integrate multimodal information and leverage the advantages of both X-ray and infrared technologies to meet the high-precision and high-efficiency detection requirements for internal defects in the manufacturing process of multiprocessor chips. Summary of the Invention

[0008] To address the technical problems of existing technologies, such as a lack of reliability, efficiency, and comprehensive identification capabilities for complex defects, as well as the inability to fuse multimodal information and leverage the respective advantages of X-ray and infrared technologies, this invention provides a multiprocessor-based non-destructive defect detection method based on multimodal information fusion. The technical solution is as follows:

[0009] Step 1: Perform three-dimensional-two-dimensional spatial registration on the acquired industrial CT data and active infrared thermal imaging data to obtain pixel-level aligned CT slice image sequences and infrared thermogram sequences.

[0010] Step 2: Divide the registered CT image and infrared thermogram into multiple image blocks of fixed size, and extract primary features for each image block;

[0011] Step 3: Based on the DS evidence theory, a screening process is carried out to construct an identification framework that includes defective, non-defective, and uncertain images. Basic probability values ​​are assigned and multi-feature evidence is fused using the Dempster synthesis rule to screen out suspicious image blocks and remove non-defective blocks.

[0012] Step 4: Input the CT image block and infrared image block corresponding to the suspicious image block into a dual-stream convolutional neural network. This network includes parallel CT feature extraction branches and infrared feature extraction branches, feature fusion layer and classification layer, and outputs a set of suspicious image blocks.

[0013] Step 5: Post-process the output of the dual-stream convolutional neural network, mark connected regions, merge adjacent image blocks of the same category, calculate defect parameters and filter out noise regions smaller than the preset area threshold to generate the final detection result.

[0014] Further, step 1 specifically involves: setting at least three non-coplanar marker points on the multiprocessor surface; extracting the coordinates of the marker points in the CT three-dimensional coordinate system and the pixel coordinates in the infrared camera coordinate system; obtaining the rigid transformation matrix from the infrared camera coordinate system to the CT coordinate system by solving the perspective n-point problem; and based on the rigid transformation matrix, constructing a spatial mapping relationship from infrared thermal image pixels to CT slice coordinates to generate a registered infrared thermal image sequence and a corresponding CT slice image sequence.

[0015] Furthermore, the primary features in step 2 include: the gray mean, gray variance, gradient energy, and gray entropy of the CT block; and the average temperature rise, temperature rise rate, temperature variance, and thermal diffusion time constant of the infrared block.

[0016] Furthermore, step 3 specifically includes:

[0017] Step 3.1, Define the recognition framework: Where D indicates that the image patch contains defects, N indicates that it has no defects, and U indicates uncertainty;

[0018] Step 3.2: Based on the primary features extracted from each image patch, construct the basic probability assignment for each subset;

[0019] Step 3.3: If the same image patch has multiple independent evidence sources, then the Dempster synthesis rule is used for fusion.

[0020] Step 3.4: Based on the basic probability assignment after synthesis, calculate the trust function and likelihood function, and filter suspicious image blocks and defect-free blocks according to the preset threshold.

[0021] Furthermore, the basic probability assignment in step 3.2 specifically involves:

[0022] like and ,but ;

[0023] like and ,but ;

[0024] like and Or, conversely, ;

[0025] If all features are in the middle range, then ;

[0026] in The threshold is set to statistically analyze the feature distribution of defect-free samples. Assign values ​​to the basic probabilities of each subset. These are CT feature anomaly indicators and infrared feature anomaly indicators, measured using primary features.

[0027] Furthermore, step 4 uses an expanded region centered on the suspicious image patch and larger than the image patch as the network input.

[0028] Furthermore, the two-stream convolutional neural network in step 4 specifically comprises:

[0029] CT feature extraction branch: It consists of multiple convolutional blocks, each of which contains a convolutional layer, a batch normalization layer, a ReLU activation function and a max pooling layer;

[0030] Infrared feature extraction branch: It adopts a network structure similar to that of the CT feature extraction branch, but the number of input channels is the number of infrared time frames;

[0031] Feature fusion layer: Introduces an attention mechanism to weightedly fuse the output features of the two branches mentioned above;

[0032] Classification layer: The fused features are input into the fully connected layer, and finally the softmax function is connected to output the probability of each category. The category with the highest probability is taken as the defect detection result of the image patch. If the highest probability is lower than the predetermined threshold, the output is uncertain.

[0033] Furthermore, step 5 specifically includes:

[0034] Step 5.1: Input a set of suspicious image patches. Each image patch contains the following information: center coordinates of the image patch, size of the image patch, defect category, confidence level of each category. If the category is uncertain, record the maximum probability value.

[0035] Step 5.2: Filter suspicious image blocks according to the preset confidence threshold;

[0036] Step 5.3: For the retained image blocks, determine whether they are adjacent based on their center coordinates. The adjacent condition is that the absolute difference between the center coordinates in the x-axis and y-axis directions is not greater than the size of the image block.

[0037] Step 5.4: Using a breadth-first search or disjoint-set data structure algorithm, mark adjacent image patches of the same category as the same connected component;

[0038] Step 5.5: For each connected component, calculate the following parameters: defect category, coordinates of the top left and bottom right corners of the defect region, width and height, coordinates of the defect center, defect area, and confidence level;

[0039] Step 5.6: Based on actual application requirements, set a minimum defect area threshold. If the area of ​​a connected component is less than the minimum defect area threshold, it is determined to be a noise region and removed.

[0040] Step 5.7: For the retained connected components, generate the final defect detection results, including: defect ID number, defect category, defect location, defect size, and confidence level, and summarize and output them.

[0041] Beneficial effects

[0042] This invention employs a combination of image segmentation and DS evidence theory for rapid coarse screening, effectively eliminating a large number of defect-free areas and reducing the computational burden on subsequent deep networks. The uncertain subset in DS theory can explicitly handle sensor information conflicts, avoiding missed detections due to misjudgment by a single technology. The dual-stream convolutional neural network automatically learns complementary features from CT and infrared images to achieve high-precision defect classification. The overall method balances efficiency and accuracy and is suitable for online or offline detection of defects within multiprocessors. Attached Figure Description

[0043] Figure 1 The flowchart shows a multiprocessor-based nondestructive defect detection method based on multimodal information fusion.

[0044] Figure 2 This is a flowchart illustrating the spatial registration process according to an embodiment of the present invention.

[0045] Figure 3 This is a flowchart of the preliminary screening process based on the DS evidence theory according to an embodiment of the present invention;

[0046] Figure 4 This is a diagram of the two-stream convolutional neural network structure according to an embodiment of the present invention;

[0047] Figure 5 This is a flowchart illustrating the aggregation and post-processing of suspicious image blocks in an embodiment of the present invention. Detailed Implementation

[0048] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0049] like Figure 1 As shown, the present invention provides a multi-processor non-destructive defect detection method based on multimodal information fusion, comprising the following steps:

[0050] Spatial registration: The three-dimensional volume data acquired by industrial CT is precisely aligned with the time-series thermal images acquired by active infrared thermal imaging to eliminate geometric distortion and positional deviation between multimodal data, ensure that the fused information comes from a unified physical region, and avoid false detections caused by misalignment.

[0051] Block-level feature extraction and DS screening: The registered CT image and infrared thermogram are divided into multiple image blocks of fixed size, and the primary statistical features of each image block are extracted. Based on DS theory, basic probability values ​​are assigned according to the feature values ​​of each image block to initially screen out suspicious pixel blocks that may have problems.

[0052] Dual-stream convolutional neural network fusion classification: The suspicious pixel blocks obtained by DS screening are subjected to deep feature extraction and fusion classification to achieve complementary enhancement of internal structural information and thermal diffusion anomaly information. Then, after passing through a fully connected layer, the final defect category and confidence level are output, realizing high-precision and fine recognition in the second stage.

[0053] Post-processing: The classification results of each suspicious image patch output by the dual-stream convolutional neural network are post-processed: if multiple adjacent image patches are all classified as the same type of defect, they are merged into one defect region. The contour and size of the defect can be fitted based on the center coordinates of each patch, restoring the true shape of the defect and avoiding missegmentation caused by discrete blocks.

[0054] like Figure 2 As shown, the process for aligning the spatial coordinates of infrared and CT images in this embodiment includes the following steps:

[0055] Step 1: Data acquisition and preprocessing.

[0056] To align infrared thermograms and CT images at the pixel level, four non-coplanar markers (small steel balls with a diameter of 1 mm) are set on the multiprocessor surface. These markers are highlighted in the CT image and can also be identified in the infrared image due to the difference in emissivity.

[0057] Subsequently, an industrial CT system was used to perform multi-angle scanning on the multiprocessor to obtain projection data. Three-dimensional volume data was then obtained through a filtered back-projection reconstruction algorithm. A sequence of two-dimensional slice images was extracted along the Z-axis, denoted as... Where x and y are spatial coordinates, z is the slice index, and the physical size of each pixel is 0.2 mm × 0.2 mm.

[0058] Simultaneously, an active infrared thermal imaging system is employed, using a pulsed flash lamp as the thermal excitation source to heat the surface of the same processor. An infrared thermal imager is used to acquire a time-series thermal image sequence, with a sampling frequency of 50Hz and an acquisition duration of 2 seconds. The resulting thermal image sequence is denoted as follows. , where u and v are the pixel coordinates of the infrared image, and t is the time frame index.

[0059] Step 2: Extract the 3D coordinates of CT markers and the 2D pixel coordinates of infrared markers.

[0060] From CT 3D volumetric data, the center coordinates of each marker point are extracted through image segmentation or manual selection, resulting in a set of 3D coordinates of the marker points in the CT coordinate system. The frame with the clearest marker points is selected from the infrared thermal image sequence. The pixel coordinates of each marker point are extracted through image processing or manual selection, resulting in a set of two-dimensional coordinates of the marker points in the infrared image coordinate system. .

[0061] Step 3: Solve for the rigid body transformation matrix.

[0062] Using n pairs of corresponding points The perspective n-point problem (PnP) algorithm is used to calculate the rigid body transformation matrix from the infrared camera coordinate system to the CT coordinate system. Where R is a 3×3 rotation matrix and t is a 3×1 translation vector, the matrix satisfies:

[0063] ,

[0064] Where K is the pre-calibrated intrinsic parameter matrix of the infrared camera. is the scale factor.

[0065] Step 4: Construct an airdrop mapping relationship from infrared images to CT images.

[0066] For any pixel in an infrared image The corresponding ray direction is represented in the CT coordinate system as follows:

[0067] ,

[0068] in, This is the distance parameter along the ray. It is calculated by intersecting the ray with the surface of the CT 3D model (or with a specified slice plane). (Intersection) to obtain the corresponding CT space coordinates This establishes a mapping function from infrared pixel coordinates to CT slice coordinates:

[0069] ,

[0070] Where (x, y) are the pixel coordinates in the CT slice image, and z is the slice index.

[0071] Step 5: Generate the registered infrared thermal image sequence.

[0072] For each frame of infrared thermal image It iterates through all pixels, uses the mapping function FF to calculate the corresponding position of each infrared pixel in the CT slice image, and obtains the registered infrared thermal image through interpolation. To make it consistent with CT slice images Align at the pixel level.

[0073] Step 6: Output the registered results.

[0074] Output the registered infrared thermal image sequence and corresponding CT slice images This is used for subsequent image segmentation and feature extraction.

[0075] Step 7: End.

[0076] like Figure 3 As shown, the preliminary screening process based on DS evidence theory in this embodiment includes the following steps:

[0077] Step 1: Divide the aligned image into blocks.

[0078] Select the CT slice image corresponding to the area to be detected (take the slice layer corresponding to the infrared thermogram). ), and a single-frame image of the peak heating phase in the infrared thermal image. The two images are divided into multiple image blocks of size N×N pixels, where N=16 in this embodiment. The division step size is 16 pixels (without overlap), resulting in a set of CT image blocks. and the corresponding set of infrared image blocks Where M is the total number of image patches.

[0079] Step 2: Block-level primary feature extraction.

[0080] The following primary features are extracted for each image patch for subsequent DS evidence theory analysis:

[0081] CT block features include: mean gray level Gray variance gradient energy ,in Gradient and gray entropy computed for the Sobel operator ,in This represents the probability of the k-th gray level in the gray-level histogram.

[0082] Infrared block characteristics include the following: average temperature rise ,in The initial temperature rise rate before heating Temperature variance ,in The average peak temperature within the block is given by the heat diffusion time constant τ. The curve showing the change of the average temperature within the block over time is used to illustrate this. Fitting the exponential decay model Obtain, among which .

[0083] Step 3: Define the recognition framework.

[0084] Set up a recognition framework Where D indicates that the image patch contains a defect, N indicates that the image patch is non-defective, and U indicates uncertainty, meaning that the current evidence is insufficient to make a definitive judgment.

[0085] Step 4: Construct the Basic Probability Assignment (BPA).

[0086] Based on the primary features extracted from each image patch, a basic probability is assigned to each subset according to preset rules. In this embodiment, the feature distribution of defect-free samples is first statistically analyzed to determine the normal range of each feature, and then a BPA is generated according to the following rules:

[0087] Let the degree of abnormality of CT features be determined by a comprehensive index. Measurement, among which , , These are the statistical values ​​for defect-free samples. As the weighting coefficient, this embodiment takes... =0.4, =0.3, =0.3;

[0088] Similarly, infrared feature anomaly indicators ,in , , These are the statistical values ​​for defect-free samples. As the weighting coefficient, this embodiment takes... =0.4, =0.3, =0.3.

[0089] like and ,but ;

[0090] like and ,but ;

[0091] like and ,or and ,but ;

[0092] like and If they are all in the middle value, then ;

[0093] Where the threshold =2.0, =0.5, =2.0, =0.5, adjust according to the specific input data.

[0094] Step 5: Perform evidence synthesis.

[0095] In this embodiment, each image block has two independent evidence sources: CT and infrared, with BPA values ​​of [missing information]. and The fusion was performed using Dempster's synthesis rules, and the resulting fused BPA was:

[0096] ,

[0097] in, It is a conflict factor.

[0098] Step 6: Calculate the trust function and the likelihood function.

[0099] Calculate the trust function based on the synthesized BPA: And likelihood function: .

[0100] Step 7: Filter suspicious image blocks.

[0101] The filtering rules for suspicious blocks are as follows: If or If the image patch is suspicious, it is marked as a suspicious image patch and retained for subsequent deep network analysis; otherwise, if If a block is identified as defect-free, it is discarded and not proceeded to the next step. This process yields a set of suspicious image blocks. .

[0102] Step 8: End.

[0103] like Figure 4As shown, the dual-stream convolutional neural network in this embodiment includes parallel CT feature extraction branches and infrared feature extraction branches, a feature fusion layer, and a classification layer. The specific processing flow is as follows:

[0104] Step 1: Prepare input data.

[0105] To provide richer contextual information to the two-stream convolutional neural network, for each suspicious image patch, its center coordinates in the original image are used. Centered on a central region, an expanded area of ​​size M×M pixels is extracted as the network input. In this embodiment, M=32, meaning a 32×32 pixel CT image block and an infrared image block are extracted, denoted as […]. .

[0106] Step 2: Feature extraction and fusion using a two-stream convolutional neural network.

[0107] A two-stream convolutional neural network is constructed, with the following structure:

[0108] CT flow (processing) ):

[0109] Input layer: 32×32×1;

[0110] Convolutional layer C1: 32 3×3 convolutional kernels, stride 1, padding 1, followed by batch normalization and ReLU activation → output 32×32×32;

[0111] Pooling layer P1: 2×2 max pooling, step size 2 → output 16×16×32;

[0112] Convolutional layer C2: 64 3×3 convolutional kernels, stride 1, padding 1, batch normalization + ReLU → output 16×16×64;

[0113] Pooling layer P2: 2×2 max pooling, step size 2 → output 8×8×64;

[0114] Convolutional layer C3: 128 3×3 convolutional kernels, stride 1, padding 1, batch normalization + ReLU → output 8×8×128;

[0115] Global Average Pooling (GAP) layer → Outputs 128-dimensional feature vector .

[0116] Infrared Flow (Processing) ):

[0117] It adopts the exact same network structure as CT flow, with an input of 32×32×1 and an output of 128-dimensional feature vector. .

[0118] Step 3: Perform feature fusion of CT features and infrared features.

[0119] Will and The features are concatenated to obtain a 256-dimensional fused feature vector. .

[0120] Step 4: Introduce an attention mechanism.

[0121] First, calculate the attention weights. ,in , The sigmoid function outputs... Weighted fusion features .

[0122] Step 5: Classification layer processing.

[0123] The fused features The input is fed into the classification layer, which consists of the following structure:

[0124] Fully connected layer FC1: 128 neurons, ReLU activated, followed by a Dropout layer (dropout probability 0.5) to prevent overfitting.

[0125] Fully connected FC2: 64 neurons, ReLU activation, followed by a Dropout layer (dropout probability 0.5).

[0126] Output layer: 3 neurons, softmax activation, outputting three class probabilities:

[0127] Step 6: Output the classification results.

[0128] Fusing features of each suspicious image patch The input to the classification layer yields the probability of the block belonging to each category. The final determination rule is: the category with the highest probability is taken as the defect detection result for that image block. If the highest probability is less than a threshold... If the output is uncertain, a manual review will be prompted.

[0129] Step 7: End.

[0130] like Figure 5 As shown, the suspicious image block aggregation and post-processing process in this embodiment includes the following steps:

[0131] Step 1: Input the set of suspicious image patches and their classification results.

[0132] The input is a set of suspicious image patches filtered by DS and classified by a two-stream convolutional neural network (CNN). Each image patch contains the following information: image patch center coordinates Image block size Defect categories in dual-stream CNN output Confidence levels for each category If the category is uncertain, record the maximum probability value. .

[0133] Step 2: Filter out low-confidence and uncertain blocks.

[0134] Suspicious image patches are filtered based on a set confidence threshold, with the following rules: if the image patch category is uncertain and... (This embodiment takes) If the value is 0.5, the block is considered unreliable and discarded, not participating in subsequent aggregation. If an image block is classified as defect-free (i.e., classified as defect-free by dual-stream CNN), but is marked as suspicious in the DS screening and enters the CNN, then if... (This embodiment takes) If the value is 0.8, then the block is reclassified as defect-free and removed. The remaining image blocks are retained to form a set of blocks to be aggregated. .

[0135] Step 3: Establish spatial adjacency relationships.

[0136] For all blocks to be aggregated, construct spatial adjacency relationships based on their center coordinates. The condition for two image blocks to be adjacent is defined as follows: if the center of block A... Center of Block B satisfy: If so, then the two image blocks are determined to be adjacent blocks.

[0137] Step 4: Connected component labeling.

[0138] A breadth-first search or disjoint-set data structure algorithm is used to label adjacent image patches of the same category as the same connected component. Specifically, two patches must be of the same category (both pores or both cracks) to be merged. If two patches are of different categories (e.g., one a pore and the other a crack), they are not merged even if they are adjacent, and are considered different defects. This process yields several connected components. Each component contains a set of similar and adjacent image blocks.

[0139] Step 5: Calculate the defect parameters for each connected component.

[0140] For each connected component The following parameters are calculated: defect category, coordinates of the top left and bottom right corners of the defect area, width and height, defect center coordinates, defect area, and confidence level (in this embodiment, the average confidence level of all image blocks within the component is selected as the confidence level of the connected component).

[0141] Step 6: Filter out small noise areas.

[0142] Set a minimum defect area threshold based on actual application requirements. If the area of ​​a connected component is < If the area is not clearly defined as noise, it will be removed. In this embodiment, based on the image resolution (0.2 mm / pixel) and the minimum size of the defect of interest (e.g., a pore with a diameter of 0.5 mm), the minimum size is determined. =10 pixels (approximately 0.4mm²), which can be adjusted according to actual needs.

[0143] Step 7: Generate the final detection results.

[0144] For each retained connected component, the final defect detection result is generated, including: defect ID number, defect category, defect location, defect size, and confidence level.

[0145] Step 8: Output the test report.

[0146] All defect detection results are summarized to generate a structured inspection report.

[0147] Step 9: End.

[0148] Through the above steps, this invention balances efficiency and accuracy in multi-processor non-destructive testing, solving problems such as high false negative rate, slow detection speed, and weak uncertainty handling capability of single technologies. It can reliably detect various types of defects such as porosity and cracks, providing an efficient and accurate non-destructive testing solution for quality control in key fields such as aerospace, nuclear energy equipment, and pressure vessels.

[0149] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-processor non-destructive defect detection method based on multi-modal information fusion, characterized in that, Includes the following steps: Step 1: Perform three-dimensional-two-dimensional spatial registration on the acquired industrial CT data and active infrared thermal imaging data to obtain pixel-level aligned CT slice image sequences and infrared thermogram sequences. Step 2: Divide the registered CT image and infrared thermogram into multiple image blocks of fixed size, and extract primary features for each image block; Step 3: Based on the DS evidence theory, a screening process is carried out to construct an identification framework that includes defective, non-defective, and uncertain images. Basic probability values ​​are assigned and multi-feature evidence is fused using the Dempster synthesis rule to screen out suspicious image blocks and remove non-defective blocks. Step 4: Input the CT image block and infrared image block corresponding to the suspicious image block into a dual-stream convolutional neural network. This network includes parallel CT feature extraction branches and infrared feature extraction branches, feature fusion layer and classification layer, and outputs a set of suspicious image blocks. Step 5: Post-process the output of the dual-stream convolutional neural network, mark connected regions, merge adjacent image blocks of the same category, calculate defect parameters and filter out noise regions smaller than the preset area threshold to generate the final detection result.

2. The method for multi-processor non-destructive defect detection based on multi-modal information fusion as claimed in claim 1, wherein, Step 1 specifically involves: setting at least three non-coplanar marker points on the multiprocessor surface; extracting the coordinates of the marker points in the CT three-dimensional coordinate system and the pixel coordinates in the infrared camera coordinate system; and obtaining the rigid body transformation matrix from the infrared camera coordinate system to the CT coordinate system by solving the perspective n-point problem. Based on the rigid body transformation matrix, a spatial mapping relationship between infrared thermal image pixels and CT slice coordinates is constructed, generating a registered infrared thermal image sequence and a corresponding CT slice image sequence.

3. The multi-processor non-destructive defect detection method based on multi-modal information fusion as claimed in claim 1, wherein, The primary features in step 2 include: the mean gray level, gray level variance, gradient energy, and gray level entropy of the CT block; and the average temperature rise, temperature rise rate, temperature variance, and thermal diffusion time constant of the infrared block.

4. The multi-processor non-destructive defect detection method based on multi-modal information fusion as claimed in claim 1, wherein, Step 3 specifically involves: Step 3.1, define the recognition framework: wherein D indicates that the image block contains a defect, N indicates no defect, and U indicates uncertainty; Step 3.2: Based on the primary features extracted from each image patch, construct the basic probability assignment for each subset; Step 3.3: If the same image patch has multiple independent evidence sources, then the Dempster synthesis rule is used for fusion. Step 3.4: Based on the basic probability assignment after synthesis, calculate the trust function and likelihood function, and filter suspicious image blocks and defect-free blocks according to the preset threshold.

5. The multi-processor non-destructive defect detection method based on multi-modal information fusion as claimed in claim 4, wherein, The basic probability assignment in step 3.2 is specifically as follows: If and then ; If and then ; If and or the opposite, then ; If all features are in the intermediate range, then ; wherein is a threshold value set by statistical distribution of defect-free samples, is a basic probability value of each subset, is a CT feature anomaly index and an infrared feature anomaly index, measured by a primary feature.

6. The method for multi-processor non-destructive defect detection based on multi-modal information fusion as claimed in claim 1, wherein: Step 4 uses an expanded region centered on the suspicious image patch and larger than the image patch as the network input.

7. The multi-processor non-destructive defect detection method based on multi-modal information fusion as claimed in claim 6, wherein, The dual-stream convolutional neural network in step 4 is specifically as follows: CT feature extraction branch: It consists of multiple convolutional blocks, each of which contains a convolutional layer, a batch normalization layer, a ReLU activation function and a max pooling layer; Infrared feature extraction branch: It adopts a network structure similar to that of the CT feature extraction branch, but the number of input channels is the number of infrared time frames; Feature fusion layer: Introduces an attention mechanism to weightedly fuse the output features of the two branches mentioned above; Classification layer: The fused features are input into the fully connected layer, and finally the softmax function is connected to output the probability of each category. The category with the highest probability is taken as the defect detection result of the image patch. If the highest probability is lower than the predetermined threshold, the output is uncertain.

8. The method for multi-processor non-destructive defect detection based on multi-modal information fusion as claimed in claim 1, wherein: Step 5 specifically involves: Step 5.1: Input a set of suspicious image patches. Each image patch contains the following information: center coordinates of the image patch, size of the image patch, defect category, confidence level of each category. If the category is uncertain, record the maximum probability value. Step 5.2: Filter suspicious image blocks according to the preset confidence threshold; Step 5.3: For the retained image blocks, determine whether they are adjacent based on their center coordinates. The adjacent condition is that the absolute difference between the center coordinates in the x-axis and y-axis directions is not greater than the size of the image block. Step 5.4: Using a breadth-first search or disjoint-set data structure algorithm, mark adjacent image patches of the same category as the same connected component; Step 5.5: For each connected component, calculate the following parameters: defect category, coordinates of the top left and bottom right corners of the defect region, width and height, coordinates of the defect center, defect area, and confidence level; Step 5.6: Based on actual application requirements, set a minimum defect area threshold. If the area of ​​a connected component is less than the minimum defect area threshold, it is determined to be a noise region and removed. Step 5.7: For the retained connected components, generate the final defect detection results, including: defect ID number, defect category, defect location, defect size, and confidence level, and summarize and output them.

Images