A multi-modal fusion defect detection method and system based on ultrasonic coupling quality judgment

Abstract

The application provides a multimodal fusion defect detection method and system based on ultrasonic coupling quality judgment, which comprises the following steps: S1, collecting product ultrasonic echo original data and continuously monitoring the coupling state of an ultrasonic probe and the surface of a product to be detected; S2, performing primary filtering on the ultrasonic echo data to obtain suspected defect echo data; S3, generating C-scan images and T-scan images of different levels in the product; S4, superimposing and fusing the C-scan images and the T-scan images; based on the shape contour of the product to be detected and the scanning coordinate system, realizing pixel-level alignment of the double-mode images; in combination with the suspected defect echo data extracted in step S2, performing feature enhancement and extraction on the fused images to obtain multimodal fusion feature data; S5, performing model training and defect recognition and analysis to determine the position, type and size of the defect and output the final defect detection result.

Description

Technical Field

[0001] This invention relates to the field of ultrasonic nondestructive testing technology, and in particular to a multimodal fusion defect detection method and system based on ultrasonic coupling quality judgment. Background Technology

[0002] Currently, many industrial products require ultrasonic testing to detect internal defects in materials, such as pores, cracks, inclusions, and delamination generated during production. Traditional ultrasonic testing systems receive the reflected echoes from defects using a probe and determine the presence and size of defects based on the echo's location and amplitude. However, in actual testing, the process is frequently interfered with by various non-defect factors, leading to misjudgments. These interferences mainly include: 1. Unstable coupling: Acoustic coupling between the probe and the workpiece requires a couplant. Uneven couplant thickness, the presence of air bubbles, or uneven workpiece surface can all lead to unstable acoustic energy transmission, resulting in amplitude fluctuations or abnormal echoes, which can easily be misjudged as defects.

[0003] 2. Interference from the workpiece's own structure: such as the workpiece's edges, grooves, threads and other geometric structures, which will generate specific non-defect echoes.

[0004] 3. Material noise: For coarse-grained materials (such as austenitic stainless steel and cast iron), ultrasonic waves will be scattered at the grain boundaries, forming grass noise, which can mask the signals of small defects or generate false signals.

[0005] 4. Electronic noise: The electronic system of the detection equipment itself can also introduce noise.

[0006] 5. Some special materials inside the product, such as the tin-rich active non-metals used in the solder of the ceramic substrate, may appear as defects in the image during ultrasonic scanning, which can easily lead to misjudgment of defects.

[0007] Most existing ultrasonic testing equipment relies on the operator's experience to identify and eliminate these interferences, resulting in low automation, low testing efficiency, and results that are greatly affected by subjective factors. Although some advanced equipment has fixed threshold filtering or DAC / TCG curves, its adaptability to complex and variable interference sources is poor, and it still cannot effectively solve the problem of high false positive rates.

[0008] Therefore, it is necessary to design a multimodal fusion defect detection method and system based on ultrasonic coupling quality judgment to solve the above problems. Summary of the Invention

[0009] The purpose of this invention is to provide a multimodal fusion defect detection method based on ultrasonic coupling quality judgment. By comparing real-time ultrasonic signals with typical non-defect interference characteristics and combining the coupling state, it can effectively distinguish between real defect echoes and interference signals caused by poor coupling, workpiece structure, material noise, etc., thereby reducing interference misjudgment.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal fusion defect detection method based on ultrasonic coupling quality judgment, comprising the following steps: S1. Collect raw ultrasonic echo data of the product and continuously monitor the coupling state between the ultrasonic probe and the surface of the product to be tested, and retain ultrasonic echo data that meet the coupling state. S2. Perform primary filtering on the ultrasonic echo data retained in step S1 to eliminate interference signals and obtain suspected defect echo data. S3. The suspected defect echo data in step S2 is processed by imaging to generate C-scan and T-scan images of different levels inside the product. S4. The C-scan image and T-scan image generated in step S3 are superimposed and fused using an adaptive weighted fusion combined with a soft alignment attention multimodal fusion algorithm; pixel-level alignment of the dual-modal images is achieved based on the outline of the product to be detected and the scanning coordinate system; combined with the suspected defect echo data extracted in step S2, feature enhancement and extraction are performed on the fused image to obtain multimodal fusion feature data; S5. Based on the multimodal fusion feature data, perform model training and defect identification and analysis to determine the location, type and size of the defect, and output the final defect detection result.

[0011] As a further improved technical solution of the present invention, in step S1, the coupling state is monitored through the following steps: real-time analysis of the initial pulse characteristics and the first bottom echo characteristics of the C-scan ultrasound signal, and obtaining a quantified coupling quality index (CQI) based on the initial pulse characteristics and the first bottom echo characteristics; setting a CQI threshold, if the current CQI is lower than the threshold, marking the current frame data as invalid and prompting to adjust the coupling state; if the current CQI is higher than the threshold, retaining the ultrasound echo data and the corresponding CQI value.

[0012] As a further improvement to the present invention, the Coupling Quality Index (CQI) is calculated in the following manner: First, calculate the peak value of the ultrasonic echo amplitude: amp = gain × amp_tmp, where amp is the peak-to-peak value of the ultrasonic echo amplitude, gain is the soft gain coefficient, and amp_tmp is the peak or trough value of the amplitude obtained from the acquisition. Then determine the input range DIR based on the number of data acquisition cards: When amp > 0, DIR = 32767 ÷ 2^(16 - dataBitSize); When amp≤0, DIR=−32768÷2^(16-dataBitSize); Where dataBitSize is the number of bits in the acquisition card, and 32767 and -32768 are the standard quantization extreme values ​​of the 16-bit acquisition system; Finally, the coupling quality index is calculated: CQI = 254 × amp ÷ DIR. The value of CQI ranges from 0 to 254. The larger the value, the better the coupling quality. CQI ≥ 60 is set as the coupling quality is qualified.

[0013] As a further improvement to the present invention, in step S2, the primary filtering includes: S21. Retrieve the preset interference feature database, match the ultrasonic echo data retained in step S1 with the interference features, and determine whether the ultrasonic echo data is an interference signal. S22. Perform a time-domain consistency check on the ultrasonic echo data retained in step S1; Ultrasonic echo data that simultaneously meets the following conditions is used as suspected defect echo data: it is not identified as an interference signal in step S21, and it satisfies time-domain consistency in step S22.

[0014] As a further improved technical solution of the present invention, the interference feature database is constructed through the following steps: collecting and saving interference signals under different operating conditions through single-point acquisition; performing surface tracking, peak calculation, Fourier transform and wavelet transform on the interference signals in sequence, separating and extracting interference signal fingerprint features including time-domain waveform, spectral centroid, bandwidth, and wavelet decomposition coefficients, and storing them according to operating condition type to construct the interference feature database; the operating condition type includes surface roughness, material and specific structural parts.

[0015] As a further improved technical solution of the present invention, the interference feature matching step in step S21 is as follows: extract the time domain, frequency domain, and wavelet features of the corresponding ultrasonic echo signal from the ultrasonic echo data retained in step S1, and perform similarity calculation with the feature templates in the interference feature database; set a similarity threshold, and if the similarity reaches the similarity threshold, it is determined to be an interference signal.

[0016] As a further improvement of the present invention, the time domain consistency judgment condition is as follows: analyze the time domain performance of the ultrasound echo data on the C-scan probe and the T-scan probe, and determine whether the ultrasound echo signal has a ±1us delay, amplitude gradient and time axis fluctuation stability within 0.005us within the probe focusing range; if it does not have these properties, it is judged as an interference signal.

[0017] As a further improvement to the present invention, in step S4, the C-scan images and T-scan images of different levels are superimposed using a Stack model or a Transformer model. The image superposition process includes: S41. Construction of dual-stream coding layer: The encoder-decoder soft-aligned attention paradigm is adopted to construct a dual-stream parallel coding structure. The shared encoder is built based on SwinTransformer, and the private encoder adopts a dual-branch independent CNN structure to adapt to the feature extraction requirements of C-scan and T-scan images respectively. S42. Global and Local Feature Extraction: Global context and spatial contour features shared by C-scan and T-scan images are extracted synchronously through a shared encoder; local detail features unique to C-scan images and hierarchical structure features unique to T-scan images are extracted through two independent branches of a private encoder, preserving the differential information of the dual-modal images. S43. Soft-aligned attention feature fusion: Define Query(Q) as the shared feature Fshare, and Key(K) and Value(V) as the concatenated result of private features [FC, FT]; calculate the soft-aligned attention weights and fused features: , The fusion feature is calculated as Ffusion = Fshare + Attention(Q, K, V). S44. Image Reconstruction and Output: The fused features Ffusion are input into the transposed convolutional decoder and upsampled to the original size; after 1×1 convolution dimensionality reduction, the superimposed image Ifinal is output, which is inversely normalized to restore the original pixel value range, and the multimodal fused feature data is output at the same time.

[0018] As a further improvement to the present invention, step S5 includes the image training process as follows: S51. Determine training data and test data: Use the multimodal fusion feature data and the corresponding gray-white area of ​​the superimposed fusion image output in step S4 as training data, and the black area as test data. S52. First-level model training and new feature construction: The training data is input into the logistic regression base model for training. The pixel values ​​of the C-scan and T-scan images are flattened into one-dimensional feature vectors. The L2 regularization intensity is selected as a hyperparameter, and K-fold cross-validation is used to evaluate the model performance and optimize the hyperparameters. The calculation formula for K-fold cross-validation is as follows: , Where K represents the number of non-overlapping folds into which the training dataset is divided. This represents the binary cross-entropy loss value calculated by the model on the validation set when the i-th fold of data is used as the validation set. This represents the average loss value of K-fold cross-validation, used to quantify the model's generalization ability; After training is complete, the defect probability predictions generated by the logistic regression model on the training data are collected as a new feature dataset. S53, Second-level model training: From the original superimposed and fused images corresponding to the training data, based on the defect center, 128×128 pixel defect image details are extracted as input data. A CNN convolutional neural network is used for training, and L2 regularization intensity is selected as a hyperparameter. Dropout layer and batch normalization layer are added to the convolutional layer, and L2 regularization is added to the fully connected layer to suppress overfitting.

[0019] The present invention also aims to provide a multimodal fusion defect detection method based on ultrasonic coupling quality judgment.

[0020] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal fusion defect detection system based on ultrasonic coupling quality judgment, comprising: The ultrasonic acquisition and control unit is used to acquire raw ultrasonic echo data of the product according to the product formula; The coupling status monitoring unit is used to monitor and evaluate the coupling status between the ultrasonic probe and the product under test in real time, and output the coupling quality index. Interference feature database unit, used to construct interference feature database; The intelligent decision unit is connected to the ultrasonic acquisition control unit, the coupling state monitoring unit, and the interference feature database unit, respectively, and is used to determine the coupling state based on the coupling quality index and to determine whether the original ultrasonic echo data is interference data based on the interference feature database. The image generation unit is used to generate high-precision C-scan and T-scan images of different levels inside the product; The image overlay unit is used to overlay the generated C-scan and T-scan images into CT scan images; Image data training unit, used for data training on the generated C-scan, T-scan, and T-scan images; The defect detection and processing unit is used for defect identification and analysis, determining the location, type, and size of defects, and outputting the final defect detection results.

[0021] As can be seen from the above technical solutions, this invention can effectively distinguish between real defect echoes and interference signals caused by poor coupling, workpiece structure, material noise, etc. by real-time monitoring of the coupling state between the probe and the product and by comparing real-time ultrasonic signals with typical non-defect interference characteristics, thereby reducing interference misjudgment. Furthermore, by generating high-precision C-scan and T-scan images of different levels inside the product, and performing image overlay, image training, and defect detection processing on the overlaid images, interference misjudgment is reduced, and the actual defects inside the product and the level at which the defects occur are detected. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of image training in one embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the image overlay effect of the Stack model in Example 1. The left image is the C-scan image, the middle image is the T-scan image, and the right image is the overlay image after the C-scan image and the T-scan image are superimposed.

[0024] Figure 3 This is a schematic diagram of the image overlay effect of the Stack model in Example 2, where the left image is the C-scan image, the middle image is the T-scan image, and the right image is the overlay image after the C-scan image and the T-scan image are superimposed.

[0025] Figure 4 This is a schematic diagram of the overlay effect of the Transformer model images in Example 3. The upper left image is the C-scan image, the upper right image is the T-scan image, the lower left image is the overlay image of the C-scan image and the T-scan image, and the lower right image is the overlay image of the individual T-scan images.

[0026] Figure 5 This is a schematic diagram of the overlay effect of the Transformer model images in Example 3. The upper left image is the C-scan image, the upper right image is the T-scan image, the lower left image is the overlay image of the C-scan image and the T-scan image, and the lower right image is the overlay image of the individual T-scan images. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] The following implementation methods all use copper-clad ceramic plates (aluminum nitride ceramic substrate thickness 0.3-1.0mm, copper cladding layer thickness 0.1-0.2mm) as the detection object, focusing on the inherent structural characteristics of the product—the elimination of interference from the tin-rich phase (the tin-rich phase is an inherent structure formed during the brazing and copper cladding process of the copper-clad ceramic plate, not a defect, but its acoustic impedance is somewhat similar to that of defects (cracks, pores, inclusions, incomplete penetration), which can easily interfere with defect detection). The detection method of this invention can eliminate 99% of the interference from the tin-rich phase and accurately detect the true defects of the product.

[0029] Example 1

[0030] The test object in this embodiment is the finished copper-clad ceramic plate on the mass production line of copper-clad ceramic plate, with a specification of 0.5 mm aluminum nitride ceramic base thickness and 0.15 mm copper cladding layer thickness.

[0031] Equipment preparation: A dual-frequency ultrasonic probe (C-scan probe frequency 50MHz, T-scan probe frequency 10MHz) was used; the ultrasonic acquisition card number was 12-bit; the coupling agent was a high-viscosity ultrasonic coupling gel adapted to the smooth surface of the copper-clad ceramic board; a scanning platform was set up, the scanning step was set to 0.28mm, and the scanning range covered the entire surface of the copper-clad ceramic board (standard size 50mm×50mm), with a focus on the interface area (0.15-0.16mm depth, corresponding to the interface between the copper layer and the ceramic layer, where the tin-rich phase is most concentrated and the interference is most obvious).

[0032] Step S1: Start the ultrasonic acquisition system and continuously acquire raw ultrasonic echo data of the copper-clad ceramic substrate, while simultaneously monitoring the coupling state between the ultrasonic probe and the surface of the copper layer of the copper-clad ceramic substrate; calculate the coupling quality index (CQI) as follows: First, calculate the peak value of the ultrasonic echo amplitude: amp = gain × amp_tmp, where amp is the peak-to-peak value of the ultrasonic echo amplitude, gain is the soft gain coefficient, and amp_tmp is the peak or trough value of the amplitude obtained from the acquisition. Then determine the input range DIR based on the number of data acquisition cards: When amp > 0, DIR = 32767 ÷ 2^(16 - dataBitSize); When amp≤0, DIR=−32768÷2^(16-dataBitSize); Where dataBitSize is the number of bits in the acquisition card, and 32767 and -32768 are the standard quantization extreme values ​​of the 16-bit acquisition system; Finally, the coupling quality index is calculated: CQI = 254 × amp ÷ DIR. A CQI ≥ 60 is set as the acceptable threshold (interface area detection requires high coupling stability; to avoid coupling fluctuations combined with tin-rich phase interference, leading to misjudgment of defects); if CQI < 60, the system automatically prompts for adjustment of probe pressure and coupling agent dosage, marks the current frame data as invalid, and discards it; if CQI ≥ 60, the ultrasonic echo data and corresponding CQI values ​​are retained, each frame of data is associated with the scan coordinates, and the echo data of the interface area is the primary focus (tin-rich phase interference is mainly concentrated here).

[0033] Step S2: Retrieve the preset copper-clad ceramic plate interference feature database, focusing on supplementing the tin-rich phase interference feature template (collect the ultrasonic signal of the tin-rich phase at the interface of the defect-free copper-clad ceramic plate, extract its inherent time-domain, frequency-domain, and wavelet features as the core interference template), while also including conventional interference features such as copper layer surface roughness and interface solder residue; perform interference feature matching on the ultrasonic echo data retained in S1, focusing on comparing the similarity between the tin-rich phase interference template and the echo signal. The similarity judgment method is as follows: extract features from the ultrasonic echo data and map them to the feature space to obtain the measured feature point (x2, y2); retrieve the standard feature point (x1, y1) corresponding to the tin-rich phase interference feature template; calculate the Euclidean distance between the two points and convert it into similarity, using a similarity threshold of 80% as the judgment criterion. When the similarity is greater than or equal to the threshold, the corresponding echo signal is determined to be a tin-rich phase interference signal. The Euclidean distance calculation formula is as follows: Similarity = 1 / (1+ d Combined with time-domain consistency judgment (analyzing the delay and amplitude gradient of C-scan and T-scan echo signals with a deviation ≤ 0.1μs), it accurately distinguishes between tin-rich phase interference signals and suspected defect echo signals; it outputs echo data that are not identified as tin-rich phase or other interference signals and meet time-domain consistency as suspected defect echo data, thus achieving preliminary elimination of tin-rich phase interference.

[0034] Step S3: Image processing is performed on the suspected defect echo data to generate C-scan images (resolution 0.05mm×0.05mm, focusing on the planar distribution features of interface cracks and incomplete solder penetration) and T-scan images (resolution 0.05mm, focusing on the depth distribution of interface defects) at different levels inside the copper-clad ceramic board. The grayscale difference between defects and tin-rich phases is enhanced by imaging algorithms (tin-rich phase areas show a stable low grayscale uniform distribution in the image, while defect areas show irregular grayscale abrupt changes), clearly distinguishing the copper-clad layer, ceramic layer, interface tin-rich phase areas, and suspected defect areas, providing a visual basis for subsequent fusion processing and interference elimination.

[0035] Step S4: The generated C-scan and T-scan images are superimposed and fused using an adaptive weighted fusion combined with a soft-alignment attention multimodal fusion algorithm; pixel-level alignment of the dual-modal images is achieved based on the outline of the product to be inspected and the scanning coordinate system; combined with the suspected defect echo data extracted in step S2, feature enhancement and extraction are performed on the fused image to obtain multimodal fusion feature data. The specific process includes: S41. Dual-stream coding layer construction: A dual-stream parallel coding structure is constructed using an encoder-decoder soft-aligned attention paradigm. The shared encoder is built based on SwingTransformer, while the private encoder adopts a dual-branch independent CNN structure to adapt to the feature extraction requirements of C-scan and T-scan images respectively. The focus is on extracting the planar contour features of defects in C-scan images and the depth features of defects in T-scan images, while strengthening the feature differences between tin-rich phases and defects (focusing on extracting the uniformity features of tin-rich phases and the irregularity features of defects). S42. Global and Local Feature Extraction: Global context and spatial contour features shared by C-scan and T-scan images are extracted synchronously through a shared encoder; local detail features unique to C-scan images and hierarchical structure features unique to T-scan images are extracted through two independent branches of a private encoder, preserving the differential information of the dual-modal images. S43. Soft-aligned attention feature fusion: Define Query(Q) as the shared feature Fshare, and Key(K) and Value(V) as the concatenated result of private features [FC, FT]; calculate the soft-aligned attention weights and fused features: , The fusion feature is calculated as Ffusion = Fshare + Attention(Q, K, V). S44. Image Reconstruction and Output: The fused features (Ffusion) are input into the transposed convolutional decoder and upsampled to their original size. After 1×1 convolution dimensionality reduction, the superimposed image (Ifinal) is output. Inverse normalization is performed to restore the original pixel value range, and multimodal fused feature data is also output. The fused features are calculated using a soft-aligned attention algorithm, with a focus on weighting defect features and suppressing interference from tin-rich phase features. The superimposed image is reconstructed by the transposed convolutional decoder, and multimodal fused feature data is extracted (focusing on grayscale, edge, and depth features of defects, while weakening the influence of tin-rich phase features).

[0036] Figure 2 The image above is a partial image overlay result from Example 1. The C-scan image (left image) and T-scan image (middle image) simultaneously show defect features (the white part in the upper half of the left image and the gray part in the upper half of the middle image). After overlaying the images using the Stack model, the overlay image (right image) shows the defect features (the bright yellow part), which is determined to be a real defect.

[0037] Step S5: Based on the multimodal fusion feature data, perform model training and defect identification and analysis to determine the location, type, and size of the defect, and output the final defect detection result. The image training process includes: S51. Determine training and test data: Use the multimodal fusion feature data and the corresponding gray-white areas of the superimposed fusion image output in step S4 as training data, and the black areas as test data. The training data should primarily include mixed samples of "tin-rich phase interference + real defects", pure tin-rich phase samples, and pure defect samples, with tin-rich phase interference samples accounting for 40% to ensure the model has strong interference rejection capabilities. Please refer to... Figure 1 The diagram shown is a schematic of the image training process in this embodiment. The red arrow indicates the test data where parts of the image are covered with black lines. The red box indicates the real defects in the product image as training data. The white stripes and white dots are artificial defects. The bright white area on the right is the real defect of the product.

[0038] S52. First-level model training and new feature construction: The training data is input into the logistic regression base model for training. The pixel values ​​of the C-scan and T-scan images are flattened into one-dimensional feature vectors. The L2 regularization intensity is selected as a hyperparameter, and K-fold cross-validation is used to evaluate the model performance and optimize the hyperparameters. The calculation formula for K-fold cross-validation is as follows: , Where K represents the number of non-overlapping folds into which the training dataset is divided. This represents the binary cross-entropy loss value calculated by the model on the validation set when the i-th fold of data is used as the validation set. This represents the average loss value of K-fold cross-validation, used to quantify the model's generalization ability; After training, the defect probability predictions generated by the logistic regression model on the training data are collected as a new feature dataset.

[0039] S53, Second-level model training: From the original superimposed and fused images corresponding to the training data, based on the defect center, 128×128 pixel defect image details are extracted as input data. A CNN convolutional neural network is used for training, and L2 regularization intensity is selected as a hyperparameter. Dropout layer and batch normalization layer are added to the convolutional layer, and L2 regularization is added to the fully connected layer to suppress overfitting.

[0040] Furthermore, defect features are extracted and standardized. Combined with the weakened tin-rich phase information in the fusion features, the defect location (accurate to 0.1mm), type (interface crack, incomplete penetration) and size are accurately determined. A standardized inspection report is output, marking qualified / unqualified products and clearly indicating the elimination of tin-rich phase interference.

[0041] Example 2

[0042] The specifications of the copper-clad ceramic plate tested in this implementation are: aluminum nitride ceramic base thickness 0.8mm and copper cladding layer thickness 0.2mm.

[0043] Equipment preparation: High-frequency ultrasonic probes (C-scan probe frequency 35MHz, T-scan probe frequency 20MHz); 12-bit ultrasonic acquisition card; low-viscosity ultrasonic coupling agent (suitable for irregular areas on the surface after welding, ensuring uniform coupling and avoiding dual interference from welding residue and tin-rich phase); the scanning platform is set with a scanning step of 0.1mm, focusing on the welding interface (depth 0.2-0.3mm) for precise scanning.

[0044] Step S1: Collect raw ultrasonic echo data of the copper-clad ceramic plate after welding. When monitoring the coupling state, adjust the CQI qualification threshold to ≥70. When calculating CQI, set the soft gain coefficient gain to 1.2 to improve the amplitude recognition accuracy of the weld interface defect echo signal (the acoustic impedance of tin-rich materials is low, and the echo amplitude is close to that of small defects, so the defect signal needs to be enhanced). Retain echo data with CQI≥70 and the corresponding CQI value, focusing on retaining the scanning data of the weld interface area, and simultaneously record the scanning coordinates of the weld interface.

[0045] Step S2: Update the interference feature database and add a new interference feature template for tin-rich phase aggregation at the welding interface (collect ultrasonic signals of tin-rich phase aggregation at the defect-free welding interface, extract fingerprint features such as spectral bandwidth and wavelet decomposition coefficients, and compare them with welding defect signals to identify differences). It also includes conventional interference features such as welding residue and oxide layer. When matching interference features, the similarity threshold is adjusted to 85%. The focus is on comparing the signal differences between tin-rich phase aggregation and minor defects. In the temporal consistency judgment, the time axis stability deviation is ≤0.05μs. Double screening is used to exclude tin-rich phase interference and other interference signals, accurately extract the echo data of suspected defects at the welding interface, and initially achieve the elimination of tin-rich phase interference.

[0046] Step S3: Image processing is performed on the suspected defect echo data. The C-scan image focuses on the tiny defects at the welding interface, with the resolution improved to 0.02mm×0.02mm, clearly showing the planar morphology of the defects (irregular patches, linear cracks), which is clearly distinguishable from the uniform morphology of the tin-rich phase aggregation. The T-scan image focuses on the depth direction of the welding interface, clearly distinguishing the three layers of copper clad layer, welding layer and metal substrate, clearly showing the depth and thickness distribution of the defects, which differs from the uniform depth distribution of the tin-rich phase aggregation, further enhancing the image differentiation between the tin-rich phase and the defects.

[0047] Step S4: Use the Stack model to perform dual-modal image overlay and fusion. In the dual-stream coding layer, the CNN branch of the private encoder is optimized into a deep convolutional structure to improve the feature discrimination ability between small defects and tin-rich phase clusters. During soft-aligned attention fusion, the weight allocation of Query, Key, and Value is adjusted to focus on strengthening defect features and suppressing tin-rich phase cluster features, highlighting the differences in grayscale, texture, and shape between the two. After fusion, the image is enhanced to improve the edge contrast of the defect area and weaken the features of the tin-rich phase cluster area, outputting multimodal fusion feature data (focusing on the density, size, and depth features of small defects, maximizing the elimination of tin-rich phase interference).

[0048] Figure 3 The image above is a partial image overlay result from Example 2. The C-scan image (left image) does not show defect features, while the T-scan image (middle image) shows defect features (the gray area in the upper half of the middle image). After overlaying the images using the Stack model, the overlay image (right image) does not show defect features and is determined to be an interference signal.

[0049] Step S5: During model training, the training data will primarily include mixed samples of "tin-rich phase aggregation interference at the welding interface + minor defects" (tin-rich phase interference samples account for 45%). Simultaneously, pure tin-rich phase samples, pure defect samples, and welding residue samples will be added for comparative training. K-fold cross-validation K=6, and the L2 regularization strength will be adjusted to 0.001 to suppress model overfitting and enhance the model's interference rejection capability. During defect identification and analysis, the type determination threshold will be set to 90%. Combining the differences between tin-rich phase interference characteristics and defect characteristics, the defect type and size will be accurately determined. When outputting the inspection report, the tin-rich phase interference rejection rate, specific defect parameters, and inspection reliability will be highlighted.

[0050] Example 3 The specifications of the copper-clad ceramic plate tested in this implementation are: aluminum nitride ceramic base thickness 0.3mm and Cu copper cladding layer thickness 0.1mm.

[0051] Equipment preparation: A miniature high-frequency ultrasonic probe (C-scan and T-scan probe diameter 6mm, frequency 40MHz) is used to inspect ultra-thin copper-clad ceramic boards, while improving the ability to distinguish subtle features of micro-defects and tin-rich phase inclusions; the ultrasonic acquisition card has 12 bits, and the coupling agent is atomized ultrasonic coupling agent; the probe pressure is controlled within 0.1MPa, the scanning step is 0.2mm, the scanning range covers the entire surface of the copper-clad ceramic board, and the focus is on the interior of the copper layer (0-0.1mm depth, the concentrated area of ​​tin-rich phase inclusions).

[0052] Step S1: When collecting raw ultrasonic echo data, reduce the ultrasonic transmission power (to avoid damaging the ultra-thin product with excessive power), and monitor the coupling status simultaneously. Set the CQI qualification threshold to ≥65. When calculating CQI, collect the amplitude trough value using amp_tmp (the echo amplitude of the ultra-thin product is relatively low, and there are slight differences in the trough values ​​of tin-rich phase inclusions and micro-defects, which makes it easy to distinguish). Retain echo data with CQI ≥65.

[0053] Step S2: The interference feature database adds a "Template for Interference Features of Tin-Rich Phase Inclusions in Copper-Clad Layers" (collects ultrasonic signals of tin-rich phase inclusions inside the copper-clad layer of defect-free ultra-thin copper-clad ceramic boards, extracts its core features such as wavelet decomposition coefficients and amplitude gradients, and compares them with microcrack and micropore signals to clarify the differences). It also includes conventional interference features such as ultra-thin ceramic layer stress and copper-clad layer thinning. During the initial filtering, the temporal consistency judgment focuses on analyzing the amplitude gradient of the echo signal (the allowable deviation range is ≤0.02V). During interference feature matching, the wavelet decomposition coefficients are extracted as core features to distinguish the signal differences between tin-rich phase inclusions and micro-defects. The similarity threshold is set to 82% to accurately exclude tin-rich phase inclusion interference and extract suspected defect echo data.

[0054] In step S3, during imaging processing, the imaging algorithm is optimized to reduce noise interference. The C-scan image resolution is 0.03mm×0.03mm, capturing the planar features of micro-defects (micro-crack width ≥0.01mm, pore diameter ≥0.01mm) inside the copper-clad layer. The T-scan image resolution is 0.03mm, clearly presenting the hierarchical structure of the copper-clad layer and ceramic layer, highlighting the depth, location, and size distribution of micro-defects.

[0055] Step S4: Multimodal fusion is performed using the Transformer model. The shared encoder is optimized to the lightweight SwinTransformer to reduce computation and improve the ability to distinguish features between micro-defects and tin-rich phase mixtures. During soft-aligned attention fusion, local feature weights are increased to enhance the subtle features of micro-defects. The fused image undergoes edge enhancement processing to make the micro-defect features clearer and outputs multimodal fusion feature data (including the size, depth, and number of defects).

[0056] Figure 4The image superposition results in Example 3 show a partial result. The C-scan image (top left) and T-scan image (top right) both display defect features (the white portion in the upper half of the top left image and the gray portion in the upper half of the top right image). After superimposing the Transformer model images, the superimposed image (bottom right) shows the defect features (the bright yellow portion in the upper half), which is identified as a real defect. However, when the T-scan image is superimposed using the Transformer model image alone, the superimposed image (bottom right) does not display defect features, indicating that a standalone T-scan image cannot distinguish between real and interfering defects.

[0057] Figure 5 The image superposition results in Example 3 show that the C-scan image (top left) does not show defect features, while the T-scan image (top right) shows defect features (the gray area in the upper half). After superimposing the Transformer model images, the superimposed image (bottom left) does not show defect features, indicating a misjudgment due to interference. In this example, the defect features in the T-scan image show continuity, indicating that the interference feature is tin-rich phase interference. In other examples, if the defect features in the T-scan image show discreteness, it would be considered interference other than tin-rich phase. Furthermore, superimposing the T-scan image using the Transformer model alone does not show defect features in the superimposed image (bottom right), demonstrating that a standalone T-scan image cannot distinguish between real defects and interfering defects.

[0058] Step S5: During model training, a mixed sample of "tin-rich phase inclusion interference within the copper cladding layer + micro-defects" is added (tin-rich phase interference samples account for 50%). A lightweight convolutional layer is added to the secondary CNN model to avoid overfitting. Simultaneously, pure tin-rich phase samples, pure micro-defect samples, and tin impurity samples are added for comparative training to enhance the model's interference rejection capability. During defect identification and analysis, the defect size calculation accuracy is improved to 0.01mm, and micro-defects can be accurately identified. When outputting the inspection report, the tin-rich phase interference rejection rate (≥99%), specific parameters of micro-defects, and their impact on product reliability are indicated, providing a basis for product quality assessment. Statistical analysis of the inspection results from the above embodiments and existing methods (without introducing a coupled quality judgment process) is presented in Table 1.

[0059] Table 1. Statistical Table of Detection Results for Examples and Existing Methods

[0060] As shown in Table 1, the present invention can realize non-destructive testing of ultra-thin copper-clad ceramic plates, with a tin-rich phase interference elimination rate of ≥99%, a defect detection rate of ≥97%, and no product damage during the testing process. It is suitable for the micro-defect detection requirements of ultra-thin copper-clad ceramic plates and effectively avoids misjudgment caused by tin-rich phase interference.

[0061] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. The understanding of this specification should be based on those skilled in the art. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still make modifications or equivalent substitutions to the present invention. All technical solutions and improvements that do not depart from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal fusion defect detection method based on ultrasonic coupling quality judgment, characterized in that, Includes the following steps: S1. Collect raw ultrasonic echo data of the product and continuously monitor the coupling state between the ultrasonic probe and the surface of the product to be tested, and retain ultrasonic echo data that meet the coupling state. S2. Perform primary filtering on the ultrasonic echo data retained in step S1 to eliminate interference signals and obtain suspected defect echo data. S3. The suspected defect echo data in step S2 is processed by imaging to generate C-scan and T-scan images of different levels inside the product. S4. The C-scan image and T-scan image generated in step S3 are superimposed and fused using an adaptive weighted fusion combined with a soft alignment attention multimodal fusion algorithm; pixel-level alignment of the dual-modal images is achieved based on the outline of the product to be detected and the scanning coordinate system; combined with the suspected defect echo data extracted in step S2, feature enhancement and extraction are performed on the fused image to obtain multimodal fusion feature data; S5. Based on the multimodal fusion feature data, perform model training and defect identification and analysis to determine the location, type and size of the defect, and output the final defect detection result.

2. The multimodal fusion defect detection method based on ultrasonic coupling quality judgment as described in claim 1, characterized in that, In step S1, the coupling status is monitored through the following steps: real-time analysis of the initial pulse characteristics and the first bottom echo characteristics of the C-scan ultrasound signal, and obtaining a quantified coupling quality index (CQI) based on the initial pulse characteristics and the first bottom echo characteristics; setting a CQI threshold, if the current CQI is lower than the threshold, marking the current frame data as invalid and prompting for adjustment of the coupling status; if the current CQI is higher than the threshold, retaining the ultrasound echo data and the corresponding CQI value.

3. The multimodal fusion defect detection method based on ultrasonic coupling quality judgment as described in claim 2, characterized in that, The Coupling Quality Index (CQI) is calculated as follows: First, calculate the peak value of the ultrasonic echo amplitude: amp = gain × amp_tmp, where amp is the peak-to-peak value of the ultrasonic echo amplitude, gain is the soft gain coefficient, and amp_tmp is the peak or trough value of the amplitude obtained from the acquisition. Then determine the input range DIR based on the number of data acquisition cards: When amp > 0, DIR = 32767 ÷ 2^(16 - dataBitSize); When amp≤0, DIR=−32768÷2^(16-dataBitSize); Where dataBitSize is the number of bits in the acquisition card, and 32767 and -32768 are the standard quantization extreme values ​​of the 16-bit acquisition system; Finally, the coupling quality index is calculated: CQI = 254 × amp ÷ DIR. The value of CQI ranges from 0 to 254. The larger the value, the better the coupling quality. CQI ≥ 60 is set as the coupling quality is qualified.

4. The multimodal fusion defect detection method based on ultrasonic coupling quality judgment as described in claim 1, characterized in that, In step S2, the primary filtering includes: S21. Retrieve the preset interference feature database, match the ultrasonic echo data retained in step S1 with the interference features, and determine whether the ultrasonic echo data is an interference signal. S22. Perform a time-domain consistency check on the ultrasonic echo data retained in step S1. Ultrasonic echo data that simultaneously meets the following conditions is output as suspected defect echo data: it is not identified as an interference signal in step S21, and it satisfies time-domain consistency in step S22.

5. The multimodal fusion defect detection method based on ultrasonic coupling quality judgment as described in claim 4, characterized in that, The interference feature database is constructed through the following steps: interference signals under different operating conditions are collected and saved using a single-point acquisition method; surface tracking, peak calculation, Fourier transform, and wavelet transform are performed on the interference signals in sequence to separate and extract interference signal fingerprint features, including time-domain waveform, spectral centroid, bandwidth, and wavelet decomposition coefficients, and these features are classified and stored according to operating condition type to construct the interference feature database; the operating condition type includes surface roughness, material, and specific structural parts.

6. The multimodal fusion defect detection method based on ultrasonic coupling quality judgment as described in claim 4, characterized in that, The interference feature matching step in step S21 is as follows: extract the time domain, frequency domain, and wavelet features of the corresponding ultrasound echo signal from the ultrasound echo data retained in step S1, and perform similarity calculation with the feature templates in the interference feature database; set a similarity threshold, and if the similarity reaches the similarity threshold, it is determined to be an interference signal.

7. The multimodal fusion defect detection method based on ultrasonic coupling quality judgment as described in claim 1, characterized in that, The time-domain consistency judgment condition is: analyze the time-domain performance of ultrasound echo data on C-scan and T-scan probes, and determine whether the ultrasound echo signal has a ±1us delay, amplitude gradient and time axis fluctuation stability within 0.005us within the probe focusing range; if it does not have these conditions, it is judged as an interference signal.

8. The multimodal fusion defect detection method based on ultrasonic coupling quality judgment as described in claim 1, characterized in that, In step S4, the C-scan and T-scan images at different levels are superimposed using a Stack model or a Transformer model. The image superposition process includes: S41. Construction of dual-stream coding layer: The encoder-decoder soft-aligned attention paradigm is adopted to construct a dual-stream parallel coding structure. The shared encoder is built based on SwinTransformer, and the private encoder adopts a dual-branch independent CNN structure to adapt to the feature extraction requirements of C-scan and T-scan images respectively. S42. Global and Local Feature Extraction: Global context and spatial contour features shared by C-scan and T-scan images are extracted synchronously through a shared encoder; local detail features unique to C-scan images and hierarchical structure features unique to T-scan images are extracted through two independent branches of a private encoder, preserving the differential information of the dual-modal images. S43. Soft-aligned attention feature fusion: Define Query(Q) as the shared feature Fshare, and Key(K) and Value(V) as the concatenated result of private features [FC, FT]; calculate the soft-aligned attention weights and fused features: , The fusion feature is calculated as Ffusion = Fshare + Attention(Q, K, V). S44. Image Reconstruction and Output: The fused features Ffusion are input into the transposed convolutional decoder and upsampled to the original size; after 1×1 convolution dimensionality reduction, the superimposed image Ifinal is output, which is inversely normalized to restore the original pixel value range, and the multimodal fused feature data is output at the same time.

9. The multimodal fusion defect detection method based on ultrasonic coupling quality judgment as described in claim 1, characterized in that, In step S5, the multimodal fusion feature training process includes: S51. Determine training data and test data: Use the multimodal fusion feature data and the corresponding gray-white area of ​​the superimposed fusion image output in step S4 as training data, and the black area as test data. S52. First-level model training and new feature construction: The training data is input into the logistic regression base model for training. The pixel values ​​of the C-scan and T-scan images are flattened into one-dimensional feature vectors. The L2 regularization intensity is selected as a hyperparameter, and K-fold cross-validation is used to evaluate the model performance and optimize the hyperparameters. The calculation formula for K-fold cross-validation is as follows: , Where K represents the number of non-overlapping folds into which the training dataset is divided. This represents the binary cross-entropy loss value calculated by the model on the validation set when the i-th fold of data is used as the validation set. This represents the average loss value of K-fold cross-validation, used to quantify the model's generalization ability; After training is complete, the defect probability predictions generated by the logistic regression model on the training data are collected as a new feature dataset. S53, Second-level model training: From the original superimposed and fused images corresponding to the training data, based on the defect center, 128×128 pixel defect image details are extracted as input data. A CNN convolutional neural network is used for training, and L2 regularization intensity is selected as a hyperparameter. Dropout layer and batch normalization layer are added to the convolutional layer, and L2 regularization is added to the fully connected layer to suppress overfitting.

10. A multimodal fusion defect detection system based on ultrasonic coupling quality judgment, used to implement the detection method as described in any one of claims 1-9, characterized in that, include The ultrasonic acquisition and control unit is used to acquire raw ultrasonic echo data of the product according to the product formula; The coupling status monitoring unit is used to monitor and evaluate the coupling status between the ultrasonic probe and the product under test in real time, and output the coupling quality index. Interference feature database unit, used to construct interference feature database; The intelligent decision unit is connected to the ultrasonic acquisition control unit, the coupling state monitoring unit, and the interference feature database unit, respectively, and is used to determine the coupling state based on the coupling quality index and to determine whether the original ultrasonic echo data is interference data based on the interference feature database. The image generation unit is used to generate high-precision C-scan and T-scan images of different levels inside the product; The image overlay unit is used to overlay the generated C-scan and T-scan images into CT scan images; Image data training unit, used for data training on the generated C-scan, T-scan, and T-scan images; The defect detection and processing unit is used for defect identification and analysis, determining the location, type, and size of defects, and outputting the final defect detection results.

Images