Ultrasonic fin brazing recognition method based on improved Mask2Former network

Abstract

The present invention discloses an ultrasonic fin brazing identification method based on an improved Mask2Former network, comprising the following steps: Step 1: segmenting the fin welding area in the ultrasonic image using the improved Mask2Former network; Step 2: optimizing the matching of the welding area template and the fin welding area by a differential evolution algorithm, minimizing the intersection-over-union ratio of the welding area template and the fin welding area, and obtaining the optimal matching parameters including the welding area template rotation angle and the horizontal and vertical coordinate translation distance; Step 3: performing spatial transformation on the optimal matching parameters obtained by synchronously applying the theoretical welding area template and the inherent void structure template; Step 4: calculating the quotient of the number of pixels exceeding the defect threshold within the welding area template and outside the void template and the difference between the number of pixels of the two templates based on the transformed template position, and obtaining the brazing rate. Even when the void outline is invisible, the void position can be accurately located, and its interference can be eliminated when calculating the brazing rate, thereby improving the calculation accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of ultrasonic non-destructive testing, and specifically provides an ultrasonic fin brazing identification method based on an improved Mask2Former network. Background Art

[0002] Fins are extended surface structures typically made of highly thermally conductive metals such as aluminum and copper. Attaching fins to a substrate significantly increases the heat exchange area, thereby improving heat dissipation efficiency. Consequently, they are widely used in heat dissipation applications such as batteries, chips, and data centers. The brazing rate is a key metric in evaluating fin welding quality. This is defined as the ratio of the actual brazing area to the theoretical contact area. The industry typically requires a brazing rate of at least 80% to ensure good thermal conductivity and structural reliability.

[0003] However, in actual production, due to the inherent non-weldable void areas in the fin structure, these areas are filled in the inherent void areas due to solder overflow, resulting in unclear contours in the ultrasonic image and easy confusion with the welding area, which brings difficulties to the calculation of the brazing rate. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an ultrasonic fin brazing recognition method based on an improved Mask2Former network, which can accurately locate the cavity position even when the cavity outline is not visible in the ultrasonic image, eliminate its interference when calculating the brazing rate, and improve the calculation accuracy of the brazing rate.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] An ultrasonic fin brazing recognition method based on an improved Mask2Former network comprises the following steps:

[0007] Step 1: Use the improved Mask2Former network to segment the fin welding area in the ultrasonic image, including:

[0008] 11) Extracting multi-scale features from the input fin ultrasonic image through a feature extraction network;

[0009] 12) The multi-scale features are used to obtain the selected mask features and selected category information through the query vector selection module, and are used as the query vector of the Transformer decoder;

[0010] 13) During the training of the Transformer decoder, a noise module is introduced. The noise module converts the category of the labeled object into another category among all detection categories with a set probability and encodes it with a learnable word embedding vector. The noise module also adds noise to the mask of the labeled object through random transformation with a set probability.

[0011] 14) Samples with an IoU ratio of the noise mask to the true mask greater than or equal to 0.5 are considered positive samples, and samples with an IoU ratio less than 0.5 are considered negative samples;

[0012] 15) The positive samples are input into the Transformer decoder for denoising training to reconstruct the true mask and obtain the labeled target, and the negative samples are fitted to the empty set;

[0013] Step 2: Optimize the matching between the fin welding area template and the fin welding area obtained in step 1 by using the differential evolution algorithm to minimize the intersection-over-union ratio between the fin welding area template and the fin welding area predicted by the network, and obtain the optimal matching parameters including the rotation angle, horizontal coordinate translation distance, and vertical coordinate translation distance of the fin welding area template;

[0014] Step 3: The theoretical welding area template and the template of the inherent void structure in the fin welding area are simultaneously subjected to spatial transformation using the optimal matching parameters obtained in step 2;

[0015] Step 4: Based on the transformed template position, calculate the quotient of the number of pixels below the defect threshold within the fin welding area template and outside the void template and the difference between the number of pixels of the two templates to obtain the brazing rate.

[0016] Furthermore, in step 12), the method for obtaining the selected mask feature and the selected category feature by querying the vector selection module is:

[0017] Extract multi-scale features from the input fin ultrasonic image through the feature extraction network , multi-scale features After 1×1 convolution, the dimension is reduced to a unified channel dimension to obtain multi-scale features. ;

[0018] Multi-scale features Obtaining mask features through convolution , in multi-scale features Generate a mask feature for each position ; Mask feature features After flattening and splicing operations, the mask features are obtained ;

[0019] Multi-scale features Get categorical features by flattening and concatenating ;

[0020] The categorical features Output category information through multi-layer perceptron and Softmax layer ;

[0021] Category information and mask features Apply Varifocal loss function; for category information Each row in the table calculates the maximum value along the category dimension and obtains the maximum value vector , sort the maximum value vector in descending order, select the original row index corresponding to the first K maximum values ​​to form the output index set ; From category information Select index collection The corresponding rows constitute the selection category information , from the mask feature Select index collection The corresponding rows constitute the selection mask feature ; will select category information and mask features As part of the query vector for the Transformer decoder.

[0022] Furthermore, in step 13), at least one perturbation selected from translation, scaling, rotation, elastic deformation, or Gaussian noise is applied to the mask of the labeled object to generate a noisy mask.

[0023] Furthermore, the noisy mask feature output by the noisy module and the noisy category feature are concatenated to form a noisy query vector, and the selected mask feature output by the query vector selection module and the selected category information are concatenated to form a selected query vector;

[0024] In the Transformer decoder, an attention matrix is ​​added between the self-attention Transformer transformation, and the values ​​of the selection query vector and the selection query vector area of ​​the attention matrix are set to 0, the values ​​of the noisy query vector and the noisy query vector area of ​​the attention matrix are set to 0, and the values ​​of the noisy query vector and the selection query vector area of ​​the attention matrix are set to negative infinity; so that when performing the Transformer transformation, the selection query vector is only calculated with the selection query vector, the noisy query vector is only calculated with the noisy query vector, and the noisy query vector is not calculated with the selection query vector.

[0025] The beneficial effects of the present invention are:

[0026] The ultrasonic fin brazing recognition method based on the improved Mask2Former network has the following advantages:

[0027] (1) In the application of non-AI pure template matching technology, when the fins are tilted or offset significantly, traditional template matching technology will cause matching errors. However, the method of the present invention uses AI to ignore the effects of rotation and translation, and can achieve pixel-level detection even if the offset and translation are large;

[0028] (2) When there are inherent non-weldable void areas in the fin structure, and the void areas are not clearly visible in the ultrasonic image due to solder filling, and are confused with the welded areas, which makes it difficult to calculate the brazing rate, the method of the present invention can still accurately find the invisible non-weldable void areas on the ultrasonic image, thereby accurately calculating the fin brazing rate;

[0029] (3) The randomly initialized queries in the original Mask2Former lack spatial priors, resulting in unstable training and slow convergence. To this end, the proposed method uses a query vector selection module to obtain selected mask features and selected category information, and uses them as query vectors for position queries and content queries in the Transformer decoder, effectively improving positioning accuracy and training stability.

[0030] (4) To address the initial instability of Mask2Former’s DETR-based bipartite graph matching, the proposed method further introduces a mask-noising training mechanism through a noise-adding module. By using a noisy mask to guide the decoder to reconstruct the true mask, the consistency between the query and the target is significantly enhanced, the convergence speed is accelerated, and the matching accuracy is improved.

[0031] In summary, the ultrasonic fin brazing recognition method based on the improved Mask2Former network in the present invention can accurately locate the cavity position even when the cavity outline is invisible in the ultrasonic image, eliminate its interference when calculating the brazing rate, and improve the calculation accuracy of the brazing rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to make the purpose, technical solutions and beneficial effects of the present invention more clear, the present invention provides the following drawings for illustration:

[0033] Figure 1 To improve the structural principle diagram of Mask2Former network;

[0034] Figure 2 Schematic diagram of the module for query vector selection;

[0035] Figure 3 Schematic diagram of the theoretical welding area template;

[0036] Figure 4 is a schematic diagram of the test results;

[0037] Figure 5 This is the original image of the fin.

[0038] In the diagram of the detection results, the red area is the detected defect, the light green small block is the void and the undetected area, and the light blue area is the welding area detected by the improved Mask2Former model. DETAILED DESCRIPTION

[0039] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0040] This embodiment is based on an improved Mask2Former network ultrasonic fin brazing recognition method, which includes the following steps.

[0041] Step 1: Use the improved Mask2Former network to segment the fin welding area in the ultrasonic image.

[0042] Specifically, such as Figure 1 As shown, this embodiment mainly improves the query vector based on Mask2Former. The input fin ultrasonic image is passed through the feature extraction network to obtain multi-scale features. :

[0043]

[0044] , ,

[0045] in: 、 and Represent different resolution features respectively. and Represent the height and width of the network input resolution respectively.

[0046] Multi-scale features The query vector selection module generates the selected mask features and selected category features for the Transformer decoder input. The denoising module is used to add noise to the labeled objects, allowing the model's Transformer decoder to output true values ​​based on the input noisy labeled objects, thereby assisting model learning and reducing the difficulty of learning bipartite graph matching. The pixel decoder is used to fuse the multi-scale features output by the feature extraction network and upsample them to a higher resolution to obtain the final features used for segmentation. The Transformer decoder performs feature query and selection based on the input query vector and multi-scale image features, outputting predictions for N category-mask pairs.

[0047] Existing Mask2Former query vectors are all learnable and initialized with zero parameters or randomly. This presents the following issues: First, because the initial query vectors are randomly or zero-initialized, the model struggles to align image regions in the early stages, often experiencing oscillations, exploding gradients, or vanishing gradients. This leads to unstable training and widely varying results. Second, the initial query vectors lack explicit spatial priors. Static query vectors treat all pixels equally, making it difficult to quickly locate the center of an object. Zero-initialized query vectors lack any positional information, requiring learning to assign image regions to each query vector. This lack of spatial information guides the attention mechanism, resulting in almost equal coverage of the entire image in the early stages, resulting in low efficiency.

[0048] This embodiment addresses the above shortcomings of Mask2Former and improves the query vector generation method of Mask2Former. Based on the multi-scale features generated by the Mask2Former encoder, the network is used to directly predict the target mask and category prediction. Then, the Varifocal loss function is used to select the frontier with high category scores and high intersection over union (IoU) between the predicted mask and the true mask. The mask and the corresponding category information are used as a rough candidate region and as the Transformer decoder The data-driven candidate regions replace the “blindly initialized” query vectors, providing accurate spatial and content information to the Transformer decoder.

[0049] Specifically, the method for segmenting the fin welding area in the ultrasonic image using the improved Mask2Former network includes the following steps.

[0050] 11) Extract multi-scale features from the input fin ultrasonic image through the feature extraction network Specifically, multi-scale features Expressed as:

[0051]

[0052] , ,

[0053] in: 、 and Represent multi-scale features Different resolution features in and Represent the height and width of the network input resolution respectively.

[0054] 12) Multi-scale features The selection mask features and selection category information are obtained through the query vector selection module and used as the query vector of the Transformer decoder.

[0055] like Figure 2 As shown, the method for obtaining the selection mask feature and the selection category feature through the query vector selection module is as follows.

[0056] Extract multi-scale features from the input fin ultrasonic image through the feature extraction network , multi-scale features After 1×1 convolution, the dimension is reduced to a unified channel dimension of 256 to obtain multi-scale features. :

[0057]

[0058] , ,

[0059] in: 、 and Representing multi-scale features Different resolution features in .

[0060] Multi-scale features Obtaining mask features through convolution , to multi-scale features Generate a mask feature for each position , mask features Expressed as:

[0061]

[0062] , ,

[0063] in: Expressed as ; 、 and Represents mask features Different resolution features in .

[0064] The purpose of this operation is to generate a mask feature for each position on the multi-scale feature map. The mask feature does not need to be very accurate. At the same time, in order to balance the inference speed, the mask feature dimension is designed to be consistent with the minimum resolution of the multi-scale feature.

[0065] Mask features After flattening and splicing operations, the mask features are obtained :

[0066]

[0067]

[0068] in: Indicates the number of queries.

[0069] A similar method can be used to obtain category features. Specifically, the multi-scale features Get categorical features by flattening and concatenating ; The category features Output category information through multi-layer perceptron and Softmax layer ;in: Indicates the number of categories to detect.

[0070] The number of all candidate query vectors Too many, choose the most representative For inference of the Transformer decoder, in this embodiment, Select 300. When selecting the most representative query vector, the category information predicted by the network is High score and mask features The high IoU between the real mask and the real mask is connected. We filter based on the category scores. For each category, we first find the category with the highest score as its predicted category. We then select the 300 best category features with the highest scores across all predicted categories and obtain the top 300 indexes. We also use the varifocal loss function to ensure that the mask features corresponding to the top 300 indexes of category features with high category scores also have a high IoU with the ground-truth mask. Query vectors with both high category scores and accurate masks are considered good query vectors.

[0071] In this embodiment, the category information and mask features Apply Varifocal loss function; for category information Each row in the table calculates the maximum value along the category dimension and obtains the maximum value vector , sort the maximum value vector in descending order, select the original row index corresponding to the first K maximum values ​​to form the output index set ; From category information Select index collection The corresponding rows constitute the selection category information , from the mask feature Select index collection The corresponding rows constitute the selection mask feature ; will select category information and mask features As part of the query vector for the Transformer decoder.

[0072] 13) During the training of the Transformer decoder, a noise module is introduced. The noise module converts the category of the labeled object to another category among all detected categories with a set probability and encodes it using a learnable word embedding vector. The noise module also adds noise to the mask of the labeled object through random transformation with a set probability. In this embodiment, the mask of the labeled object is subjected to at least one perturbation selected from translation, scaling, rotation, elastic deformation, or Gaussian noise to generate the noisy mask.

[0073] Mask2Former follows the DETR ensemble prediction paradigm. During training, the predicted mask targets for a fixed number of query vectors output by the model are paired with the true mask targets through bipartite graph matching. Classification and masking losses are then calculated for the matching results. Due to the discrete nature of Hungarian matching and the stochastic nature of model training, matching predicted targets with true targets becomes a dynamic and unstable process. Within different training cycles, the same predicted query vector often matches different true targets, meaning its matching targets frequently switch, especially in the early stages of training. This creates ambiguity in model optimization, making optimization difficult and unstable, ultimately resulting in slow convergence. Therefore, this embodiment introduces a target denoising process during Mask2Former training. Part of the query vector input to the Transformer decoder is replaced with a target that has been noisyed, and the output is a reconstruction of the original labeled target. This allows the model to predict the true target based on the noisy target, thereby facilitating model learning and reducing learning difficulty. The learning objective is clear: the input from which the labeled target is noisy will be used to predict the corresponding labeled target, thus avoiding the ambiguity inherent in Hungarian matching.

[0074] This embodiment introduces a target denoising process during Mask2Former training. Part of the query vector input to the Transformer decoder is replaced with a noisy labeled target, and the output reconstructs the original labeled target. This allows the model to predict the true target based on the noisy target, thereby facilitating learning and reducing the learning difficulty. The learning objective is clear: the input derived from a specific labeled target will be used to predict the corresponding labeled target, thus avoiding the ambiguity inherent in Hungarian matching.

[0075] Specifically, a labeled object contains two parts: the category and the mask. Noise is applied to each of these two parts. When noising the category, it is converted to one of the detected categories with a certain probability, and then encoded into a 256-dimensional vector using a learnable word embedding. When noising the ground truth mask, various random transformations are applied with certain probabilities. The transformations are shown below.

[0076] (1) Translation: translate the foreground area in random directions of the horizontal and vertical coordinates.

[0077] (2) Scaling: Randomly scale the mask within a certain range to change the length and width of the minimum bounding rectangle of the mask area.

[0078] (3) Rotation: Rotate the mask by a random angle.

[0079] (4) Elastic deformation: Apply the global elastic distortion transformation ElasticTransform and the grid local control elastic distortion GridElasticDeform to the mask area.

[0080] (5) Gaussian noise: Add noise to the mask to produce random jagged edges or small holes.

[0081] The above transformations are applied sequentially to the same true mask with a certain probability to add noise.

[0082] 14) Samples with an IoU ratio of the noise mask to the true mask greater than or equal to 0.5 are considered positive samples, and samples with an IoU ratio less than 0.5 are considered negative samples.

[0083] Specifically, in order to guide the model to restore accurate predictions under slight perturbations, and to learn to reject overly offset prediction masks, the noisy targets are divided into positive samples and negative samples, thereby accelerating convergence, stabilizing bipartite graph matching and suppressing repeated detection. In this embodiment, the noisy masks with an IoU greater than or equal to 0.5 with the true mask are regarded as positive samples, and those less than or equal to 0.5 are regarded as negative samples. Adding noise to the category does not affect the positive or negative nature of the sample, and the positive and negative samples are only determined by the degree of noise added to the mask. When adding noise, 100 positive and negative samples are generated in the same training batch, and the total noisy query vectors are 200. The positive samples are used to input into the Transformer decoder of the network for denoising to obtain labeled targets, while the negative samples make the network fit into the empty set ∅ "no target".

[0084] 15) The positive samples are input into the Transformer decoder for denoising training to reconstruct the true mask and obtain the labeled target, and the negative samples are fitted as an empty set.

[0085] In this embodiment, the noisy mask feature output by the noisy module and the noisy category feature are concatenated to form a noisy query vector, and the selected mask feature output by the query vector selection module and the selected category information are concatenated to form a selected query vector.

[0086] During training, the query vector input to the Transformer decoder is composed of 300 selection query vectors and 200 noisy query vectors. In order to prevent the selection query vector from seeing the labeled target information in the noisy query vector during training, an additional attention matrix is ​​added between the self-attention Transformer transformation of the decoder. By setting the value of the selection query vector and the selection query vector area of ​​the attention matrix to 0, the value of the noisy query vector and the noisy query vector area of ​​the attention matrix to 0, and the value of the noisy query vector and the selection query vector area of ​​the attention matrix to negative infinity, the selection query vector only performs attention calculation with the selection query vector, the noisy query vector only performs attention calculation with the noisy query vector, and the noisy query vector does not perform attention calculation with the selection query vector during the Transformer transformation. In this way, information leakage can be prevented, and the selection query vector for the bipartite graph matching part can be prevented from seeing the labeled target information in the denoised query vector for the denoising part, resulting in the learning of shortcut solutions and learning failure. In the testing phase, since there is no labeled target information for inference, the input query vector of the network only uses the selection query vector, and the noisy query vector is only used to stabilize the training process, such as Figure 1 shown.

[0087] Step 2: Optimize the matching of the fin welding area template and the fin welding area obtained in step 1 through the differential evolution algorithm, so that the intersection ratio of the fin welding area template and the fin welding area is minimized, and the optimal matching parameters including the rotation angle of the fin welding area template, the horizontal coordinate translation distance, and the vertical coordinate translation distance are obtained.

[0088] When calculating the brazing rate of the fin, first use the method described in step 1 to detect the welding area of ​​the fin, and then use the template of the welding area (such as Figure 3 The template (shown in Figure 1) is then matched with the weld area detected by the method described in step 1 to achieve the minimum IoU between the two. The template's rotation angle and translation distance in the horizontal and vertical coordinates are then determined. During template matching, this embodiment applies a differential evolution algorithm for optimization. During the optimization process, the population is set to 10 individuals, each with 3 dimensions, mapped to an offset in the horizontal and vertical coordinates, and a rotation angle. The fitness function is expressed as the IoU between the weld area template and the detected weld area template. The optimal population obtained after the differential evolution algorithm is the optimal translation distance and rotation angle.

[0089] Step 3: The theoretical weld area template and the inherent void structure template are spatially transformed using the optimal matching parameters obtained in Step 2. This allows the inherent void structure and outline to be identified on the ultrasound image based on the rotational and translational position of the inherent void template, even though it is not visible on the ultrasound image.

[0090] Step 4: Based on the transformed template position, calculate the quotient of the number of pixels below the defect threshold within the fin welding area template and outside the void template and the difference between the number of pixels of the two templates, and obtain the brazing rate of 89.2%, as shown in Figure 4 and Figure 5 shown.

[0091] The above embodiments are merely preferred embodiments for the purpose of fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the claims.

Claims

1. A method for ultrasonic fin brazing identification based on an improved Mask2Former network, characterized by: The steps include: Step 1: Use the improved Mask2Former network to segment the fin welding area in the ultrasonic image, including: 11) Extracting multi-scale features from the input fin ultrasonic image through a feature extraction network; 12) The multi-scale features are used to obtain the selected mask features and selected category information through the query vector selection module, and are used as the query vector of the Transformer decoder; 13) During the training of the Transformer decoder, a noise module is introduced. The noise module converts the category of the labeled object into another category among all detection categories with a set probability and encodes it with a learnable word embedding vector. The noise module also adds noise to the mask of the labeled object through random transformation with a set probability. 14) Samples with an IoU ratio of the noise mask to the true mask greater than or equal to 0.5 are considered positive samples, and samples with an IoU ratio less than 0.5 are considered negative samples; 15) The positive samples are input into the Transformer decoder for denoising training to reconstruct the true mask and obtain the labeled target, and the negative samples are fitted to the empty set; Step 2: Optimize the matching between the fin welding area template and the fin welding area obtained in step 1 by using the differential evolution algorithm to minimize the intersection-over-union ratio between the fin welding area template and the fin welding area predicted by the network, and obtain the optimal matching parameters including the rotation angle, horizontal coordinate translation distance, and vertical coordinate translation distance of the fin welding area template; Step 3: The theoretical welding area template and the template of the inherent void structure in the fin welding area are simultaneously subjected to spatial transformation using the optimal matching parameters obtained in step 2; Step 4: Based on the transformed template position, the quotient of the number of pixels below the defect threshold within the fin welding area template and outside the void template and the difference between the number of pixels of the two templates is calculated to obtain the brazing rate; In step 12), the method for obtaining the selected mask feature and the selected category feature by querying the vector selection module is: Extract multi-scale features from the input fin ultrasonic image through the feature extraction network , multi-scale features After 1×1 convolution, the dimension is reduced to a unified channel dimension to obtain multi-scale features. ; Multi-scale features Obtaining mask features through convolution , in multi-scale features Generate a mask feature for each position ; Mask features After flattening and splicing operations, the mask features are obtained ; Multi-scale features Get categorical features by flattening and concatenating ; The categorical features Output category information through multi-layer perceptron and Softmax layer ; Category information and mask features Apply Varifocal loss function; for category information Each row in the table calculates the maximum value along the category dimension and obtains the maximum value vector , sort the maximum value vector in descending order, select the original row index corresponding to the first K maximum values ​​to form the output index set ; From category information Select index collection The corresponding rows constitute the selection category information , from the mask feature Select index collection The corresponding rows constitute the selection mask feature ; will select category information and mask features As part of the query vector for the Transformer decoder.

2. The ultrasonic fin brazing identification method based on the improved Mask2Former network according to claim 1 is characterized in that: In step 13), at least one perturbation selected from translation, scaling, rotation, elastic deformation, or Gaussian noise is applied to the mask of the labeled object to generate a noisy mask.

3. The ultrasonic fin brazing identification method based on the improved Mask2Former network according to claim 1 is characterized in that: The noise mask feature output by the noise addition module and the noise category feature are concatenated to form a noise query vector, and the selection mask feature output by the query vector selection module and the selection category information are concatenated to form a selection query vector; In the Transformer decoder, an attention matrix is ​​added between the self-attention Transformer transformation, and the values ​​of the selection query vector and the selection query vector area of ​​the attention matrix are set to 0, the values ​​of the noisy query vector and the noisy query vector area of ​​the attention matrix are set to 0, and the values ​​of the noisy query vector and the selection query vector area of ​​the attention matrix are set to negative infinity; so that when performing the Transformer transformation, the selection query vector is only calculated with the selection query vector, the noisy query vector is only calculated with the noisy query vector, and the noisy query vector is not calculated with the selection query vector.

Images