A hypergraph contrastive learning based SMT solder joint detection method

Abstract

The application provides an SMT solder joint detection method based on hypergraph contrast learning, comprising obtaining an input image set, extracting a multi-modal feature set from the input image set. The multi-modal feature set is input into a conditional diffusion model, and a cross attention mechanism is used to fuse the features of multiple modalities to generate a solder joint sample set. Based on the multi-modal feature set and the solder joint sample set, a hypergraph structure is constructed, and hypergraph convolution is performed on the hypergraph structure to extract a hypergraph node feature set. A multi-level attention fusion mechanism is used to fuse the multi-modal feature set and the hypergraph node feature set, and based on the fused features, a student network is trained using knowledge distillation technology, and the student network is used to detect SMT solder joints. The solder joint sample set generated by the conditional diffusion model expands the training data and improves the generalization ability of the model. The hierarchical attention fusion mechanism can integrate multi-modal and structured information to provide more comprehensive and more robust feature representation, and the student network has the advantage of real-time detection of SMT solder joints.

Description

Technical Field

[0001] This invention relates to the field of SMT processing technology, and in particular to an SMT solder joint detection method based on hypergraph comparison learning. Background Technology

[0002] With the rapid development and technological innovation of the electronics manufacturing industry, surface mount technology (SMT) has become a core process technology in modern electronic product manufacturing. As a critical connection point between electronic components and printed circuit boards, the quality of SMT solder joints directly determines the electrical performance, mechanical reliability, and lifespan of the entire electronic product. Therefore, accurate, efficient, and real-time quality inspection of SMT solder joints has significant engineering and economic value.

[0003] Traditional SMT solder joint quality inspection methods mainly rely on manual visual inspection and machine vision systems based on preset rules. While manual inspection methods offer some flexibility, they suffer from inherent drawbacks such as low inspection efficiency, strong subjective judgment, operator fatigue, and inconsistent inspection standards, making them unsuitable for the demands of modern large-scale automated production. Although traditional machine vision methods improve inspection efficiency and objectivity to some extent, their inspection accuracy and system robustness remain significantly insufficient when facing complex and varied solder joint shapes, dynamically changing lighting conditions, and complex background interference in real-world industrial environments.

[0004] In recent years, deep learning technology has made significant progress in computer vision and image recognition, providing a new technical path for the development of SMT solder joint inspection technology. However, existing deep learning-based inspection methods still face many technical challenges and practical problems in SMT solder joint inspection applications.

[0005] First, data scarcity severely restricts the performance of deep learning models. Acquiring high-quality SMT solder joint defect samples is complex and costly, especially for rare defect types and complex defect combinations, where the number of samples is extremely limited, leading to insufficient training data and consequently affecting the model's generalization ability and detection accuracy. Second, limitations in feature representation capabilities restrict further improvements in detection performance. Traditional convolutional neural networks primarily focus on capturing local image features, making it difficult to effectively model the complex spatial relationships and global contextual information between SMT solder joints, and failing to fully utilize the structured characteristics of solder joint distribution to improve detection accuracy. Third, the imperfection of multimodal information fusion mechanisms affects the overall performance of the detection system. SMT solder joint detection involves sensor information from multiple modalities, such as visible light images, infrared thermal imaging, and 3D point cloud data. Existing methods lack effective multimodal data fusion strategies, failing to fully leverage the complementary advantages of different modalities. Fourthly, the contradiction between industrial real-time requirements and detection accuracy is becoming increasingly prominent. Modern electronic manufacturing production lines have extremely high requirements for detection speed, and need to achieve millisecond-level response speed while ensuring detection accuracy. However, existing deep learning models often have high computational complexity and long inference time, making it difficult to meet the real-time requirements of industrial applications.

[0006] Therefore, there is an urgent need for a detection method that can fully utilize multimodal sensor information, effectively model the correlation of complex spaces, possess strong generalization ability, and have the advantage of real-time detection of SMT solder joints. Summary of the Invention

[0007] To overcome the problems existing in related technologies, the purpose of this invention is to provide an SMT solder joint detection method based on hypergraph contrast learning. This method can make full use of multimodal sensor information, effectively model the correlation relationship of complex space, has strong generalization ability, and has the advantage of real-time detection of SMT solder joints.

[0008] A method for SMT solder joint detection based on hypergraph contrast learning includes:

[0009] Obtain an input image set, which includes visible light images, infrared images, and depth images;

[0010] A multimodal feature set is extracted from the input image set, the feature set including visible light features, infrared features and depth features;

[0011] The multimodal feature set is input into the conditional diffusion model, and the features of multiple modalities are fused using a cross-attention mechanism to generate a solder joint sample set.

[0012] A hypergraph structure is constructed based on the multimodal feature set and the solder joint sample set. Hypergraph convolution is performed on the hypergraph structure to extract the hypergraph node feature set.

[0013] A multi-level attention fusion mechanism is used to fuse the multimodal feature set and the hypergraph node feature set to obtain fused features;

[0014] Based on the fusion features, a student network is trained using knowledge distillation technology, and the student network is used to detect SMT solder joints.

[0015] In a preferred embodiment of the present invention, the step of inputting the multimodal feature set into a conditional diffusion model and fusing features from multiple modalities using a cross-attention mechanism to generate a solder joint sample set includes:

[0016] Construct a conditional diffusion model;

[0017] A cross-attention mechanism is used to fuse the t-th noisy data and the conditional control vector to obtain the t-th attention matrix;

[0018] The multimodal feature set and the t-th attention matrix are input into the conditional diffusion model, and the t-th reverse iteration is performed to obtain the (t-1)-th solder joint sample.

[0019] In a preferred embodiment of the present invention, the construction of the hypergraph structure based on the multimodal feature set and the solder joint sample set includes:

[0020] Multiple target solder joints that meet the connection conditions are selected to obtain a node set; the connection conditions include any one of the following: spatial distance is less than a distance threshold, they belong to the same solder joint type, and there is a causal relationship between the solder joints;

[0021] The paired target solder joints are connected using superedges to obtain a superedge set;

[0022] A weight matrix is ​​constructed based on the distance and similarity between the target solder joints;

[0023] The set of nodes, the set of hyperedges, and the weight matrix are combined to form a hypergraph structure.

[0024] In a preferred embodiment of the present invention, constructing a weight matrix based on the distance and similarity between the target solder joints includes:

[0025] Calculate the weight matrix using the following formula:

[0026] ;

[0027] Among them, W i,j Let f be the weight in the i-th row and j-th column of the weight matrix, softmax be the normalization exponential function, d be the feature dimension, and f be the weight in the i-th row and j-th column of the weight matrix.i For the i-th target solder joint, f j For the j-th target solder joint; sim(f i ,f j Let ) be the similarity function between the i-th target solder joint and the j-th target solder joint, exp be the exponential function, and d be the similarity function between them. i,j Let be the Euclidean distance between the i-th target solder joint and the j-th target solder joint. is the distance attenuation coefficient, and . represents the multiplication operation.

[0028] In a preferred embodiment of the present invention, the step of performing hypergraph convolution on the hypergraph structure to extract the hypergraph node feature set includes:

[0029] The hypergraph node feature set is extracted using the following formula:

[0030] ;

[0031] ;

[0032] in, For the (l+1)th hypergraph node features, For the features of the l-th hypergraph node, For the sigmoid function, D v Let D be the degree matrix of the nodes. e H is the hypermarginality matrix; H is the incidence matrix; W is the weight matrix. T It is the transpose of the incidence matrix. is the learnable parameter matrix of the l-th layer; Laplace represents the hypergraph Laplacian matrix; HyperConv is the hypergraph convolutional layer, which is used to iteratively update the hypergraph node features;

[0033] All the hypergraph node features are combined into a hypergraph node feature set.

[0034] In a preferred embodiment of the present invention, the step of fusing the multimodal feature set and the hypergraph node feature set using a multi-level attention fusion mechanism to obtain fused features includes:

[0035] Calculate the intramodal attention weights;

[0036] Calculate the intermodal attention weights;

[0037] Based on the intra-modal attention weights and inter-modal attention weights, the feature matrices of all solder joints in each modality are weighted and summed to obtain the fused features.

[0038] In a preferred embodiment of the present invention, training the student network using knowledge distillation technology based on the fusion features includes:

[0039] A teacher network is constructed, comprising a multimodal feature extraction module, a multimodal diffusion generation module, a hypergraph feature extraction module, and a multimodal feature fusion module;

[0040] Train the teacher network until it converges;

[0041] The soft labels of the teacher network are used to guide the training of the student network;

[0042] If the number of training iterations of the student network is greater than or equal to the training iteration threshold, then training is stopped.

[0043] In a preferred embodiment of the present invention, the step of using the soft labels of the teacher network to guide the training of the student network includes:

[0044] The student network was trained using the distillation loss function.

[0045] ;

[0046] Among them, L KD Let L be the distillation loss function, and balance be the balance coefficient. CE Let cross-entropy be the loss function. Here, KL represents the information divergence parameter, and temperature is the temperature parameter. For the sigmoid function, z s For the output of the student network, z t This is the output of the teacher network.

[0047] In a preferred embodiment of the present invention, the step of inputting the multimodal feature set and the t-th attention matrix into the conditional diffusion model, performing the t-th reverse iteration, and obtaining the (t-1)-th solder joint sample includes:

[0048] The following formula is used to calculate the (t-1)th solder joint sample:

[0049] ;

[0050] Where N is a Gaussian distribution, x t-1 For the (t-1)th solder joint sample, x t For the t-th solder joint sample, Let A be the probability distribution of changing from the t-th solder joint sample to the (t-1)-th solder joint sample; A is the attention weight. Let A be the mean value of the t-th solder joint sample at time t with attention weight A. Let be the variance of the t-th solder joint sample at time t. These are the weight parameters for the reverse denoising process.

[0051] In a preferred embodiment of the present invention, the step of extracting a multimodal feature set from the input image set includes:

[0052] The visible light features are extracted from the visible light image using a ResNet neural network;

[0053] Infrared features are extracted from the infrared image using a convolutional neural network.

[0054] A 3D convolutional network is used to extract depth features from the depth image.

[0055] The beneficial effects of this invention are as follows:

[0056] The SMT solder joint detection method based on hypergraph contrastive learning provided by this invention includes acquiring an input image set, which includes visible light images, infrared images, and depth images. A multimodal feature set is extracted from the input image set, including visible light features, infrared features, and depth features. Visible light features include visual features such as the geometric shape parameters, surface texture features, and color distribution statistics of the solder joints; infrared features include thermal information such as temperature distribution patterns and thermal conductivity parameters; and depth features include spatial geometric information such as the three-dimensional geometric structure, height distribution statistics, and surface roughness of the solder joints. The multimodal feature set is input into a conditional diffusion model, and a cross-attention mechanism is used to fuse the features of multiple modalities to generate a solder joint sample set. The solder joint sample set generated by the conditional diffusion model expands the training data and improves the model's generalization ability. The cross-attention mechanism enables the generated solder joint samples to carry multimodal information. A hypergraph structure is constructed based on the multimodal feature set and the solder joint sample set. Hypergraph convolution is performed on the hypergraph structure to extract the hypergraph node feature set. A hypergraph structure is constructed based on spatial distance, solder joint type similarity, and welding causal relationships. This captures high-order correlations between solder joints, thereby enhancing the discriminative power of the features. A multi-level attention fusion mechanism is employed to fuse multimodal feature sets and hypergraph node feature sets, resulting in fused features. This multi-level attention fusion mechanism integrates multimodal and structured information, providing more comprehensive and robust feature representations. Based on the fused features, a student network is trained using knowledge distillation technology to detect SMT solder joints. The teacher network includes a multimodal diffusion generation module and a hypergraph learning module. The distillation process allows the teacher network to transfer the learned multimodal data and relationships between data to the student network, which is lightweight. Using the student network to identify SMT solder joints fully utilizes multimodal sensor information, effectively models complex spatial relationships, possesses strong generalization capabilities, and offers the advantage of real-time SMT solder joint detection. Attached Figure Description

[0057] Figure 1 This is a flowchart of the SMT solder joint detection method based on hypergraph comparison learning of the present invention;

[0058] Figure 2 This is a flowchart of the present invention for constructing a hypergraph structure based on a multimodal feature set and a solder joint sample set;

[0059] Figure 3 This is a flowchart of the computational fusion features of the present invention. Detailed Implementation

[0060] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0061] Example 1

[0062] like Figure 1 As shown, this embodiment provides an SMT solder joint detection method based on hypergraph contrast learning, including:

[0063] S1: Obtain the input image set, which includes visible light images, infrared images, and depth images.

[0064] S2: Extract a multimodal feature set from the input image set, the feature set including visible light features, infrared features and depth features.

[0065] S3: Input the multimodal feature set into the conditional diffusion model, and use the cross-attention mechanism to fuse the features of multiple modalities to generate a solder joint sample set.

[0066] S4: Construct a hypergraph structure based on the multimodal feature set and the solder joint sample set, perform hypergraph convolution on the hypergraph structure, and extract the hypergraph node feature set.

[0067] S5: The multi-level attention fusion mechanism is used to fuse the multimodal feature set and the hypergraph node feature set to obtain the fused features.

[0068] S6: Based on the fusion features, a student network is trained using knowledge distillation technology, and the student network is used to detect SMT solder joints.

[0069] The step of inputting the multimodal feature set into the conditional diffusion model and fusing features from multiple modalities using a cross-attention mechanism to generate a solder joint sample set includes:

[0070] S31: Construct a conditional diffusion model.

[0071] S32: The t-th noisy data and the conditional control vector are fused using a cross-attention mechanism to obtain the t-th attention matrix.

[0072] S33: Input the multimodal feature set and the t-th attention matrix into the conditional diffusion model, perform the t-th reverse iteration, and obtain the (t-1)-th solder joint sample.

[0073] This invention uses the Denoising Diffusion Probabilistic Models (DDPM) as the architecture of a multimodal diffusion generation network, and adds a cross-attention mechanism to DDPM to distinguish the importance of different modalities.

[0074] This invention targets the geometric and textural features of SMT solder joints, using a multi-scale U-Net structure as the core denoising network. The forward diffusion process is defined as a Markov chain, where Gaussian noise is added to the data at each step. The forward diffusion process of the conditional diffusion model is described by the following formula:

[0075] ;

[0076] Where N represents a Gaussian distribution, x t Let x represent the t-th solder joint sample. t-1 This represents the (t-1)th solder joint sample; Let I be the t-th noise scheduling parameter, and let I be the identity matrix. This represents the forward diffusion from the (t-1)th solder joint sample to the tth solder joint sample.

[0077] The forward diffusion process of DDPM is a fixed Markov chain with progressively added noise, controlled by variance scheduling parameters. The clear solder joint sample x0 is transformed into approximately standard Gaussian noise within t steps, providing a training target for inverse denoising. An adaptive scheduling strategy is adopted to dynamically adjust the noise addition intensity based on the complexity and geometric features of the SMT solder joint defects, thereby achieving better noise generation results.

[0078] The step of inputting the multimodal feature set and the t-th attention matrix into the conditional diffusion model, performing the t-th reverse iteration, and obtaining the (t-1)-th solder joint sample includes:

[0079] The following formula is used to calculate the (t-1)th solder joint sample:

[0080] ;

[0081] Where N is a Gaussian distribution, x t-1 For the (t-1)th solder joint sample, x t For the t-th solder joint sample, Let A be the probability distribution of changing from the t-th solder joint sample to the (t-1)-th solder joint sample; A is the attention weight. Let A be the mean value of the t-th solder joint sample at time t with attention weight A. Let be the variance of the t-th solder joint sample at time t. These are the weight parameters for the inverse denoising process. The inverse generation process approximates the true inverse distribution through a learned neural network and utilizes a cross-attention mechanism to fuse multimodal conditional information.

[0082] Visible light characteristics f vis Visual features including weld joint geometry parameters, surface texture features, and color distribution statistics, as well as infrared features. ir Includes thermal information such as temperature distribution patterns and thermal conductivity parameters, with deep features f depth It includes spatial geometric information such as the three-dimensional geometry of the weld joint, height distribution statistics, and surface roughness.

[0083] The t-th attention matrix is ​​calculated using the following formula:

[0084] ;

[0085] Among them, Q t Let K be the t-th query matrix. t Let V be the t-th bond matrix. t Let t be the value matrix, and Attention(Q) t ,K t V t Let be the t-th attention matrix; softmax is the normalized exponential function. Let be the transpose of the t-th key matrix. V is the vector length of the t-th key matrix. t Let t be the t-th value matrix.

[0086] The SMT solder joint detection method based on hypergraph contrastive learning provided in this embodiment includes acquiring an input image set, which includes visible light images, infrared images, and depth images. A multimodal feature set is extracted from the input image set, including visible light features, infrared features, and depth features. Visible light features include visual features such as the geometric shape parameters, surface texture features, and color distribution statistics of the solder joints; infrared features include thermal information such as temperature distribution patterns and thermal conductivity parameters; and depth features include spatial geometric information such as the three-dimensional geometric structure, height distribution statistics, and surface roughness of the solder joints. The multimodal feature set is input into a conditional diffusion model, and a cross-attention mechanism is used to fuse the features of multiple modalities to generate a solder joint sample set. The solder joint sample set generated by the conditional diffusion model expands the training data and improves the model's generalization ability. The cross-attention mechanism enables the generated solder joint samples to carry multimodal information. A hypergraph structure is constructed based on the multimodal feature set and the solder joint sample set. Hypergraph convolution is performed on the hypergraph structure to extract the hypergraph node feature set. A hypergraph structure is constructed based on spatial distance, solder joint type similarity, and welding causal relationships. This captures high-order correlations between solder joints, thereby enhancing the discriminative power of the features. A multi-level attention fusion mechanism is employed to fuse multimodal feature sets and hypergraph node feature sets, resulting in fused features. This multi-level attention fusion mechanism integrates multimodal and structured information, providing more comprehensive and robust feature representations. Based on the fused features, a student network is trained using knowledge distillation technology to detect SMT solder joints. The teacher network includes a multimodal diffusion generation module and a hypergraph learning module. The distillation process allows the teacher network to transfer the learned multimodal data and relationships between data to the student network, which is lightweight. Using the student network to identify SMT solder joints fully utilizes multimodal sensor information, effectively models complex spatial relationships, possesses strong generalization capabilities, and offers the advantage of real-time SMT solder joint detection.

[0087] Example 2

[0088] like Figure 1 As shown, this embodiment provides an SMT solder joint detection method based on hypergraph contrast learning. This embodiment describes the differences between it and Embodiment 1. The method includes:

[0089] S1: Obtain the input image set, which includes visible light images, infrared images, and depth images.

[0090] S2: Extract a multimodal feature set from the input image set, the feature set including visible light features, infrared features and depth features.

[0091] S3: Input the multimodal feature set into the conditional diffusion model, and use the cross-attention mechanism to fuse the features of multiple modalities to generate a solder joint sample set.

[0092] S4: Construct a hypergraph structure based on the multimodal feature set and the solder joint sample set, perform hypergraph convolution on the hypergraph structure, and extract the hypergraph node feature set.

[0093] S5: The multi-level attention fusion mechanism is used to fuse the multimodal feature set and the hypergraph node feature set to obtain the fused features.

[0094] S6: Based on the fusion features, a student network is trained using knowledge distillation technology, and the student network is used to detect SMT solder joints.

[0095] like Figure 2 As shown, the construction of the hypergraph structure based on the multimodal feature set and the solder joint sample set includes:

[0096] S41: Select multiple target solder joints that meet the connection conditions to obtain a node set; the connection conditions include any one of the following: spatial distance is less than a distance threshold, they belong to the same solder joint type, and there is a causal relationship between the solder joints.

[0097] S42: Connect the paired target solder joints using superedges to obtain a superedge set.

[0098] S43: Construct a weight matrix based on the distance and similarity between the target solder joints.

[0099] S44: The set of nodes, the set of hyperedges, and the weight matrix are combined to form a hypergraph structure.

[0100] Traditional graph neural networks can only model binary relationships between nodes. However, SMT solder joint detection involves multiple higher-order relationships between solder joints. This embodiment constructs a hypergraph Super=(V,E,W) to represent these higher-order relationships, where V is the set of nodes, E is the set of hyperedges, and W is the weight matrix. The node set contains multiple target solder joints, and each hyperedge in the hyperedge set connects num nodes, where num≥2. Each hyperedge in the hyperedge set is assigned a scalar weight, where the scalar weight is ≥0.

[0101] Each target solder joint is treated as a node in a node set, and each node contains the multimodal features of the corresponding target solder joint. A set of hyperedges E is constructed based on the association type between the target solder joints, and a weight matrix W is calculated based on the similarity and association strength between the target solder joints.

[0102] Hyperedges in the set are divided into three categories: spatial proximity hyperedges, functionally related hyperedges, and defect propagation hyperedges. Spatially proximity hyperedges connect target solder joints with a spatial distance less than a distance threshold, such as 10 pixels. Functionally related hyperedges connect target solder joints of the same type, such as num solder joints that are all BGA solder joints. Causal hyperedges connect solder joints that have a causal relationship, such as solder joints in upstream and downstream processes.

[0103] Use an adjacency list or adjacency matrix to store hyperedge information to ensure that the same hyperedge is not added repeatedly. The following strategies are adopted to prevent hyperedge intersection and duplication: (1) Encode each hyperedge with a unique identifier. (2) Maintain a hyperedge set and check whether the hyperedge already exists before adding it. (3) Establish a hyperedge list corresponding to each target solder joint.

[0104] The step of constructing a weight matrix based on the distance and similarity between the target solder joints includes:

[0105] Calculate the weight matrix using the following formula:

[0106] ;

[0107] Among them, W i,j Let f be the weight in the i-th row and j-th column of the weight matrix, softmax be the normalization exponential function, d be the feature dimension, and f be the weight in the i-th row and j-th column of the weight matrix. i For the i-th target solder joint, f j For the j-th target solder joint; sim(f i ,f j Let ) be the similarity function between the i-th target solder joint and the j-th target solder joint, exp be the exponential function, and d be the similarity function between them. i,j Let be the Euclidean distance between the i-th target solder joint and the j-th target solder joint. W represents the distance attenuation coefficient, and . represents the multiplication operation. The higher the similarity between the i-th and j-th target solder joints, the greater the Euclidean distance between them. i,j The larger the similarity function, the better. In this embodiment, the similarity function is the cosine similarity function.

[0108] The step of performing hypergraph convolution on the hypergraph structure to extract the hypergraph node feature set includes:

[0109] The hypergraph node feature set is extracted using the following formula:

[0110] ;

[0111] ;

[0112] in, For the (l+1)th hypergraph node features, For the features of the l-th hypergraph node, For the sigmoid function, D v Let D be the degree matrix of the nodes. e H is the hypermarginality matrix; H is the incidence matrix; W is the weight matrix. T It is the transpose of the incidence matrix. is the learnable parameter matrix of the l-th layer; Laplace represents the hypergraph Laplacian matrix; HyperConv is the hypergraph convolutional layer, which is used to iteratively update the hypergraph node features;

[0113] All the hypergraph node features are combined into a hypergraph node feature set.

[0114] Node degree matrix D v Given an n×n diagonal matrix, where n is the total number of nodes and D is the diagonal element... ii The degree of node i is represented by the number of hyperedges connected to node i. This invention uses hypergraph convolution to update and propagate features, and employs contrastive learning to optimize node representations. The contrastive learning loss function is calculated using the following formula:

[0115] ;

[0116] Among them, h i Let i be the feature representation of node i. Let j be the feature representation of the positive sample node. Let be the feature representation of the negative sample node k, sim be the similarity function, and exp be the exponential function. Let ln be the temperature parameter, and ln be the logarithmic function with the natural constant as the base.

[0117] The initial features are represented using a multilayer perceptron:

[0118] ;

[0119] in, The initial hypergraph node features are defined by MLP, which stands for Multilayer Perceptron. i For the i-th target solder joint, c vis For visible light condition information, c ir For infrared conditional information, c depth For depth-conditional information, the initial hypergraph node features It incorporates multimodal conditional information.

[0120] Visible light features, infrared features, and depth features are extracted using different neural networks within the model to characterize the data distribution characteristics under different modalities. Visible light conditional information, infrared conditional information, and depth conditional information are external inputs used to control the direction of feature generation. The l-th layer features of node i are input into the hypergraph convolutional layer for hypergraph convolution to obtain the (l+1)-th layer features of node i.

[0121] This embodiment enhances the model's discriminative ability by minimizing the similarity of negative sample pairs through a contrastive learning loss function, while maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. A multilayer perceptron is employed to fuse multimodal conditional information, and features are iteratively updated through hypergraph convolutional layers.

[0122] Example 3

[0123] like Figure 1 As shown, this embodiment provides an SMT solder joint detection method based on hypergraph contrast learning. This embodiment describes the differences between it and Embodiment 1. The method includes:

[0124] S1: Obtain the input image set, which includes visible light images, infrared images, and depth images.

[0125] S2: Extract a multimodal feature set from the input image set, the feature set including visible light features, infrared features and depth features.

[0126] S3: Input the multimodal feature set into the conditional diffusion model, and use the cross-attention mechanism to fuse the features of multiple modalities to generate a solder joint sample set.

[0127] S4: Construct a hypergraph structure based on the multimodal feature set and the solder joint sample set, perform hypergraph convolution on the hypergraph structure, and extract the hypergraph node feature set.

[0128] S5: The multi-level attention fusion mechanism is used to fuse the multimodal feature set and the hypergraph node feature set to obtain the fused features.

[0129] S6: Based on the fusion features, a student network is trained using knowledge distillation technology, and the student network is used to detect SMT solder joints.

[0130] The step of extracting a multimodal feature set from the input image set includes:

[0131] S21: Extract visible light features from the visible light image using a ResNet neural network.

[0132] S22: Use a convolutional neural network to extract infrared features from the infrared image.

[0133] S23: Use a 3D convolutional network to extract depth features from the depth image.

[0134] ResNet neural networks employ skip connections to directly pass identity mappings, allowing the network to learn only the residuals instead of the complete mapping, thus avoiding gradient vanishing or exploding. Convolutional neural networks are used to extract infrared features from infrared images, offering high computational efficiency. The 3D convolutional kernels of 3D convolutional networks simultaneously slide through spatial and temporal dimensions, capturing both appearance and motion information in one go, forming spatiotemporal deep fusion features.

[0135] like Figure 3 As shown, the multi-level attention fusion mechanism is used to fuse the multimodal feature set and the hypergraph node feature set to obtain fused features, including:

[0136] S51: Calculate the intramodal attention weights.

[0137] S52: Calculate the intermodal attention weights.

[0138] S53: Based on the intra-modal attention weights and inter-modal attention weights, the feature matrices of all solder joints in each modality are weighted and summed to obtain the fused features.

[0139] This invention designs a multi-layered attention fusion mechanism, including two layers: intra-modal attention and inter-modal attention. The intra-modal attention mechanism is used to extract important features within each modality, while the inter-modal attention mechanism is used to fuse feature information from different modalities.

[0140] The intramodal attention weights are calculated using the following formula:

[0141] ;

[0142] Where softmax represents the normalization exponential function, Let m be the intramodal attention weights. Let f be the transpose of the learnable weight matrix, tanh be the hyperbolic tangent function, and f be the weight matrix. m The characteristic matrix of mode m contains the characteristics of all solder joints under mode m, b m Let represent the learnable bias vector corresponding to mode m. Design a loss function for intra-modal attention, iteratively reduce this loss function, and update the weight using backpropagation. m and b m This continues until the loss function of intramodal attention reaches its minimum value.

[0143] The eigenmatrix of mode m is calculated using the following formula:

[0144] ;

[0145] Among them, Encoder m For the m-th feature extraction network, x m This is a sample of the weld points for mode m.

[0146] The intermodal attention weights are calculated using the following formula:

[0147] ;

[0148] Where softmax is the normalization exponential function, Let m be the intermodal attention weights. Let be the transpose matrix of the cross-modal attention weights. Let m be the visible light feature matrix. For the m-th infrared feature matrix, Let m be the m-th depth matrix.

[0149] The fusion features are calculated using the following formula:

[0150] ;

[0151] ;

[0152] Among them, f fused As a feature of fusion, Let M be the feature matrix of mode m after in-modal attention adjustment, and M be the total number of modes. f is the intramodal attention weight for the i-th solder joint sample in mode m. m,i Let be the feature matrix of the i-th weld point sample of mode m.

[0153] Preferably, since different modal data have different distributions, a domain adaptation technique is used to align the feature spaces:

[0154] ;

[0155] Among them, L align Let u be the feature space alignment loss function. vis Let u be the eigenvalue vector of the visible light modes. ir Let u be the characteristic mean vector of the infrared mode. depth Let be the feature mean vector of the deep modality, and ||||2 denotes the 2-norm operation.

[0156] This embodiment combines intra-modal attention and inter-modal attention mechanisms to fuse features from multiple modalities and multiple samples to obtain fused features. The fused features have the following functions: (1) serving as the initial feature representation of hypergraph nodes. (2) used to calculate the similarity and association strength between nodes. (3) used as input to the classifier for final defect detection. (4) used to calculate the value of the contrastive learning loss function.

[0157] Example 4

[0158] like Figure 1 As shown, this embodiment provides an SMT solder joint detection method based on hypergraph contrast learning. This embodiment describes the differences between it and Embodiment 1. The method includes:

[0159] S1: Obtain the input image set, which includes visible light images, infrared images, and depth images.

[0160] S2: Extract a multimodal feature set from the input image set, the feature set including visible light features, infrared features and depth features.

[0161] S3: Input the multimodal feature set into the conditional diffusion model, and use the cross-attention mechanism to fuse the features of multiple modalities to generate a solder joint sample set.

[0162] S4: Construct a hypergraph structure based on the multimodal feature set and the solder joint sample set, perform hypergraph convolution on the hypergraph structure, and extract the hypergraph node feature set.

[0163] S5: The multi-level attention fusion mechanism is used to fuse the multimodal feature set and the hypergraph node feature set to obtain the fused features.

[0164] S6: Based on the fusion features, a student network is trained using knowledge distillation technology, and the student network is used to detect SMT solder joints.

[0165] The step of training the student network using knowledge distillation techniques based on the fusion features includes:

[0166] S61: Construct a teacher network, which includes a multimodal feature extraction module, a multimodal diffusion generation module, a hypergraph feature extraction module, and a multimodal feature fusion module.

[0167] S62: Train the teacher network until the teacher network converges.

[0168] S63: Use the soft labels of the teacher network to guide the training of the student network.

[0169] S64: If the number of training iterations of the student network is greater than or equal to the training iteration threshold, then training is stopped.

[0170] The teacher network consists of all modules from steps S1 to S5, including a complete multimodal diffusion generation module and a hypergraph learning module. First, the complex teacher network is trained until convergence. The soft labels of the teacher network guide the student network's learning. The degree of knowledge transfer is adjusted using a temperature parameter, optimizing the distillation loss and thus achieving model compression.

[0171] The method of using the soft labels of the teacher network to guide the training of the student network includes:

[0172] The student network was trained using the distillation loss function.

[0173] ;

[0174] Among them, L KD Let L be the distillation loss function, and balance be the balance coefficient. CE Let cross-entropy be the loss function. Here, KL represents the information divergence parameter, and temperature is the temperature parameter. For the sigmoid function, z s For the output of the student network, z t This is the output of the teacher network.

[0175] The distillation process ensures that the student network inherits the multimodal data and relationships learned by the teacher network, while retaining the core functionality of the teacher network but with a lighter structure. Using knowledge distillation can reduce model complexity, improve inference efficiency, and, while maintaining high detection accuracy, facilitate the transition from offline training to online detection.

[0176] Table 1 Comparison of detection performance of different methods

[0177]

[0178] Extensive comparative experiments conducted on standard SMT solder joint datasets demonstrate that the method of this invention has significant performance advantages over existing technologies. Experimental results show that the method of this invention achieves an accuracy of 92.8%, a recall of 90.5%, an F1 score of 91.6%, and a mean average precision (mAP) of 89.3%, representing improvements of 14.3%, 18.2%, 16.4%, and 17.5% respectively compared to traditional machine vision methods.

[0179] The optimized lightweight model achieves excellent real-time performance while maintaining high detection accuracy. On an NVIDIA RTX 3080 graphics card, the detection time for a single image is only 2.3 milliseconds, fully meeting the real-time requirements of industrial production lines. The compressed model size is only 12.8MB, an 85% reduction compared to the original model, and power consumption is reduced by 67%, making it suitable for deployment on edge devices.

[0180] The knowledge distillation loss function in this embodiment uses information divergence, i.e., KL divergence, as one of the loss terms. Compared with general knowledge distillation techniques, it has the following advantages: (1) Introducing a temperature parameter The squared term enhances the influence of soft labels. (2) The sigmoid function is applied to the outputs of both the teacher network and the student network, which is more suitable for multi-label classification scenarios. (3) The importance of soft labels and hard labels is dynamically balanced by balancing parameters.

[0181] Example 5

[0182] This embodiment provides an SMT solder joint inspection device based on hypergraph comparison learning, including:

[0183] An input image set acquisition module is used to acquire an input image set, which includes visible light images, infrared images, and depth images;

[0184] The feature extraction module is used to extract a multimodal feature set from the input image set, the feature set including visible light features, infrared features and depth features;

[0185] The multimodal feature fusion module is used to input the multimodal feature set into the conditional diffusion model, and use a cross-attention mechanism to fuse features from multiple modalities to generate a solder joint sample set.

[0186] The hypergraph node feature set extraction module is used to construct a hypergraph structure based on the multimodal feature set and the solder joint sample set, perform hypergraph convolution on the hypergraph structure, and extract the hypergraph node feature set;

[0187] The feature fusion module is used to fuse the multimodal feature set and the hypergraph node feature set using a multi-level attention fusion mechanism to obtain fused features;

[0188] The SMT solder joint detection module is used to train a student network based on the fused features using knowledge distillation technology, and then use the student network to detect SMT solder joints.

[0189] The SMT solder joint detection device based on hypergraph comparison learning in this embodiment is used to execute the SMT solder joint detection method based on hypergraph comparison learning in any one of the embodiments 1-4.

[0190] This embodiment also provides a computer device, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor in this computer design provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communication with external terminals via a network connection.

[0191] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements an SMT solder joint detection method based on hypergraph contrast learning. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0192] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0193] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for SMT solder joint detection based on hypergraph contrast learning, characterized in that, include: Obtain an input image set, which includes visible light images, infrared images, and depth images; A multimodal feature set is extracted from the input image set, the feature set including visible light features, infrared features and depth features; The multimodal feature set is input into the conditional diffusion model, and the features of multiple modalities are fused using a cross-attention mechanism to generate a solder joint sample set. A hypergraph structure is constructed based on the multimodal feature set and the solder joint sample set. Hypergraph convolution is performed on the hypergraph structure to extract the hypergraph node feature set. A multi-level attention fusion mechanism is used to fuse the multimodal feature set and the hypergraph node feature set to obtain fused features; The multi-level attention fusion mechanism is used to fuse the multimodal feature set and the hypergraph node feature set to obtain fused features, including: Calculate the intramodal attention weights; Calculate the intermodal attention weights; Based on the intra-modal attention weights and inter-modal attention weights, the feature matrices of all solder joints in each modality are weighted and summed to obtain the fused features; Based on the fusion features, a student network is trained using knowledge distillation technology, and the student network is used to detect SMT solder joints. The construction of the hypergraph structure based on the multimodal feature set and the solder joint sample set includes: Multiple target solder joints that meet the connection conditions are selected to obtain a node set; the connection conditions include any one of the following: spatial distance is less than a distance threshold, they belong to the same solder joint type, and there is a causal relationship between the solder joints; The paired target solder joints are connected using superedges to obtain a superedge set; A weight matrix is ​​constructed based on the distance and similarity between the target solder joints; The set of nodes, the set of hyperedges, and the weight matrix are combined to form a hypergraph structure.

2. The SMT solder joint detection method based on hypergraph contrast learning according to claim 1, characterized in that, The step of inputting the multimodal feature set into the conditional diffusion model and fusing features from multiple modalities using a cross-attention mechanism to generate a solder joint sample set includes: Construct a conditional diffusion model; A cross-attention mechanism is used to fuse the t-th noisy data and the conditional control vector to obtain the t-th attention matrix; The multimodal feature set and the t-th attention matrix are input into the conditional diffusion model, and the t-th reverse iteration is performed to obtain the (t-1)-th solder joint sample.

3. The SMT solder joint detection method based on hypergraph contrast learning according to claim 1, characterized in that, The step of constructing a weight matrix based on the distance and similarity between the target solder joints includes: Calculate the weight matrix using the following formula: ； Among them, W i,j Let f be the weight in the i-th row and j-th column of the weight matrix, softmax be the normalization exponential function, d be the feature dimension, and f be the weight in the i-th row and j-th column of the weight matrix. i For the i-th target solder joint, f j For the j-th target solder joint; sim(f i ,f j Let ) be the similarity function between the i-th target solder joint and the j-th target solder joint, exp be the exponential function, and d be the similarity function between them. i,j Let be the Euclidean distance between the i-th target solder joint and the j-th target solder joint. is the distance attenuation coefficient, and . represents the multiplication operation.

4. The SMT solder joint detection method based on hypergraph contrast learning according to claim 1, characterized in that, The step of performing hypergraph convolution on the hypergraph structure to extract the hypergraph node feature set includes: The hypergraph node feature set is extracted using the following formula: ； ； in, For the (l+1)th hypergraph node features, For the features of the l-th hypergraph node, For the sigmoid function, D v Let D be the degree matrix of the nodes. e H is the hypermarginality matrix; H is the incidence matrix; W is the weight matrix. T It is the transpose of the incidence matrix. is the learnable parameter matrix of the l-th layer; Laplace represents the hypergraph Laplacian matrix; HyperConv is the hypergraph convolutional layer, which is used to iteratively update the hypergraph node features; All the hypergraph node features are combined into a hypergraph node feature set.

5. The SMT solder joint detection method based on hypergraph contrast learning according to claim 1, characterized in that, The step of training the student network using knowledge distillation techniques based on the fusion features includes: A teacher network is constructed, comprising a multimodal feature extraction module, a multimodal diffusion generation module, a hypergraph feature extraction module, and a multimodal feature fusion module; Train the teacher network until it converges; The soft labels of the teacher network are used to guide the training of the student network; If the number of training iterations of the student network is greater than or equal to the training iteration threshold, then training is stopped.

6. The SMT solder joint detection method based on hypergraph contrast learning according to claim 5, characterized in that, The method of using the soft labels of the teacher network to guide the training of the student network includes: The student network was trained using the distillation loss function. ； Among them, L KD Let L be the distillation loss function, and balance be the balance coefficient. CE Let cross-entropy be the loss function. Here, KL represents the information divergence parameter; For the sigmoid function, z s For the output of the student network, z t , where represents the output of the teacher network, and . represents the multiplication operation.

7. The SMT solder joint detection method based on hypergraph contrast learning according to claim 1, characterized in that, The step of inputting the multimodal feature set and the t-th attention matrix into the conditional diffusion model, performing the t-th reverse iteration, and obtaining the (t-1)-th solder joint sample includes: The following formula is used to calculate the (t-1)th solder joint sample: ； Where N is a Gaussian distribution, x t-1 For the (t-1)th solder joint sample, x t For the t-th solder joint sample, Let A be the probability distribution of changing from the t-th solder joint sample to the (t-1)-th solder joint sample; A is the attention weight. Let A be the mean value of the t-th solder joint sample at time t with attention weight A. Let be the variance of the t-th solder joint sample at time t. These are the weight parameters for the reverse denoising process.

8. The SMT solder joint detection method based on hypergraph contrast learning according to claim 1, characterized in that, The step of extracting a multimodal feature set from the input image set includes: The visible light features are extracted from the visible light image using a ResNet neural network; Infrared features are extracted from the infrared image using a convolutional neural network. A 3D convolutional network is used to extract depth features from the depth image.

Images