A pcba multi-defect collaborative diagnosis method fusing supergraph diffusion and cross-modal contrast

Abstract

The application provides a PCBA multi-defect collaborative diagnosis method fusing supergraph diffusion and cross-modal contrast, comprising constructing a multi-layer supergraph structure, generating multi-modal defect samples based on the multi-layer supergraph structure by using a conditional supergraph diffusion model. A multi-layer alignment mechanism is used to align the cross-modal features of the multi-modal defect samples, and a defect interaction matrix of the aligned features is calculated. According to the defect interaction matrix, the message passing mechanism of the graph neural network is used to update the node features of each node in the node layer; the node features are input into a multi-task learning framework, the defects of each node are predicted, and node defect prediction results are obtained. The above method can effectively model the complex correlation between PCBA defects, fully utilize multi-modal sensor information, and has real-time collaborative diagnosis capability, so as to meet the urgent needs of modern electronic manufacturing industry for high-precision, high-efficiency and high-reliability quality detection.

Description

Technical Field

[0001] This invention relates to the field of PCBA defect detection and image processing technology, and in particular to a collaborative diagnostic method for multiple PCBA defects that integrates hypergraph diffusion and cross-modal comparison. Background Technology

[0002] Printed Circuit Board Assembly (PCBA) is a crucial step in modern electronic product manufacturing, and its quality directly determines the electrical performance, mechanical reliability, and lifespan of the entire electronic product. With the rapid development of electronic products towards miniaturization, high integration, and multifunctionality, the component density on PCBAs is constantly increasing, circuit structures are becoming increasingly complex, and defect types are showing a trend towards diversification and complexity. Traditional single-defect detection methods can no longer meet the urgent needs of modern manufacturing for high-precision and high-efficiency quality control.

[0003] Common defects in PCBA manufacturing encompass multiple levels and stages. Soldering defects, including cold solder joints, bridging, solder balls, and solder balls, directly affect the reliability of electrical connections. Component defects, such as misalignment, missing components, incorrect polarity, and incorrect model numbers, can lead to abnormal circuit function. Circuit defects, including short circuits, open circuits, impedance abnormalities, and signal integrity issues, severely impact circuit performance. Surface defects, such as scratches, contamination, oxidation, and corrosion, can cause long-term reliability problems. Adding to the complexity, these different types of defects are often interconnected and mutually influential; a single defect can trigger a chain reaction, leading to the simultaneous occurrence of multiple defects and forming complex defect combination patterns.

[0004] Existing PCBA defect detection technologies exhibit numerous technical limitations and practical problems when dealing with such complex and ever-changing defect patterns. Firstly, their ability to model defect correlations is severely inadequate. Traditional detection methods typically treat various defects as independent events, neglecting the inherent connections and mutual influence mechanisms between defects. This fails to effectively capture the propagation patterns and combination characteristics of defects, resulting in limited detection accuracy for complex defect patterns, particularly when facing multiple concurrent defects.

[0005] Secondly, the insufficient utilization of multimodal information limits the overall performance of the inspection system. PCBA quality inspection involves multiple inspection methods and data modalities, such as optical imaging, X-ray imaging, thermal imaging, and electrical testing, each providing unique defect information and inspection perspectives. However, existing methods lack effective cross-modal information fusion mechanisms, failing to fully leverage the complementary advantages of different modal data, leading to redundancy and waste of inspection data.

[0006] Third, insufficient real-time collaborative diagnostic capabilities affect production efficiency. Modern electronics manufacturing requires real-time synchronous collaborative diagnosis of multiple defects. Existing methods often require separate detection of various defects, followed by post-processing fusion. This serial processing approach is not only inefficient but also prone to losing correlation information between defects.

[0007] In summary, there is an urgent need to develop a multi-defect intelligent diagnostic method that can effectively model the complex correlations between PCBA defects, fully utilize multimodal sensor information, and has real-time collaborative diagnostic capabilities, in order to meet the urgent needs of modern electronics manufacturing for high-precision, high-efficiency, and high-reliability quality inspection. Summary of the Invention

[0008] To overcome the problems existing in related technologies, the purpose of this invention is to provide a collaborative diagnostic method for multiple defects in PCBA that integrates hypergraph diffusion and cross-modal comparison. This method can effectively model the complex correlation between PCBA defects, make full use of multimodal sensor information, and has real-time collaborative diagnostic capabilities.

[0009] A collaborative diagnostic method for multiple defects in PCBAs that integrates hypergraph diffusion and cross-modal comparison includes:

[0010] Construct a multi-layered hypergraph structure, which includes a node layer, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges;

[0011] A conditional hypergraph diffusion model is constructed, and multimodal defect samples are generated based on the multi-layer hypergraph structure using the conditional hypergraph diffusion model.

[0012] A multi-layer alignment mechanism is used to perform cross-modal feature alignment on the multimodal defect samples to obtain aligned features;

[0013] Calculate the defect interaction matrix of the aligned features, which is used to model the mutual influence relationship between different defect types;

[0014] Based on the defect interaction matrix, the node features of each node in the node layer are updated using the message passing mechanism of a graph neural network;

[0015] The node features are input into a multi-task learning framework to predict the defects of each node, thus obtaining the node defect prediction results.

[0016] In a preferred embodiment of the present invention, the construction of the multi-layer hypergraph structure includes:

[0017] Treat each detection area on the PCBA as a node;

[0018] Obtain multimodal feature information for the region corresponding to each node; the multimodal feature information includes optical features, X-ray features, thermal features, and electrical features;

[0019] The multimodal feature information is converted into node feature vectors;

[0020] All the node feature vectors are combined into a node layer;

[0021] Hyperedges are constructed based on the relationships between different nodes in the node layer to obtain a multi-layered hypergraph structure.

[0022] In a preferred embodiment of the present invention, the step of constructing hyperedges based on the association relationships between different nodes in the node layer to obtain a multi-layer hypergraph structure includes:

[0023] Construct causal hyperedges based on the following formula:

[0024] ;

[0025] Among them, e causal V represents the causal relationship hyperedge. i Let v represent the feature vector of the i-th node. j Let (v) represent the feature vector of the j-th node. i ,v j () represents the superedge between the i-th node and the j-th node; defect i Denotes the i-th type of defect, defect j Indicates the j-th type of defect. This indicates that the i-th type of defect leads to the j-th type of defect;

[0026] The node layer, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges form a multi-layered hypergraph structure.

[0027] In a preferred embodiment of the present invention, the step of generating multimodal defect samples based on the multilayer hypergraph structure and using the conditional hypergraph diffusion model includes:

[0028] In the forward propagation process of the hypergraph diffusion model, noise is propagated through hyperedges;

[0029] In the reverse process of the hypergraph diffusion model, the multi-layer hypergraph structure is used as a condition to remove noise from the noise samples and generate multimodal defect samples.

[0030] In a preferred embodiment of the present invention, the step of using the multi-layer hypergraph structure as a condition to remove noise from noise samples and generate multimodal defect samples includes:

[0031] Based on the following formula, a hypergraph convolutional denoising network is used to remove noise from noisy samples:

[0032] ;

[0033] ;

[0034] HyperGCN represents a hypergraph convolutional denoising network, x t Let t represent the noise sample at time t, where t represents time. The term "feature concatenation operation" is defined here; "embed(t)" represents multi-scale temporal embedding; "C" represents the defect combination condition vector; and "embed(C)" represents the defect combination condition vector embedding. H represents the multi-layer hypergraph structure. Represents a multimodal defect sample, c defect Indicates the defect type, c severity Indicates the severity of the defect, c location Indicates the location of the defect, c interaction This indicates the interaction between different defects.

[0035] In a preferred embodiment of the present invention, the step of employing a multi-layer alignment mechanism to perform cross-modal feature alignment on the multimodal defect samples to obtain aligned features includes:

[0036] The following formula is used to perform cross-modal feature alignment on the multimodal defect samples:

[0037] ;

[0038] Among them, L semantic f represents the semantic alignment function. i f represents the feature of the defect sample in the i-th mode. j Let represent the features of the defect sample in the j-th mode, and let project represent the projection function;

[0039] The following formula is used to align the structures of different modes:

[0040] ;

[0041] Among them, L structure A represents the structure alignment function. i Let A represent the attention matrix for the i-th modality. j Let F represent the attention matrix for the j-th modality, and let F denote the Frobenius norm.

[0042] The following formula is used to align the feature distributions of different modes:

[0043] ;

[0044] Among them, L distributionThe feature distribution alignment function is represented by KL, which represents the divergence operation, and P... i P represents the characteristic distribution of mode i. j Represents the characteristic distribution of mode j. Indicates from P i To P j divergence, Indicates from P j To P i The divergence.

[0045] In a preferred embodiment of the present invention, calculating the defect interaction matrix of the aligned features includes:

[0046] Calculate the defect interaction matrix using the following formula:

[0047] ;

[0048] Among them, M ij Let represent the feature matrix of the defect sample in the i-th mode and the defect interaction matrix between the features of the defect sample in the j-th mode; sigmoid represents the sigmoid activation function, and * represents element-wise multiplication. This represents the transpose of the interaction weight matrix.

[0049] In a preferred embodiment of the present invention, updating the node features of each node in the node layer using a graph neural network message passing mechanism based on the defect interaction matrix includes:

[0050] The node features of each node in the node layer are updated using the following formula:

[0051] ;

[0052] ;

[0053] in, Represents node i at time t. Let j and e be nodes at time t. ij This represents the edge connecting node i and node j; Message represents the message passing mechanism. This represents the message transmitted from node i to node j at time t+1; Aggregate represents the aggregation operation, N i Let i represent the set of neighboring nodes, and Update represent the node update operation. Let i represent node i at time t+1.

[0054] In a preferred embodiment of the present invention, the step of inputting the node features into a multi-task learning framework to predict the defects of each node and obtain node defect prediction results includes:

[0055] The following formula is used to generate the node defect prediction result based on node i:

[0056] ;

[0057] ;

[0058] ;

[0059] in, This represents the predicted probability of the i-th type of defect occurring in the PCBA. Represents a non-linear activation function. h represents the transpose of the weight matrix of node i. i Let b represent the i-th node. i This represents the bias matrix of the i-th node with respect to the defect type; Let represent the predicted probability of the severity of the i-th type of defect in the PCBA, where softmax is the normalized exponential function. b is the transpose of the weight matrix representing the severity of the i-th defect. s The bias matrix representing the i-th node with respect to the severity of the defect; This represents the predicted probability of the correlation between the i-th defect and the j-th defect in a PCBA. b is the transpose of the weight matrix representing the correlation between the i-th defect and the j-th defect. int The bias matrix represents the correlation between the i-th defect and the j-th defect.

[0060] In a preferred embodiment of the present invention, the calculation formula for the multi-scale temporal embedding is as follows:

[0061] ;

[0062] in, This represents multi-scale temporal embedding, where sin represents the sine function, cos represents the cosine function, i represents the dimension index, and d represents the model embedding dimension.

[0063] The beneficial effects of this invention are as follows:

[0064] The PCBA multi-defect collaborative diagnosis method provided by this invention, which integrates hypergraph diffusion and cross-modal comparison, includes constructing a multi-layer hypergraph structure. This structure comprises a node layer, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges. The multi-layer hypergraph structure is an extended graph structure; by introducing hyperedges (edges connecting multiple nodes) and a layering mechanism, it can more effectively model complex high-order relationships. Compared to traditional simple graphs, its advantage lies in its ability to simultaneously capture multi-dimensional relationships and cross-layer associations. A conditional hypergraph diffusion model is constructed. Based on the multi-layer hypergraph structure, this model generates multi-modal defect samples. The conditional hypergraph diffusion model combines the powerful generation capabilities of diffusion generation models with the relationship modeling capabilities of hypergraph neural networks, generating corresponding multi-modal defect samples based on given defect combination conditions. A multi-layer alignment mechanism is used to align the cross-modal features of the multi-modal defect samples, obtaining aligned features. The defect interaction matrix of the aligned features is calculated, and this matrix is ​​used to model the mutual influence relationships between different defect types. Based on the defect interaction matrix, the node features of each node in the node layer are updated using a graph neural network message passing mechanism. Node features are input into a multi-task learning framework to predict defects at each node, yielding node defect prediction results. The hypergraph diffusion generation model constructs a hypergraph data structure to model high-order correlations between defects at different locations and of different types on the PCBA, leveraging the powerful generative capabilities of the conditional diffusion model to generate multi-defect combination samples that conform to the true defect distribution characteristics. This method effectively models the complex correlations between PCBA defects, fully utilizes multimodal sensor information, and possesses real-time collaborative diagnostic capabilities, meeting the urgent needs of modern electronics manufacturing for high-precision, high-efficiency, and high-reliability quality inspection. Attached Figure Description

[0065] Figure 1 This is a flowchart of the PCBA multi-defect collaborative diagnosis method integrating hypergraph diffusion and cross-modal comparison of the present invention;

[0066] Figure 2 This is a flowchart of the construction of a multi-layer hypergraph structure according to the present invention;

[0067] Figure 3 This is a flowchart of the present invention for generating multimodal defect samples using a conditional hypergraph diffusion model. Detailed Implementation

[0068] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0069] Example 1

[0070] like Figure 1 As shown, this embodiment provides a collaborative diagnostic method for multiple defects in PCBA that integrates hypergraph diffusion and cross-modal comparison, including:

[0071] S1: Construct a multi-layered hypergraph structure, which includes a node layer, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges;

[0072] S2: Construct a conditional hypergraph diffusion model, and generate multimodal defect samples based on the multi-layer hypergraph structure and the conditional hypergraph diffusion model;

[0073] S3: Employ a multi-layer alignment mechanism to perform cross-modal feature alignment on the multimodal defect samples to obtain aligned features;

[0074] S4: Calculate the defect interaction matrix of the aligned features; the defect interaction matrix is ​​used to model the mutual influence relationship between different defect types;

[0075] S5: Based on the defect interaction matrix, update the node features of each node in the node layer using the message passing mechanism of a graph neural network;

[0076] S6: Input the node features into the multi-task learning framework to predict the defects of each node and obtain the node defect prediction results.

[0077] like Figure 2 As shown, the construction of the multi-layer hypergraph structure includes:

[0078] S11: Treat each detection area on the PCBA as a node;

[0079] S12: Obtain multimodal feature information of the region corresponding to each node; the multimodal feature information includes optical features, X-ray features, thermal features, and electrical features;

[0080] S13: Convert the multimodal feature information into node feature vectors;

[0081] S14: Combine all the node feature vectors into a node layer;

[0082] S15: Construct hyperedges based on the relationships between different nodes in the node layer to obtain a multi-layer hypergraph structure.

[0083] Optical features offer high resolution and can reveal surface defects in PCBAs. X-rays have strong penetrating power and can reveal internal defects in PCBAs. Infrared imaging of PCBAs, where pixel intensity is positively correlated with temperature, allows for the timely detection of abnormal temperature areas by identifying certain PCBA defects or other electronic components within the circuitry. Electrical features include voltage, current, resistance, capacitance, and inductance. Issues such as poor soldering, bridging, solder balls, and cold solder joints can affect the electrical performance of PCBAs.

[0084] Each node stores multimodal feature information of the corresponding region on the PCBA, using optical, X-ray, thermal, and electrical features as feature elements, and combining all feature elements to form a node feature vector. Each node feature vector represents the multimodal feature information of a region on the PCBA, and all node feature vectors are combined to form a node layer.

[0085] The process of constructing hyperedges based on the relationships between different nodes in the node layer to obtain a multi-layer hypergraph structure includes:

[0086] Construct causal hyperedges based on the following formula:

[0087] ;

[0088] Among them, e causal H represents the causal hyperedge. i Let h represent the i-th node. j Represents the j-th node, (h i ,h j () represents the superedge between the i-th node and the j-th node; defect i Denotes the i-th type of defect, defect j Indicates the j-th type of defect. This indicates that the i-th type of defect leads to the j-th type of defect.

[0089] The node layer, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges form a multi-layered hypergraph structure.

[0090] Causal relationship hyperedge e causal Based on defect causal relationships, this is used to model the causal propagation chain between defects. If there is a causal relationship between the i-th node and the j-th node, it indicates that there is a relationship between the i-th type of defect and the j-th type of defect.

[0091] Construct spatially related hyperedges according to the following formula:

[0092] ;

[0093] Among them, espatial This represents a spatially associated hyperedge, where dist represents the distance between two physical coordinates, and pos represents the distance between them. i Represents the physical coordinates of the i-th node, pos j Let r represent the physical coordinates of the j-th node. spatial Represents the spatial correlation radius parameter, and the spatial correlation hyperedge e. spatial Based on physical proximity, it is used to model the defect propagation relationship between adjacent regions of PCBA.

[0094] Construct functionally related hyperedges using the following formula:

[0095] ;

[0096] Among them, e functional Indicates a functionally related hyperedge, module(h) i ) represents the functional type of the i-th node, module(h j ) represents the function type of the j-th node.

[0097] The step of inputting the node features into a multi-task learning framework to predict the defects of each node and obtain node defect prediction results includes:

[0098] The following formula is used to generate the node defect prediction result based on node i:

[0099] ;

[0100] ;

[0101] ;

[0102] in, This represents the predicted probability of the i-th type of defect occurring in the PCBA. Represents a non-linear activation function. h represents the transpose of the weight matrix of node i. i Let b represent the i-th node. i This represents the bias matrix of the i-th node with respect to the defect type; Let represent the predicted probability of the severity of the i-th type of defect in the PCBA, where softmax is the normalized exponential function. b is the transpose of the weight matrix representing the severity of the i-th defect. s The bias matrix representing the i-th node with respect to the severity of the defect; This represents the predicted probability of the correlation between the i-th defect and the j-th defect in a PCBA. b is the transpose of the weight matrix representing the correlation between the i-th defect and the j-th defect. intThe bias matrix represents the correlation between the i-th defect and the j-th defect.

[0103] Multi-task learning is a machine learning method that improves the generalization ability of a model by jointly optimizing multiple related tasks. Its core idea is to enhance learning efficiency by sharing underlying representations across tasks, such as feature extraction layers, and leveraging task correlations. After updating nodes, the probability and severity of each defect type are calculated based on the attributes of each node, and the probability of correlation between two defects is calculated based on the attributes of two nodes.

[0104] This embodiment provides a PCBA multi-defect collaborative diagnosis method that integrates hypergraph diffusion and cross-modal comparison. The method involves constructing a multi-layer hypergraph structure, including node layers, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges. The multi-layer hypergraph structure is an extended graph structure; by introducing hyperedges (edges connecting multiple nodes) and a layering mechanism, it can more effectively model complex high-order relationships. Compared to traditional simple graphs, its advantage lies in its ability to simultaneously capture multi-dimensional relationships and cross-layer associations. A conditional hypergraph diffusion model is constructed. Based on the multi-layer hypergraph structure, the conditional hypergraph diffusion model generates multimodal defect samples; for example, the second defect sample in the first modality is an image. The conditional hypergraph diffusion model combines the powerful generation capabilities of diffusion generation models with the relationship modeling capabilities of hypergraph neural networks, generating corresponding multimodal defect samples based on given defect combination conditions. A multi-layer alignment mechanism is used to align the cross-modal features of the multimodal defect samples, obtaining aligned features. The defect interaction matrix of the aligned features is calculated, and this matrix is ​​used to model the mutual influence relationships between different defect types. Based on the defect interaction matrix, a message passing mechanism of a graph neural network is used to update the node features of each node in the node layer. These node features are then input into a multi-task learning framework to predict defects at each node, yielding the node defect prediction results. The hypergraph diffusion generation model constructs a hypergraph data structure to model the high-order correlations between defects at different locations and of different types on the PCBA, leveraging the powerful generative capabilities of the conditional diffusion model to generate multi-defect combination samples that conform to the true defect distribution characteristics. This method can effectively model the complex correlations between PCBA defects, fully utilize multimodal sensor information, and possess real-time collaborative diagnostic capabilities, thus meeting the urgent needs of modern electronics manufacturing for high-precision, high-efficiency, and high-reliability quality inspection.

[0105] Example 2

[0106] like Figure 1 As shown, this embodiment provides a collaborative diagnostic method for multiple defects in PCBA that integrates hypergraph diffusion and cross-modal comparison. This embodiment describes the differences from Embodiment 1, based on Embodiment 1. The method includes:

[0107] S1: Construct a multi-layered hypergraph structure, which includes a node layer, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges;

[0108] S2: Construct a conditional hypergraph diffusion model, and generate multimodal defect samples based on the multi-layer hypergraph structure and the conditional hypergraph diffusion model;

[0109] S3: Employ a multi-layer alignment mechanism to perform cross-modal feature alignment on the multimodal defect samples to obtain aligned features;

[0110] S4: Calculate the defect interaction matrix of the aligned features; the defect interaction matrix is ​​used to model the mutual influence relationship between different defect types;

[0111] S5: Based on the defect interaction matrix, update the node features of each node in the node layer using the message passing mechanism of a graph neural network;

[0112] S6: Input the node features into the multi-task learning framework to predict the defects of each node and obtain the node defect prediction results.

[0113] like Figure 3 As shown, the generation of multimodal defect samples based on the multi-layer hypergraph structure and using the conditional hypergraph diffusion model includes:

[0114] S22: In the forward propagation process of the hypergraph diffusion model, noise is propagated through hyperedges;

[0115] S23: In the reverse process of the hypergraph diffusion model, the multi-layer hypergraph structure is used as a condition to remove noise from the noise samples and generate multimodal defect samples.

[0116] Before step S22, there is also step S21: constructing a conditional hypergraph diffusion model.

[0117] The conditional hypergraph diffusion model is a framework that combines a multi-layered hypergraph structure with diffusion probability generation. Its core objective is to generate high-quality data under conditional control through the complex relational network of the hypergraph. The conditional hypergraph diffusion model captures high-order relationships between features through multilateral associations between hypergraph nodes and utilizes the inverse transformation of the diffusion process to generate multimodal defect samples.

[0118] The hypergraph diffusion model includes hypergraph structure embedding and conditional control mechanisms. The hypergraph connects any number of nodes through hyperedges, making it more suitable for modeling multi-entity interactions than traditional graphs. When noise propagates between hypergraph nodes during the diffusion process, the impact of hyperedge weights on information transmission needs to be considered. By introducing conditional variables such as text descriptions or category labels, the model dynamically adjusts the generation direction of hypergraph nodes during inverse denoising.

[0119] The forward propagation process of the conditional hypergraph diffusion model involves progressively adding noise to the nodes of a multi-layered hypergraph structure, with the hyperedge weights determining the noise propagation path. The backward propagation process uses a hypergraph neural network to predict the noise and then gradually recovers the data by incorporating conditional information.

[0120] The step of using the multi-layer hypergraph structure as a condition to remove noise from noise samples and generate multimodal defect samples includes:

[0121] Based on the following formula, a hypergraph convolutional denoising network is used to remove noise from noisy samples:

[0122] ;

[0123] ;

[0124] HyperGCN represents a hypergraph convolutional denoising network, x t Let t represent the noise sample at time t, where t represents time. The term "feature concatenation operation" is defined here; "embed(t)" represents multi-scale temporal embedding; "C" represents the defect combination condition vector; and "embed(C)" represents the defect combination condition vector embedding. H represents the multi-layer hypergraph structure. Represents a multimodal defect sample, c defect Indicates the defect type, c severity Indicates the severity of the defect, c location Indicates the location of the defect, c interaction This indicates the interaction between different defects.

[0125] Multi-scale temporal embedding employs sinusoidal position coding:

[0126] ;

[0127] in, This represents multi-scale temporal embedding, where sin represents the sine function, cos represents the cosine function, i represents the dimension index, and d represents the model embedding dimension.

[0128] Sine positional encoding is a method used in Transformer models to inject sequence positional information. Its core idea is to generate a unique encoding vector for each position by combining sine and cosine functions, and to use different frequency components to encode positional information. High-frequency components correspond to low-dimensional features, and low-frequency components correspond to high-dimensional features, so that the model can capture the sequence relationship between local and global features.

[0129] Extensive comparative experiments conducted on standard PCBA defect datasets demonstrate that the method provided in this invention has significant advantages in multi-defect collaborative detection. As shown in Table 1, the method provided in this invention achieves good detection results for different combinations of defect types.

[0130] Table 1 Comparison of detection accuracy for different PCBA defect type combinations

[0131]

[0132] The hypergraph diffusion generation module demonstrated superior ability to generate multi-defect combination samples. Evaluation showed that 92.3% of the generated samples were considered genuine and reliable, and the accuracy of defect correlation in the generated samples reached 89.7%. For rare defect combinations with fewer than 50 samples, the detection accuracy improved by 21.4%, and the overall detection accuracy improved by 12.6% after training with the generated samples. Accuracy, recall, and F1 score were used to evaluate the detection performance of the method for different modality combinations, and the results are shown in Table 2.

[0133] Table 2: Detection performance of the method of the present invention for different modal combinations

[0134]

[0135] The defect combination condition vector in this embodiment includes defect type, defect severity, defect location, and interaction relationship between different defects. Based on the defect combination condition, a condition control mechanism is adopted to gradually remove noise from the noise samples during the forward process of the hypergraph diffusion model, thereby generating multimodal defect samples.

[0136] Example 3

[0137] like Figure 1 As shown, this embodiment provides a collaborative diagnostic method for multiple defects in PCBA that integrates hypergraph diffusion and cross-modal comparison. This embodiment describes the differences from Embodiment 1, based on Embodiment 1. The method includes:

[0138] S1: Construct a multi-layered hypergraph structure, which includes a node layer, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges;

[0139] S2: Construct a conditional hypergraph diffusion model, and generate multimodal defect samples based on the multi-layer hypergraph structure and the conditional hypergraph diffusion model;

[0140] S3: Employ a multi-layer alignment mechanism to perform cross-modal feature alignment on the multimodal defect samples to obtain aligned features;

[0141] S4: Calculate the defect interaction matrix of the aligned features; the defect interaction matrix is ​​used to model the mutual influence relationship between different defect types;

[0142] S5: Based on the defect interaction matrix, update the node features of each node in the node layer using the message passing mechanism of a graph neural network;

[0143] S6: Input the node features into the multi-task learning framework to predict the defects of each node and obtain the node defect prediction results.

[0144] The multi-layer alignment mechanism is used to perform cross-modal feature alignment on the multimodal defect samples to obtain aligned features, including:

[0145] The following formula is used to perform cross-modal feature alignment on the multimodal defect samples:

[0146] ;

[0147] Among them, L semantic f represents the semantic alignment function. i f represents the feature of the defect sample in the i-th mode. j Let `project` represent the features of the defect sample in the j-th modality. `project` represents the projection function. The L2 norm of the difference between two projection function results is used to measure the difference between defect samples from different modalities. This process iterates through all modalities to achieve cross-modal feature alignment of multimodal defect samples. The projection function `project` uses Q-Former, a lightweight Transformer architecture whose core function is to dynamically project visual features into the language space using learnable query vectors. Q-Former actively filters visual features through learnable query vectors, preserving more fine-grained information compared to static linear projection.

[0148] The following formula is used to align the structures of different modes:

[0149] ;

[0150] Among them, L structure A represents the structure alignment function. i Let A represent the attention matrix for the i-th modality. j Let F represent the attention matrix for the j-th modality, and let F denote the Frobenius norm.

[0151] The following formula is used to align the feature distributions of different modes:

[0152] ;

[0153] Among them, L distributionThe feature distribution alignment function is represented by KL, which represents the divergence operation, and P... i P represents the characteristic distribution of mode i. j Represents the characteristic distribution of mode j. Indicates from P i To P j divergence, Indicates from P j To P i The divergence.

[0154] This invention designs three alignment layers and employs three alignment mechanisms to achieve cross-modal feature alignment. The semantic alignment layer achieves alignment by minimizing the semantic distance between features of different modalities, the structural alignment layer achieves consistency of feature structures by aligning attention structures, and the distribution alignment layer achieves alignment of feature distributions through KL divergence constraints.

[0155] The Frobenius norm, defined as the square root of the sum of the squares of all elements in a matrix, is used to measure the difference between attention matrices of different modalities. KL divergence, or relative entropy, is used to measure the asymmetry of the difference between two probability distributions. Distribution P used for measurement j Approximate distribution P i Information loss at that time Distribution P used for measurement i Approximate distribution P j Information loss during the process.

[0156] Cross-modal alignment establishes mapping relationships between different modalities in a shared feature space, enabling the model to establish the interrelationships and influences between defects of different modalities. For example, a defect of type i PCBA may lead to a defect of type j PCBA, thereby significantly improving the model's generalization ability and understanding depth.

[0157] The calculation of the defect interaction matrix of the aligned features includes:

[0158] Calculate the defect interaction matrix using the following formula:

[0159] ;

[0160] Among them, M ij Let represent the feature matrix of the defect sample in the i-th mode and the defect interaction matrix between the features of the defect sample in the j-th mode; sigmoid represents the sigmoid activation function, and * represents element-wise multiplication. The transpose of the interaction weight matrix is ​​used in this invention to model the mutual influence between different defect types.

[0161] The step of updating the node features of each node in the node layer using the message passing mechanism of a graph neural network based on the defect interaction matrix includes:

[0162] The node features of each node in the node layer are updated using the following formula:

[0163] ;

[0164] ;

[0165] in, Represents node i at time t. Let j and e be nodes at time t. ij This represents the edge connecting node i and node j; Message represents the message passing mechanism. This represents the message transmitted from node i to node j at time t+1; Aggregate represents the aggregation operation, N i Let i represent the set of neighboring nodes, and Update represent the node update operation. Let i represent node i at time t+1.

[0166] Preferably, before calculating the defect interaction matrix, a multi-head graph attention mechanism is designed. This mechanism utilizes multiple attention heads to learn the correlations between different nodes in a multi-layer hypergraph structure in parallel, enhancing the model's expressive power to achieve collaborative reasoning among defects. The multi-head graph attention mechanism can capture the correlation information between defects from different representation subspaces, and the attention weights are calculated using the following formula:

[0167] ;

[0168] in, For the attention between nodes i and j in a multi-layered hypergraph structure, LeakyReLU is a modified linear unit with leakage. W is the transpose of the attention parameter matrix. h For the shared weight matrix, h i h is the i-th node in a multi-layered hypergraph structure. j Let j be the j-th node in the multi-layer hypergraph structure. This is a concatenation operation of feature vectors.

[0169] The i-th node in the multi-layer hypergraph structure is updated using the following formula:

[0170] ;

[0171] ;

[0172] Where Htotal represents the number of attention heads. Let be the set of neighboring nodes of node i, and let Concat be the attention header concatenation operation. As the first point of attention, For the Htotal attention head, For the h-th attention head, To output the weight matrix, This refers to the i-th updated node in a multi-layered hypergraph structure. It is a non-linear activation function.

[0173] The graph neural network in this embodiment uses a message passing mechanism to iteratively update node features, thereby capturing the structural information of the graph and the high-order correlation between different defects. The message passing mechanism includes the following three steps: (1) Message sending: Each node sends a message to its neighboring nodes according to its own features, that is, it performs a linear or nonlinear transformation on the features of one node and sends the transformation result to another node. (2) Message aggregation: The receiving node aggregates messages from neighboring nodes through a permutation invariant function. The permutation invariant function is a summation, averaging, or maximum value operation. For example, if node B is taken as the central node, and nodes A and C are connected to node B respectively, a nonlinear transformation is performed on the features of node A to obtain the transformed features of node A, and a nonlinear transformation is performed on the features of node C to obtain the transformed features of node C. The transformed features of node A and node C are passed to node B, and node B aggregates the features from nodes A and C. (3) Feature updating: The receiving node combines the aggregated message with the features of its own node. The combination method is weighted summation or activation using a nonlinear activation function to update the features of the receiving node. For example, after node B aggregates the features of nodes A and C at time t, it performs a weighted summation of the aggregated features and the features of node B at time t, and uses the weighted summation result as the feature of node B at time t+1.

[0174] Example 4

[0175] like Figure 1 As shown, this embodiment provides a collaborative diagnostic method for multiple defects in PCBA that integrates hypergraph diffusion and cross-modal comparison. This embodiment describes the differences from Embodiment 1, based on Embodiment 1. The method includes:

[0176] S1: Construct a multi-layered hypergraph structure, which includes a node layer, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges;

[0177] S2: Construct a conditional hypergraph diffusion model, and generate multimodal defect samples based on the multi-layer hypergraph structure and the conditional hypergraph diffusion model;

[0178] S3: Employ a multi-layer alignment mechanism to perform cross-modal feature alignment on the multimodal defect samples to obtain aligned features;

[0179] S4: Calculate the defect interaction matrix of the aligned features; the defect interaction matrix is ​​used to model the mutual influence relationship between different defect types;

[0180] S5: Based on the defect interaction matrix, update the node features of each node in the node layer using the message passing mechanism of a graph neural network;

[0181] S6: Input the node features into the multi-task learning framework to predict the defects of each node and obtain the node defect prediction results.

[0182] This invention employs contrastive learning to optimize node representations in multi-layer hypergraph structures. It designs instance-level, category-level, and relation-level contrastive loss functions, and calculates the instance-level contrastive loss function using the following formula:

[0183] ;

[0184] in, Let represent the instance-level contrastive loss function, where log represents the logarithmic function with base 1, and exp represents the exponential function. This represents a sample with the i-th type of defect. This represents a positive sample of the i-th type of defect. This represents a sample with the k-th type of defect. The temperature parameter is represented by an instance-level contrastive learning loss function used to bring positive sample pairs closer together and push negative sample pairs further apart, thereby learning the similarity between samples.

[0185] The category-level contrastive learning loss function is calculated using the following formula:

[0186] ;

[0187] in, This represents the category-level contrastive learning loss function. Indicates the i-th type of defect. This indicates a defect type that is highly similar to the i-th type of defect. Let represent the j-th type of defect. The category-level contrastive learning loss function is used to enhance the ability to distinguish defect categories.

[0188] Calculate the relation-level contrastive learning loss function using the following formula:

[0189] ;

[0190] in, This represents the relational contrastive learning loss function. This represents the relationship between the i-th type of defect and the j-th type of defect. Indicates and Relationship features with high similarity This represents the relationship between the k-th type defect and the l-th type defect.

[0191] The cross-modal contrastive learning module designs a unified cross-modal feature representation space. Through advanced contrastive learning strategies, it learns the common and dissimilar features of data from different modalities, enhancing the model's ability to discriminate complex defect patterns and represent features. This module can integrate information from multiple modalities, including optical, X-ray, thermal imaging, and electrical testing, achieving multimodal information fusion.

[0192] This embodiment dynamically adjusts temperature parameters based on real-time feedback from the production line to balance detection accuracy and false alarm rate. ,in, Let η be the target false positive rate and η be the learning rate. Let be the false alarm rate of the i-th type of PCBA defect at time t. Let be the temperature parameter of the i-th type of PCBA defect at time t. Let be the temperature parameter of the i-th type of PCBA defect at time t+1. Furthermore, this embodiment employs a progressive knowledge distillation strategy to train the student network, transferring the functionality of the teacher network constructed in steps S1-S6 to the student network. This allows the student network to be lightweight while accurately identifying different types of PCBA defects. The student network is trained using the following formula: ,in, The first equilibrium parameter, The second equilibrium parameter, The third equilibrium parameter, For the task loss function, For the knowledge distillation loss function, For feature matching loss function, This is the total loss function.

[0193] Example 5

[0194] This embodiment provides a PCBA multi-defect collaborative diagnostic device that integrates hypergraph diffusion and cross-modal comparison, including:

[0195] A multi-layer hypergraph structure construction module is used to construct a multi-layer hypergraph structure, which includes a node layer, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges.

[0196] A multimodal defect sample generation module is used to construct a conditional hypergraph diffusion model and generate multimodal defect samples based on the multi-layer hypergraph structure and the conditional hypergraph diffusion model.

[0197] A cross-modal feature alignment module is used to perform cross-modal feature alignment on the multimodal defect samples using a multi-layer alignment mechanism to obtain aligned features;

[0198] The defect interaction matrix calculation module is used to calculate the defect interaction matrix of the aligned features; the defect interaction matrix is ​​used to model the mutual influence relationship between different defect types.

[0199] The node feature update module is used to update the node features of each node in the node layer according to the defect interaction matrix and using the message passing mechanism of the graph neural network.

[0200] The defect prediction module is used to input the node features into the multi-task learning framework, predict the defects of each node, and obtain the node defect prediction results.

[0201] The SMT solder joint detection device based on hypergraph comparison learning in this embodiment is used to execute the PCBA multi-defect collaborative diagnosis method that integrates hypergraph diffusion and cross-modal comparison in any one of the embodiments 1-4.

[0202] This embodiment also provides a computer device, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor in this computer design provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communication with external terminals via a network connection.

[0203] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements a collaborative diagnosis method for multiple defects in PCBAs by fusing hypergraph diffusion and cross-modal comparison. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0204] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0205] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A collaborative diagnostic method for multiple defects in PCBAs that integrates hypergraph diffusion and cross-modal comparison, characterized in that, include: Construct a multi-layered hypergraph structure, which includes a node layer, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges; A conditional hypergraph diffusion model is constructed, and multimodal defect samples are generated based on the multi-layer hypergraph structure using the conditional hypergraph diffusion model. A multi-layer alignment mechanism is used to perform cross-modal feature alignment on the multimodal defect samples to obtain aligned features; Calculate the defect interaction matrix of the aligned features; the defect interaction matrix is ​​used to model the mutual influence relationship between different defect types. Based on the defect interaction matrix, the node features of each node in the node layer are updated using the message passing mechanism of a graph neural network; The node features are input into a multi-task learning framework to predict the defects of each node, thus obtaining the node defect prediction results. The construction of the multi-layer hypergraph structure includes: Treat each detection area on the PCBA as a node; Obtain multimodal feature information for the region corresponding to each node; the multimodal feature information includes optical features, X-ray features, thermal features, and electrical features; The multimodal feature information is converted into node feature vectors; All the node feature vectors are combined into a node layer; Hyperedges are constructed based on the relationships between different nodes in the node layer to obtain a multi-layered hypergraph structure.

2. The PCBA multi-defect collaborative diagnosis method based on fusion of hypergraph diffusion and cross-modal comparison as described in claim 1, characterized in that, The process of constructing hyperedges based on the relationships between different nodes in the node layer to obtain a multi-layer hypergraph structure includes: Construct causal hyperedges based on the following formula: ； Among them, e causal V represents the causal relationship hyperedge. i Let v represent the feature vector of the i-th node. j Let (v) represent the feature vector of the j-th node. i ,v j () represents the superedge between the i-th node and the j-th node; defect i Denotes the i-th type of defect, defect j Indicates the j-th type of defect. This indicates that the i-th type of defect leads to the j-th type of defect; The node layer, spatially related hyperedges, functionally related hyperedges, and causally related hyperedges form a multi-layered hypergraph structure.

3. The PCBA multi-defect collaborative diagnosis method based on fusion of hypergraph diffusion and cross-modal comparison as described in claim 1, characterized in that, The generation of multimodal defect samples based on the multi-layer hypergraph structure and the conditional hypergraph diffusion model includes: In the forward propagation process of the hypergraph diffusion model, noise is propagated through hyperedges; In the reverse process of the hypergraph diffusion model, the multi-layer hypergraph structure is used as a condition to remove noise from the noise samples and generate multimodal defect samples.

4. The PCBA multi-defect collaborative diagnosis method based on fusion of hypergraph diffusion and cross-modal comparison as described in claim 3, characterized in that, The step of using the multi-layer hypergraph structure as a condition to remove noise from noise samples and generate multimodal defect samples includes: Based on the following formula, a hypergraph convolutional denoising network is used to remove noise from noisy samples: ； ； HyperGCN represents a hypergraph convolutional denoising network, x t Let t represent the noise sample at time t, where t represents time. The term "feature concatenation operation" is defined here; "embed(t)" represents multi-scale temporal embedding; "C" represents the defect combination condition vector; and "embed(C)" represents the defect combination condition vector embedding. H represents the multi-layer hypergraph structure. Represents a multimodal defect sample, c defect Indicates the defect type, c severity Indicates the severity of the defect, c location Indicates the location of the defect, c interaction This indicates the interaction between different defects.

5. The PCBA multi-defect collaborative diagnosis method based on fusion of hypergraph diffusion and cross-modal comparison as described in claim 1, characterized in that, The multi-layer alignment mechanism is used to perform cross-modal feature alignment on the multimodal defect samples to obtain aligned features, including: The following formula is used to perform cross-modal feature alignment on the multimodal defect samples: ； Among them, L semantic f represents the semantic alignment function. i f represents the feature of the defect sample in the i-th mode. j Let represent the features of the defect sample in the j-th mode, and let project represent the projection function; The following formula is used to align the structures of different modes: ； Among them, L structure A represents the structure alignment function. i Let A represent the attention matrix for the i-th modality. j Let F represent the attention matrix for the j-th modality, and let F denote the Frobenius norm. The following formula is used to align the feature distributions of different modes: ； Among them, L distribution The feature distribution alignment function is represented by KL, which represents the divergence operation, and P... i P represents the characteristic distribution of mode i. j Represents the characteristic distribution of mode j. Indicates from P i To P j divergence, Indicates from P j To P i The divergence.

6. The PCBA multi-defect collaborative diagnosis method based on fusion of hypergraph diffusion and cross-modal comparison according to claim 5, characterized in that, The calculation of the defect interaction matrix of the aligned features includes: Calculate the defect interaction matrix using the following formula: ； Among them, M ij Let represent the defect interaction matrix between the features of the defect sample in the i-th mode and the features of the defect sample in the j-th mode; sigmoid represents the sigmoid activation function, and * represents element-wise multiplication. This represents the transpose of the interaction weight matrix.

7. The PCBA multi-defect collaborative diagnosis method based on fusion of hypergraph diffusion and cross-modal comparison according to claim 1, characterized in that, The step of updating the node features of each node in the node layer using the message passing mechanism of a graph neural network based on the defect interaction matrix includes: The node features of each node in the node layer are updated using the following formula: ； ； in, Represents node i at time t. Let j and e be nodes at time t. ij This represents the edge connecting node i and node j; Message represents the message passing mechanism. This represents the message transmitted from node i to node j at time t+1; Aggregate represents the aggregation operation, N i Let i represent the set of neighboring nodes, and Update represent the node update operation. Let i represent node i at time t+1.

8. The PCBA multi-defect collaborative diagnosis method based on fusion of hypergraph diffusion and cross-modal comparison as described in claim 1, characterized in that, The step of inputting the node features into a multi-task learning framework to predict the defects of each node and obtain node defect prediction results includes: The following formula is used to generate the node defect prediction result based on node i: ； ； ； in, This represents the predicted probability of the i-th type of defect occurring in the PCBA. Represents a non-linear activation function. h represents the transpose of the weight matrix of node i. i Let b represent the i-th node. i This represents the bias matrix of the i-th node with respect to the defect type; Let represent the predicted probability of the severity of the i-th type of defect in the PCBA, where softmax is the normalized exponential function. b is the transpose of the weight matrix representing the severity of the i-th defect. s The bias matrix representing the i-th node with respect to the severity of the defect; This represents the predicted probability of the correlation between the i-th defect and the j-th defect in a PCBA. b is the transpose of the weight matrix representing the correlation between the i-th defect and the j-th defect. int The bias matrix represents the correlation between the i-th defect and the j-th defect.

9. The PCBA multi-defect collaborative diagnosis method based on fusion of hypergraph diffusion and cross-modal comparison according to claim 4, characterized in that, The calculation formula for the multi-scale temporal embedding is as follows: ； in, This represents multi-scale temporal embedding, where sin represents the sine function, cos represents the cosine function, i represents the dimension index, and d represents the model embedding dimension.

Images