PCBA circuit board welding spot detection method based on multi-modal data fusion

Abstract

The invention discloses a PCBA circuit board welding spot detection method based on multi-modal data fusion, and relates to the technical field of electronic manufacturing quality detection.The PCBA circuit board welding spot detection method comprises the steps that a distributed sensing network is constructed, multi-modal data are collected, welding spot information is obtained in an omnibearing mode, and time-space alignment of the multi-modal data is carried out; performing feature extraction on the multi-modal data, dynamically weighting each modal feature through an attention mechanism, and highlighting key defect characterization; a welding spot spatial topological graph is constructed by using a graph neural network, and a spatial relationship between welding spots is modeled. By integrating optical, X-Ray, thermal, mechanics, electricity and other multi-dimensional data, the information limitation of single-mode detection is broken through, the complementation of different mode data is utilized, the attention mechanism is combined to dynamically weight each mode feature, the complex defect is accurately identified, the graph neural network is utilized to model the welding spot space topological relation, the associated defect is further captured, and the defect detection accuracy is improved. And the defect classification accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic manufacturing quality inspection, and in particular to a PCBA circuit board solder joint inspection method based on multimodal data fusion. Background Art

[0002] PCBA (Printed Circuit Board Assembly) circuit board solder joint inspection is a critical link in the electronics manufacturing process. Solder joints are the core electrical connection between electronic components and PCB boards. Their quality directly affects the long-term reliability of the product. Defects such as cold solder joints, cold solder joints, and bridging can lead to poor contact, signal interference, and even equipment failure. Especially in high-reliability fields such as automotive electronics, aerospace, and medical equipment, solder joint defects may cause serious safety hazards. Electronic products need to adapt to extreme environments such as high temperature, high humidity, and vibration. The quality of solder joints must meet long-term stability requirements to avoid failure due to environmental stress.

[0003] In the existing technology, single-modal detection is mostly used to detect solder joints on PCBA circuit boards. Since it is based on single sensor data and lacks correlation analysis of multimodal data, it leads to a high rate of defect misjudgment and poor adaptability to complex defects. Manual intervention is required to adjust parameters. Therefore, how to use distributed sensor networks to obtain multimodal data, extract the complementary features of each modal data, introduce graph neural networks to model the spatial relationship between solder joints, and identify the associated defects of solder joints is the problem to be solved by the present invention. To this end, a PCBA circuit board solder joint detection method based on multimodal data fusion is proposed. Summary of the Invention

[0004] The present invention aims to provide a PCBA circuit board solder joint detection method based on multimodal data fusion to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A PCBA circuit board solder joint detection method based on multimodal data fusion includes the following steps:

[0007] Step 1: Build a distributed sensor network and collect multimodal data to obtain all-round solder joint information. Then, use the PTP (Pre-processing Time Protocol) to align the multimodal data in time and space.

[0008] Step 2: Extract features from multimodal data and dynamically weight the features of each modality through the attention mechanism to highlight key defect representations;

[0009] Step 3: Use a graph neural network to construct a spatial topology map of solder joints, model the spatial relationship between solder joints, and input the graph neural network to learn the contextual features of associated defects, identify associated defects of solder joints, and improve detection accuracy;

[0010] Step 4: Design an independent sub-detector for each mode, perform detection based on the characteristics of different modal data, output the defect probability, and integrate the results of each sub-detector through DS evidence theory to comprehensively judge the solder joint status;

[0011] Step 5: Build a reinforcement learning model based on historical detection data, use defect classification accuracy as the reward function, and dynamically adjust the trust weight of each modality sub-detector to achieve adaptive optimization;

[0012] Step 6: Output the defect type and location based on the comprehensive fusion results and generate a visual report.

[0013] A further improvement of the technical solution of the present invention is that: the step 1 specifically includes:

[0014] Build a distributed sensing network that covers all dimensions of solder joints, perform multi-physics field synchronous sensing, and assign a unique identifier to each sensor. The distributed sensing network includes optical modules, X-ray modules, thermal modules, mechanical modules, and electrical modules.

[0015] Using a star-bus hybrid architecture, the main controller connects to the optical, X-ray, and thermal master nodes via Ethernet. Each master node then extends the mechanical and electrical sub-nodes via the CAN bus to ensure low-latency data transmission. High-precision calibration plates are used to calibrate the optical and X-ray spatial coordinate systems. A pixel-to-physical coordinate conversion matrix is established through feature point matching. Synchronous pulse signals are used to trigger all sensors to collect full-modal data to verify the spatiotemporal alignment benchmark.

[0016] Multimodal data time synchronization is performed based on the PTP protocol. The main controller (PLC) is designated as the PTP master clock. It sends synchronization messages to each sensor sub-node using the IEEE 1588-2008 protocol. Each sensor sub-node acts as a slave clock. The slave clock calculates the link delay based on the timestamp in the message and adjusts the local clock offset. A high-speed oscilloscope is then used to synchronously capture the trigger signal of the optical camera and the frame synchronization signal of the X-ray detector to verify that the time deviation is less than 1μs.

[0017] Data from different perspectives and modalities are mapped to a unified PCB coordinate system to construct a 3D holographic model of the solder joint. The ICP (Iterative Closest Point) algorithm is used to match feature points on the calibration board, calculate the rigid body transformation matrix from the optical coordinate system to the X-Ray coordinate system, and superimpose the solder joint contours detected by optical inspection with the internal voids of the solder balls detected by X-Ray inspection to generate a "shell-core" composite model. The temperature gradient of the thermal imaging, the stress distribution of the mechanical sensor, and the impedance value of the electrical test are then annotated in the 3D model to form a thermal map of the solder joint health status.

[0018] A further improvement of the technical solution of the present invention is that: the step 2 specifically includes:

[0019] Feature extraction is performed on multimodal raw data including optical, X-ray, thermal, mechanical and electrical data. The modal features are aligned to a unified time axis through PTP synchronized timestamps. Bilinear interpolation is used to account for spatial scale differences. All features are mapped to the PCB physical coordinate system (with an accuracy of 0.01mm) to generate five sets of modal feature vectors. , dimensions are 512, 128, 64, 32, and 48 respectively;

[0020] Combine the five sets of features into joint features , with a dimension of 784, and then analyze the correlation between modalities through a multi-head self-attention mechanism (8 heads, 64 dimensions per head), concatenate and linearly transform the 8-head attention outputs, and generate a modal weight vector , and then output weighted features to highlight defect-sensitive modes;

[0021] The Grad-CAM algorithm is applied to the weighted features, and the defect classification loss is back-propagated to the feature layer to generate a heat map. The heat map is superimposed on the original optical image to intuitively display the defect location. The weighted features are input into the fully connected layer (256 dimensions) and the Softmax classifier to output the defect type and confidence.

[0022] A further improvement of the technical solution of the present invention is that: the step 3 specifically includes:

[0023] Based on the physical layout of the PCB and the actual locations of the solder joints, a spatial topology graph of the solder joints is constructed. Each solder joint is considered a node in the spatial topology graph, and the physical connections or spatial proximity between solder joints are considered edges. By extracting the coordinate information and connection relationships of the solder joints, the adjacency matrix and node feature matrix of the graph are generated.

[0024] The constructed solder joint spatial topology map is input into the graph neural network, which aggregates the neighborhood information of each solder joint node and updates the node feature representation. A two-layer GCN is used, each layer including graph convolution, batch normalization, and ReLU activation. The first layer GCN aggregates first-order neighborhood information, and the second layer GCN aggregates second-order neighborhood information, outputting 128-dimensional contextual features. A graph attention layer (GAT) is introduced after the GCN to dynamically calculate the weights of neighboring nodes, thereby expanding the solder joint features from local unimodal information to global contextual information.

[0025] Based on contextual features, the defect type and correlation of solder joints are predicted. A fully connected layer (256 dimensions) + Softmax is applied to output the defect type and confidence level. Contrastive learning loss is introduced to reduce the feature distance between solder joints with similar defects, bringing them closer together in the feature space. The feature distance between solder joints with different defect types is increased, separating them from each other in the feature space. Grad-CAM is then used to generate a graph-level heat map to highlight key neighborhood solder joints.

[0026] A further improvement of the technical solution of the present invention is that the process of predicting the type and correlation of solder joint defects is as follows:

[0027] The contextual features of each solder joint (128-dimensional features extracted by a graph neural network) are processed to predict the defect type and relevance of the solder joint. The contextual features are input into a fully connected layer (256 dimensions), where they are nonlinearly transformed to extract a higher-level feature representation. A Softmax classifier is then used to map the features to different defect categories, outputting the confidence score for each category. The Softmax classifier is then used to generate a probability distribution for each solder joint, indicating the probability of the solder joint belonging to different defect types.

[0028] By using contrastive learning loss, the feature distances between similar defective solder joints are shortened, while those between different types are further apart. This allows similar defective solder joints to cluster in the feature space, while those of different types are dispersed. This helps improve the accuracy of defect classification and enhances the model's ability to identify solder joint correlations.

[0029] The Grad-CAM algorithm is used to generate a graph-level heat map. The classification loss is back-propagated to the feature layer, and the contribution of each feature channel to defect classification is calculated. The generated graph-level heat map is then used to highlight the neighboring solder joints that have the greatest impact on defect classification, visually demonstrating the correlation between solder joints.

[0030] A further improvement of the technical solution of the present invention is that: the step 4 specifically includes:

[0031] Based on the characteristics of different modal data, independent sub-detectors are designed. Each sub-detector independently outputs the defect probability of the solder joint according to the characteristics of its modal data and defect mode.

[0032] The defect probability output by each modal sub-detector is considered as evidence and fused using the DS (Dempster-Shafer) evidence theory. By defining the basic probability assignment (BPA), the defect probability output by each sub-detector is converted into a BPA as evidence support for the solder joint status. The evidence of different modalities is fused using the combination rules of the DS evidence theory to calculate the comprehensive BPA and analyze the solder joint defect situation.

[0033] Based on the fused BPA and solder joint defect analysis results, the comprehensive status judgment results of the solder joint are output, including defect type, comprehensive defect probability and confidence information.

[0034] A further improvement of the technical solution of the present invention is that the process of fusing evidence of different modalities is as follows:

[0035] The 0-1 defect probability output by each modal sub-detector is converted into the basic probability allocation (BPA) required by DS theory, and an evidence support framework is constructed. In this framework, the BPA structure is designed to define the identification framework of the solder joint status, including: defective, non-defective and uncertain. The defect probability output by the sub-detector is divided into three parts, namely defect support, defect support, and defect support. , defect-free support and uncertainty ;

[0036] Starting from the BPA of any two modalities, calculate the new combined BPA. If a conflict occurs, calculate the product of the BPAs of the two modalities to obtain the conflict probability, and redistribute the conflict probability according to the original support ratio of the two modalities to generate a new BPA. Combine the result of the previous step with the BPA of the third modality, repeat the conflict processing and redistribution until all modalities are integrated to generate the final comprehensive BPA.

[0037] Based on the integrated BPA after fusion, the defect support value in the integrated BPA is directly taken and the confidence, that is, 1-uncertainty, is calculated. At the same time, dual thresholds are set, namely the defect threshold and the non-defect threshold. When the defect threshold is defect support > 0.8 and confidence > 0.9, it is judged as defective. When the non-defect threshold is defect support < 0.3 or confidence < 0.7, it is judged as non-defective. Other cases are marked as "pending re-inspection" to trigger manual review or supplementary inspection.

[0038] A further improvement of the technical solution of the present invention is that: the step 5 specifically includes:

[0039] The multimodal inspection system is modeled as a reinforcement learning environment. The state space is designed, the action space is defined, and the reward function is constructed. The state vector of the state space contains: the current performance of each modality, the modal conflict level, and the solder joint feature distribution. The current performance of each modality is the defect classification accuracy of the optical / X-Ray / thermal sub-detector in the past 10 inspections. The modal conflict level is the average BPA conflict probability between modalities in the last five fusions. The solder joint feature distribution is the static attributes of the current batch of solder joints, such as size and material type. The action is defined as the adjustment amount to the trust weight of each modality. The core reward is the comprehensive defect classification accuracy after fusion. If the accuracy of the current round improves by 2% compared with the previous round, the reward is +0.5. If the accuracy decreases by 2%, the penalty is -0.3. If the conflict rate decreases by 5%, an additional reward of +0.2 is received.

[0040] Production line inspection data from the past six months was collected and divided into training and validation sets. A reinforcement learning model was constructed using the Deep Q-Network (DQN) structure, incorporating an experience replay mechanism to store the historical state-action-reward-new state quadruple. Dual DQN technology was also introduced to separate the target network and evaluation network, stabilize Q-value estimation, and set training termination conditions.

[0041] The trained reinforcement learning model is deployed to the production line for real-time weight optimization and continuous system improvement. After inspecting every 100 solder joints, the reinforcement learning model outputs an action based on the current state and applies the action to update the weights of each modality. Weight adjustment limits are pre-designed, and the single adjustment range does not exceed 20% of the initial weight. If the accuracy rate drops for three consecutive rounds after adjustment, it will automatically roll back to the previous round of weight configuration. At the same time, the historical database is updated monthly to include the latest inspection data and manual re-inspection results.

[0042] A further improvement of the technical solution of the present invention is that the construction process of the reinforcement learning model is:

[0043] Six months of inspection records were extracted from the production line database, including the original inspection results, manual re-inspection labels, and initial weight configurations. A state vector was constructed to analyze modal performance and conflict levels. Modal performance was calculated by calculating the average accuracy of each modality over the past 10 inspections, and conflict level was calculated by calculating the average BPA conflict probability between modalities in the most recent five fusions. The data was then divided into a training set (the first 80%) and a validation set (the last 20%) in chronological order, generating a 15-dimensional state vector and a complete reward calculation basis.

[0044] A reinforcement learning model for learning the optimal weight adjustment strategy is constructed using the DQN structure. The network architecture is designed, including input layer, hidden layer, and output layer. The input layer is the state vector, and the hidden layer is a two-layer fully connected layer (128+64 neurons, ReLU activation). BatchNorm is introduced to accelerate convergence. The output layer is the Q value of 5 actions (corresponding to the weight adjustment amount of 5 modes). Combined with the experience replay mechanism, 100,000 historical quadruple data are stored, and random sampling breaks the data correlation. The dual DQN mechanism is used, and the evaluation network is used to calculate the current Q value, and the target network is used to calculate the target Q value. The target network parameters are synchronized every 100 rounds to stabilize the training process. The training termination condition is set as the fluctuation of the verification set reward <0.05 for 10 consecutive rounds, thus obtaining a fully trained reinforcement learning model.

[0045] A further improvement of the technical solution of the present invention is that: the step 6 specifically includes:

[0046] Comprehensively analyze the results of multimodal data fusion to identify the defect type and specific location of the solder joint. Generate a detailed visual report based on the fused data, presenting the inspection results in an intuitive manner while marking the location and type of the defect.

[0047] Review the inspection results to identify cases of false positives (normal solder joints mistakenly identified as defective) and missed detections (defective solder joints mistakenly identified as normal). Record these cases, including the solder joint's multimodal raw data, the model's inspection results, and the actual status after manual review. Conduct in-depth analysis of these false positives and missed detections to identify the causes of the misjudgments.

[0048] The data of false positive / missed positive cases are fed back into the training set of the model as new training samples for retraining and optimizing the model. By introducing false positive / missed positive cases, the model learns the feature patterns that it fails to correctly identify. Through continuous iterative optimization, the detection performance of the model is improved, forming a closed-loop system from detection to feedback to optimization, ensuring the long-term stability and accuracy of the detection system.

[0049] Due to the adoption of the above technical solution, the present invention has the following technical advancements compared to the prior art:

[0050] 1. The present invention provides a PCBA circuit board solder joint detection method based on multimodal data fusion. By integrating multi-dimensional data such as optics, X-rays, thermals, mechanics, and electricity, it breaks through the information limitations of single-modal detection, utilizes the complementarity of different modal data, and combines the attention mechanism to dynamically weight the features of each modality to accurately identify complex defects. It also uses graph neural networks to model the spatial topological relationship of solder joints, further capturing associated defects and improving the accuracy of defect classification.

[0051] 2. The present invention provides a PCBA circuit board solder joint detection method based on multimodal data fusion. By introducing a reinforcement learning model and taking defect classification accuracy as the reward function, the trust weight of each modal sub-detector is dynamically adjusted. By analyzing the modal performance, conflict level and solder joint feature distribution in historical detection data, the detection strategy is optimized in real time, so that the detection system can automatically adjust its own detection strategy according to the data changes and defect pattern changes in actual production, thereby always maintaining a high detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0053] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0054] Figure 2 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] Example 1, as Figure 1 、 Figure 2 As shown, the present invention provides a PCBA circuit board solder joint detection method based on multimodal data fusion, comprising the following steps:

[0057] Step 1: Build a distributed sensing network and collect multimodal data to obtain solder joint information in all directions, and perform spatiotemporal alignment of multimodal data through the time synchronization protocol (PTP) to ensure the synchronous capture of full-dimensional information of solder joints. Build a distributed sensing network covering full-dimensional information of solder joints, perform synchronous perception of multiple physical fields, and assign a unique identifier to each sensor. The distributed sensing network includes optical modules, X-Ray modules, thermal modules, mechanical modules, and electrical modules. For the optical module, a high-resolution industrial camera (visible light + infrared dual-channel) is deployed and installed above the SMT production line track to cover the PCB surface morphology and heat mark detection. For the X-Ray module, a micro-focus X-Ray source and a CMOS flat-panel detector are used, and the robot arm is moved to the key area of the PCB to achieve local penetration imaging. For the thermal module, an infrared thermal imager and a thermocouple array are integrated to monitor the dynamics of the welding process respectively. Temperature field and static thermal stress distribution. For the mechanical module, laser displacement sensors and strain gauges are embedded in the PCB fixture to measure the solder joint height, coplanarity and deformation caused by mechanical vibration in real time. For the electrical module, a multi-channel impedance analyzer is configured to contact the PCB test points with probes to collect electrical parameters such as contact resistance and capacitance. A star-bus hybrid architecture is adopted. The main controller connects the optical, X-Ray and thermal main nodes through Ethernet. Each main node then expands the mechanical and electrical sub-nodes through the CAN bus to ensure low-latency data transmission. A high-precision calibration plate is used to calibrate the optical and X-Ray spatial coordinate systems. The pixel-physical coordinate conversion matrix is established through feature point matching. The synchronous pulse signal is used to trigger all sensors to collect full-modal data once to verify the spatiotemporal alignment benchmark. The time synchronization of multi-modal data is performed based on the PTP protocol. The main controller (PLC) is designated as the PTP master clock. Through IEEE The 1588-2008 protocol sends synchronization messages to each sensor subnode, which acts as a slave clock. The slave clock calculates the link delay based on the timestamp in the message and adjusts the local clock offset. A high-speed oscilloscope is then used to synchronously capture the trigger signal from the optical camera and the frame synchronization signal from the X-ray detector to verify that the time deviation is less than 1 μs (which meets the requirements for dynamic monitoring of the welding process). This eliminates sensor sampling clock deviation and ensures the temporal consistency of dynamic defects. Data from different perspectives and modalities are mapped to a unified PCB coordinate system to construct a 3D holographic model of the solder joint. The ICP (Iterative Closest Point) algorithm is used to match feature points on the calibration board. The rigid body transformation matrix from the optical coordinate system to the X-ray coordinate system is calculated. The solder joint contours obtained from optical inspection are superimposed with the internal voids of the solder balls obtained from X-ray inspection to generate a "shell-core" composite model. The 3D model is then annotated with the temperature gradient from the thermal imaging, the stress distribution from the mechanical sensor, and the impedance value from the electrical test to form a heat map of the solder joint health.

[0058] The expression of the rigid body transformation matrix from the optical coordinate system to the X-Ray coordinate system is:

[0059] ;

[0060] ;

[0061] Where, is the rigid body transformation matrix, is the homogeneous coordinate in the X-Ray coordinate system, is the homogeneous coordinate in the optical coordinate system, is the rotation matrix, a 2×2 submatrix, describing the rotation relationship between coordinate systems; is the rotation parameter, is the translation vector, a 2×1 vector, describing the offset of the coordinate system origin; is a 1×2 zero vector to ensure that the matrix multiplication dimensions match. The transformation process is: Align the coordinate axis direction of the optical coordinate system to the X-Ray coordinate system by Move the origin of the optical coordinate system to the origin of the X-Ray coordinate system. The 1 at the end allows for uniform representation of rotation and translation.

[0062] Step 2: Extract features from multimodal data. Dynamically weight each modal feature through the attention mechanism to highlight key defect representations. Extract features from multimodal raw data including optical, X-ray, thermal, mechanical, and electrical data. Align each modal feature to a unified time axis through PTP synchronized timestamps. Use bilinear interpolation for spatial scale differences and map all features to the PCB physical coordinate system (with an accuracy of 0.01mm) to generate five sets of modal feature vectors. The dimensions are 512, 128, 64, 32, and 48 respectively, eliminating the spatial / temporal scale differences between modes. Among them, the optical feature uses the ResNet-50 convolutional network to extract the surface morphology features of the solder joint and output a 512-dimensional feature vector. The X-Ray feature uses 3D U-Net to segment the internal voids of the solder ball and extract 128-dimensional features including void volume and shape irregularity. The thermal feature applies the LSTM network to the infrared thermal imager time series data to capture the 64-dimensional dynamic features of the welding process including the slope and peak of the temperature curve. The mechanical feature uses PCA to reduce the dimension of the laser displacement sensor and strain gauge data to extract 32-dimensional static features including solder joint height deviation and coplanarity error. The electrical feature performs Fourier transform on the frequency domain response (1kHz~1MHz) of the impedance analyzer to extract 48-dimensional features including contact resistance and capacitance resonance point. The five groups of features are spliced into joint features. , Represents the optical characteristics, Represents X-Ray features, Represents thermal characteristics, Represents the mechanical characteristics, Represents electrical features with a dimension of 784. Then, the correlation between modalities is analyzed through a multi-head self-attention mechanism (8 heads, 64 dimensions per head). The 8-head attention output is concatenated and linearly transformed to generate a modal weight vector , represents the weight of the optical mode, Represents the weight of the X-Ray modality, represents the weight of the thermal mode, represents the weight of the mechanical mode, The weights representing the electrical modes are then output as weighted features to highlight the defect-sensitive modes. The Grad-CAM algorithm is applied to the weighted features, and the defect classification loss is back-propagated to the feature layer to generate a heat map. The heat map is superimposed on the original optical image to intuitively display the defect location. The weighted features are then input into the fully connected layer (256 dimensions) and the Softmax classifier to output the defect type and confidence level.

[0063] Step 3: Use graph neural network to construct solder joint spatial topology map, model the spatial relationship between solder joints, input graph neural network to learn contextual features of associated defects, identify associated defects of solder joints, improve detection accuracy, and construct solder joint spatial topology map based on the physical layout of PCB and the actual position of solder joints. Each solder joint is regarded as a node in the solder joint spatial topology map, and the physical connection or spatial proximity relationship between solder joints is regarded as an edge. By extracting the coordinate information and connection relationship of solder joints, the adjacency matrix and node feature matrix of the graph are generated. The constructed solder joint spatial topology map is input into the graph neural network, and the feature learning of solder joint nodes is performed through graph convolution operation to capture the local and global spatial relationships between solder joints. The system aggregates the neighborhood information of each solder joint node and updates the feature representation of the node. Two layers of GCN are used, each layer contains graph convolution, batch normalization and ReLU activation. The first layer of GCN aggregates the first-order neighborhood information, and the second layer of GCN aggregates the second-order neighborhood information. The output dimension is 128-dimensional context features. A graph attention layer (GAT) is introduced after GCN to dynamically calculate the weights of neighborhood nodes, thereby expanding the solder joint features from local unimodal information to global context information. Based on the context features, the solder joint defect type and correlation are predicted. A fully connected layer (256 dimensions) + Softmax is applied to output the defect type and confidence. Contrastive learning loss is introduced to shorten the feature distance of similar defective solder joints and push the features of different types of solder joints apart. Then, a graph-level heat map is generated through Grad-CAM to highlight key neighborhood solder joints.

[0064] In addition, the expression of the first-order neighborhood information aggregated by the first-layer GCN is as follows:

[0065] ;

[0066] Where, Represents the output feature matrix of the first layer of graph convolutional network, which is the node feature representation obtained after graph convolution operation and nonlinear activation function (ReLU) processing. is the activation function, is the degree matrix (DegreeMatrix). In the graph structure, the degree matrix is a diagonal matrix. is the degree matrix of The power is the inverse of the square root of each diagonal element of the degree matrix. The adjacency matrix with self-loops added represents the original adjacency matrix Describes the connection relationship between nodes in the spatial topology of the solder joint, and ,in is the identity matrix. Adding a self-loop means that each node is connected to itself. is the input feature matrix, each row of which corresponds to the feature vector of a node, is the trainable weight matrix of the first-layer graph convolutional network, which performs a linear transformation on the input features and its dimension is usually , is the feature dimension of the first layer output;

[0067] The second layer of GCN aggregates the second-order neighborhood information and outputs a context feature with a dimension of 128. Its expression is as follows:

[0068] ;

[0069] Where, is the output feature matrix of the second-layer graph convolutional network, is a set of real numbers, indicating that the elements in the characteristic matrix are all real numbers. is the number of nodes in the spatial topology graph representing the solder joint, It is the dimension of the output feature, indicating that after each node is processed by the second layer of GCN, its features are mapped to a 128-dimensional space;

[0070] The expression for dynamically calculating the weight of neighboring nodes is as follows:

[0071] ;

[0072] ;

[0073] Where, Represents the attention coefficient, which is used to measure the node For Node The importance weight of is the activation function, A small positive number is used to allow a small gradient when the input is negative, avoiding the "neuron death" problem caused by the ReLU function's gradient being zero in the negative region. is the attention parameter vector, which is a trainable parameter used to perform linear transformation on the concatenated node features to calculate the attention score. represents the attention parameter vector The transpose of is a weight matrix, which is a trainable parameter used to linearly transform the features of the node and map the node features to a new feature space in order to better calculate the attention. Indicates the input value. When , the output of the LeakyReLU function is ;when When , the output of the LeakyReLU function is , and They are the nodes output by the second layer of GCN and nodes The eigenvector of Represents the concatenation operation of vectors, that is It will and Concatenate two vectors into a new vector by column. For nodes The set of neighboring nodes;

[0074] The process of predicting solder joint defect types and correlations is as follows:

[0075] The contextual features of each solder joint (128-dimensional features extracted by the graph neural network) are processed to predict the defect type and correlation of the solder joint. The contextual features are input into a fully connected layer (256 dimensions), and the features are nonlinearly transformed to extract higher-level feature representations. The features are then mapped to different defect categories through the Softmax classifier, and the confidence of each category is output. The Softmax classifier is used to generate a probability distribution for each solder joint, indicating the probability that the solder joint belongs to different defect types. The feature distance of solder joints with similar defects is reduced by contrastive learning loss, so that solder joints with the same defect type are close in the feature space, and the feature distance of solder joints with different defect types is increased. Solder joints of different defect types are kept apart in the feature space. Specifically, during training, for pairs of solder joints with the same defect type, contrastive learning loss is used to penalize excessive distances between features. For pairs of solder joints with different defect types, close distances between features are penalized. This allows solder joints of the same type to cluster in the feature space, while solder joints of different types are dispersed. This helps improve the accuracy of defect classification and enhances the model's ability to identify solder joint correlations. A graph-level heat map is generated using the Grad-CAM algorithm. The classification loss is back-propagated to the feature layer, and the contribution of each feature channel to defect classification is calculated. The generated graph-level heat map is then used to highlight the neighboring solder joints that have the greatest impact on defect classification, visually demonstrating the correlation between solder joints.

[0076] The expression of contrast learning loss is as follows:

[0077] ;

[0078] Where, is the contrastive learning loss, For all sample pairs Perform the sum operation, Same type of label used to indicate samples and samples Belong to the same category, if the sample and samples Belong to the same category, then , if the sample and sample j belong to different categories, then , and The samples are and samples After the full connection layer (256 dimensions) and the attention feature vector before Softmax output, Represents the feature vector and The Euclidean distance between them is used to measure the difference between two samples in the feature space. Indicates when (i.e. sample and samples belong to different categories), ,when (i.e. sample and samples belong to the same category), , in conjunction with the maximum function, is used to impose losses on pairs of samples of different classes. is a pre-set threshold used to control the minimum distance between pairs of samples of different classes. and The characteristic distance Less than hour, As evidence, this item will produce a loss, which will increase the feature distance of samples of different types. When the feature distance is greater than or equal to When , this item is 0, and no additional loss occurs;

[0079] Step 4: Design an independent sub-detector for each mode, perform detection based on the characteristics of different modal data, output the defect probability, and integrate the results of each sub-detector through DS evidence theory to comprehensively judge the solder joint status;

[0080] Step 5: Build a reinforcement learning model based on historical detection data, use defect classification accuracy as the reward function, and dynamically adjust the trust weight of each modality sub-detector to achieve adaptive optimization;

[0081] In step 6, the defect type and location are output based on the comprehensive fusion results, a visual report is generated, and the false detection / missed detection cases are fed back to the model training set. The feature extraction and fusion strategies are continuously optimized iteratively to form a closed loop of detection-feedback-optimization.

[0082] Example 2, as Figure 1 、 Figure 2 As shown, based on Example 1, the present invention provides a technical solution: preferably, step 4 specifically includes:

[0083] According to the characteristics of different modal data, independent sub-detectors are designed respectively. Each sub-detector outputs the defect probability of the solder joint independently according to the characteristics of its modal data and defect mode. Among them, the optical sub-detector uses convolutional neural network (CNN) to analyze the surface morphology of the solder joint and output the defect probability. The X-ray sub-detector uses 3D The U-Net segmentation algorithm segments the internal structure of the solder ball, identifies defects such as internal voids, and outputs probabilities. The thermal sub-detector analyzes the temperature dynamic characteristics of the welding process based on the LSTM network and outputs the thermal defect probability. The mechanical sub-detector detects mechanical defects in the solder joint by analyzing strain and displacement data and outputs the probability. The electrical sub-detector identifies electrical parameter anomalies based on the impedance analysis results and outputs the probability. The defect probabilities output by each modal sub-detector are regarded as evidence and fused using the DS (Dempster-Shafer) evidence theory. By defining the basic probability assignment (BPA), the defect probability output by each sub-detector is converted into a BPA as evidence support for the solder joint status. The evidence of different modalities is fused using the combination rules of the DS evidence theory to calculate a comprehensive BPA. The solder joint defect situation is also analyzed. Based on the fused BPA and the analysis results of the solder joint defect situation, the comprehensive solder joint status judgment result is output, including defect type, comprehensive defect probability, and confidence information.

[0084] In addition, the process of fusing evidence from different modalities is:

[0085] The 0-1 defect probability output by each modal sub-detector is converted into the basic probability assignment (BPA) required by DS theory, and an evidence support framework is constructed. The BPA structure is designed to define the identification framework of the solder joint status, including: defective, non-defective, and uncertain. The defect probability output by the sub-detector is divided into three parts, namely defect support: ( is the modal credibility coefficient, is the defect probability), defect-free support: ( is the modal specific coefficient) and uncertainty: (The remaining probability is assigned to uncertainty). Starting from the BPA of any two modalities, calculate the new combined BPA. If a conflict occurs, calculate the product of the BPAs of the two modalities to obtain the conflict probability, and redistribute the conflict probability according to the original support ratio of the two modalities to generate a new BPA. Combine the result of the previous step with the BPA of the third modal again, repeat the conflict processing and redistribution until all modalities are integrated to generate the final comprehensive BPA, as shown in 、 , the conflict probability K = 0.7 × 0.6 = 0.42, redistribute K to D and N according to the original support ratio of the two modes (0.7:0.6), and generate a new BPA: , Indicates that after integrating the new evidence, the defect category The new basic probability distribution of , For the non-defective category The new basic probability allocation is based on the integrated BPA after fusion. The defect support value in the integrated BPA is directly taken and the confidence is calculated. The confidence is 1-uncertainty, that is, the degree of confidence in the judgment of the solder joint status. Uncertainty refers to the uncertainty of the solder joint status due to insufficient or conflicting evidence. The uncertainty is calculated by DS theory and indicates the probability of not being clearly assigned to any specific category during the fusion process. The higher the 1-uncertainty, the more confident the judgment of the solder joint status. At the same time, dual thresholds are set, namely the defect threshold and the non-defect threshold. When the defect support is greater than 0.8 and the confidence is greater than 0.9, it is judged as defective. When the non-defect threshold is defect support <0.3 or confidence <0.7, it is judged as non-defective, and other cases are marked as "pending re-inspection" to trigger manual review or supplementary inspection;

[0086] Step 5 specifically includes:

[0087] The multimodal inspection system is modeled as a reinforcement learning environment. The state space is designed, the action space is defined, and the reward function is constructed. The state vector of the state space includes: the current performance of each modality, the modal conflict level, and the solder joint feature distribution. The current performance of each modality is the defect classification accuracy of the optical / X-Ray / thermal sub-detectors in the past 10 inspections. The modal conflict level is the average BPA conflict probability between modalities in the last five fusions. The solder joint feature distribution is the static attributes of the size, material type, etc. of the solder joints in the current batch. The action is defined as the adjustment amount of the trust weight of each modality. The core reward is the comprehensive defect classification accuracy after fusion. If the accuracy of the current round is improved by 2% compared with the previous round, the reward is +0.5. If the accuracy drops by 2%, the penalty is -0.3. If the conflict rate decreases by 5%, an additional reward of +0.2 is given. The production line inspection data of the past 6 months is collected and divided into training set and validation set. DQN (Deep The reinforcement learning model is constructed using a Q-Network (Q-Network) structure, combined with an experience replay mechanism to store the historical state-action-reward-new state quadruple. Dual DQN technology is also introduced to separate the target network and evaluation network, stabilize Q-value estimation, set training termination conditions, and deploy the trained reinforcement learning model to the production line for real-time weight optimization and continuous system improvement. After inspecting every 100 solder joints, the reinforcement learning model outputs an action based on the current state and applies this action to update the weights of each modality. Weight adjustment limits are pre-designed, and a single adjustment does not exceed 20% of the initial weight. If the accuracy drops for three consecutive rounds after adjustment, it automatically rolls back to the previous round of weight configuration. Furthermore, the historical database is updated monthly to incorporate the latest inspection data and manual re-inspection results.

[0088] In addition, the construction process of the reinforcement learning model is:

[0089] Six months of inspection records are extracted from the production line database, including: original inspection results, manual re-inspection labels and initial weight configuration. The original inspection results are the defect probability of each solder joint for five modalities, including optical / X-Ray / thermal. The manual re-inspection labels are the true values of defects confirmed by metallographic microscope or X-ray fluoroscopy. The initial weight configuration is the trust weight of each modality at each fusion, and a state vector is constructed to analyze the modal performance and conflict level. The modal performance is calculated by calculating the average accuracy of the past 10 inspections of each modality. The conflict level is the average BPA conflict probability between modalities in the last five fusions. The data is then divided into a training set (the first 80%) and a validation set (the last 20%) in chronological order to generate a 15-dimensional state vector and a complete reward calculation basis, constructed using a DQN structure. A reinforcement learning model for learning optimal weight adjustment strategies was designed, with a network architecture consisting of an input layer, hidden layers, and an output layer. The input layer was a state vector, and the hidden layer was a two-layer fully connected layer (128+64 neurons, ReLU activation). BatchNorm was introduced to accelerate convergence, and the output layer contained the Q-values of five actions (corresponding to the weight adjustments for the five modalities). An experience replay mechanism was incorporated to store 100,000 historical quadruple data, using random sampling to break data correlation. A dual DQN mechanism was employed, with the evaluation network calculating the current Q-value and the target network calculating the target Q-value. The target network parameters were synchronized every 100 rounds to stabilize the training process. Training was terminated when the validation set reward fluctuated < 0.05 for 10 consecutive rounds. This resulted in a fully trained reinforcement learning model.

[0090] Step 6 specifically includes:

[0091] The results of multimodal data fusion are comprehensively analyzed to clarify the defect type and specific location of the solder joint. Based on the fused data, a detailed visual report is generated to display the inspection results in an intuitive way. The visual report includes an overlay view of multimodal data such as optical images, X-ray images, and thermal images of the solder joints, and the location and type of the defects are marked. Through the visual report, the inspection personnel can quickly and accurately understand the quality status of the solder joints, review the inspection results, and identify cases of false detection (normal solder joints are mistakenly judged as defective) and missed detection (defective solder joints are mistakenly judged as normal). For example, the false detection / missed detection cases are recorded, including the multimodal original data of the solder joints, the detection results of the model and the actual status after manual review, and the false detection / missed detection cases are deeply analyzed to find out the causes of the misjudgment. The data of the false detection / missed detection cases are fed back into the training set of the model as new training samples for retraining and optimizing the model. By introducing false detection / missed detection cases, the model learns the feature patterns that it fails to correctly identify. Through continuous iterative optimization, the detection performance of the model is improved, forming a closed-loop system from detection to feedback to optimization, ensuring the long-term stability and accuracy of the detection system.

[0092] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A PCBA circuit board solder joint detection method based on multimodal data fusion, characterized in that: The following steps are involved: Step 1: Build a distributed sensor network and collect multimodal data to obtain all-around solder joint information and perform spatiotemporal alignment of the multimodal data. Step 2: Extract features from multimodal data and dynamically weight the features of each modality through the attention mechanism to highlight key defect representations; Step 3: Use a graph neural network to construct a spatial topology map of solder joints, model the spatial relationship between solder joints, input the graph neural network to learn the contextual features of associated defects, and identify the associated defects of solder joints; Step 4: Design an independent sub-detector for each mode, perform detection based on the characteristics of different modal data, output the defect probability, and integrate the results of each sub-detector through DS evidence theory to comprehensively judge the solder joint status; Step 5: Build a reinforcement learning model based on historical detection data, use defect classification accuracy as the reward function, and dynamically adjust the trust weight of each modality sub-detector; Step 6: Output the defect type and location based on the comprehensive fusion results and generate a visual report.

2. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 1, characterized in that: The step 1 specifically includes: Build a distributed sensing network that covers all dimensions of solder joints, perform multi-physics field synchronous sensing, and assign a unique identifier to each sensor. The distributed sensing network includes optical modules, X-ray modules, thermal modules, mechanical modules, and electrical modules. Using a star-bus hybrid architecture, the main controller connects to the optical, X-ray, and thermal master nodes via Ethernet. Each master node then extends the mechanical and electrical sub-nodes via the CAN bus. High-precision calibration plates are used to calibrate the optical and X-ray spatial coordinate systems. A pixel-to-physical coordinate conversion matrix is established through feature point matching. Synchronous pulse signals are used to trigger all sensors to collect full-modal data to verify the spatiotemporal alignment benchmark. Multimodal data time synchronization is performed based on the PTP protocol. The main controller is designated as the PTP master clock and sends synchronization messages to each sensor sub-node. Each sensor sub-node becomes a slave clock. The slave clock calculates the link delay based on the timestamp in the message and adjusts the local clock offset. A high-speed oscilloscope is then used to synchronously capture the trigger signal of the optical camera and the frame synchronization signal of the X-ray detector to verify that the time deviation is less than 1μs. Data from different perspectives and modalities are mapped to a unified PCB coordinate system to construct a 3D holographic model of the solder joint. The ICP algorithm is used to match feature points on the calibration board, calculate the rigid body transformation matrix from the optical coordinate system to the X-Ray coordinate system, and superimpose the solder joint contours from optical inspection with the internal voids of the solder balls from X-Ray inspection to generate a "shell-core" composite model. The temperature gradient from thermal imaging, the stress distribution from mechanical sensors, and the impedance value from electrical testing are then annotated in the 3D model to form a thermal map of the solder joint health status.

3. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 2, characterized in that: The step 2 specifically includes: Feature extraction is performed on multimodal raw data including optical, X-ray, thermal, mechanical, and electrical data. The modal features are aligned to a unified time axis using PTP synchronized timestamps. Bilinear interpolation is used to account for spatial scale differences, and all features are mapped to the PCB physical coordinate system. Five sets of modal feature vectors are generated, with dimensions of 512, 128, 64, 32, and 48, respectively. The five sets of features are spliced into a joint feature with a dimension of 784. Then, the correlation between modalities is analyzed through a multi-head self-attention mechanism. The outputs of the eight attention heads are spliced and linearly transformed to generate a modal weight vector, which is then output as a weighted feature to highlight defect-sensitive modalities. The Grad-CAM algorithm is applied to the weighted features, and the defect classification loss is back-propagated to the feature layer to generate a heat map. The heat map is superimposed on the original optical image to intuitively display the defect location. The weighted features are input into the fully connected layer and the Softmax classifier to output the defect type and confidence.

4. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 1, characterized in that: The step 3 specifically includes: Based on the physical layout of the PCB and the actual locations of the solder joints, a spatial topology graph of the solder joints is constructed. Each solder joint is considered a node in the spatial topology graph, and the physical connections or spatial proximity between solder joints are considered edges. By extracting the coordinate information and connection relationships of the solder joints, the adjacency matrix and node feature matrix of the graph are generated. The constructed solder joint spatial topology map is input into the graph neural network, which aggregates the neighborhood information of each solder joint node and updates the node feature representation. A two-layer GCN is used, each layer including graph convolution, batch normalization, and ReLU activation. The first layer of GCN aggregates first-order neighborhood information, and the second layer aggregates second-order neighborhood information, outputting 128-dimensional contextual features. A graph attention layer is introduced after the GCN to dynamically calculate the weights of neighboring nodes, thereby expanding the solder joint features from local unimodal information to global contextual information. Based on contextual features, the defect type and correlation of solder joints are predicted. The fully connected layer + Softmax is applied to output the defect type and confidence. Contrastive learning loss is introduced to reduce the feature distance of solder joints with similar defects and increase the feature distance of solder joints with different defect types. Then, a graph-level heat map is generated through Grad-CAM to highlight key neighborhood solder joints.

5. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 4, characterized in that: The process of predicting solder joint defect types and correlations is as follows: The contextual features of each solder joint are processed to predict the defect type and relevance of the solder joint. The contextual features are input into a fully connected layer, which performs a nonlinear transformation on the features. The features are then mapped to different defect categories using a Softmax classifier, and the confidence score for each category is output. The Softmax classifier is then used to generate a probability distribution for each solder joint, indicating the probability of the solder joint belonging to different defect types. By using contrastive learning loss, the feature distances between similar defective solder joints are shortened, while the feature distances between different types of solder joints are extended. This results in similar defective solder joints being clustered in the feature space, while different types of solder joints are dispersed. The Grad-CAM algorithm is used to generate a graph-level heat map. The classification loss is back-propagated to the feature layer, and the contribution of each feature channel to defect classification is calculated. The generated graph-level heat map is then used to highlight the neighboring solder joints that have the greatest impact on defect classification, visually demonstrating the correlation between solder joints.

6. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 2, characterized in that: The step 4 specifically includes: Based on the characteristics of different modal data, independent sub-detectors are designed. Each sub-detector independently outputs the defect probability of the solder joint according to the characteristics of its modal data and defect mode. The defect probability output by each modal sub-detector is regarded as evidence and fused using DS evidence theory. The defect probability output by each sub-detector is converted into BPA as evidence support for the solder joint status. The evidence of different modalities is fused through the combination rules of DS evidence theory to calculate the comprehensive BPA and analyze the solder joint defect situation at the same time. Based on the fused BPA and solder joint defect analysis results, the comprehensive status judgment results of the solder joint are output, including defect type, comprehensive defect probability and confidence information.

7. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 6, characterized in that: The process of fusing evidence from different modalities is as follows: The 0-1 defect probability output by each modal sub-detector is converted into the distribution results required by DS theory, and an evidence support framework is constructed. In this framework, a BPA structure is designed to define the identification framework of solder joint states, including defective, non-defective, and uncertain. The defect probability output by the sub-detector is split into three parts: defect support, non-defect support, and uncertainty. Starting from the BPA of any two modalities, calculate the new combined BPA. If a conflict occurs, calculate the product of the BPAs of the two modalities to obtain the conflict probability, and redistribute the conflict probability according to the original support ratio of the two modalities to generate a new BPA. Combine the result of the previous step with the BPA of the third modality, repeat the conflict processing and redistribution until all modalities are integrated to generate the final comprehensive BPA. Based on the integrated BPA after fusion, the defect support value in the integrated BPA is directly taken and the confidence level is calculated, that is, 1-uncertainty. At the same time, dual thresholds are set, namely the defect threshold and the non-defect threshold. When the defect support level is greater than 0.8 and the confidence level is greater than 0.9, the system is judged as defective. When the non-defect threshold level is less than 0.3 or the confidence level is less than 0.7, the system is judged as non-defective. Other cases are marked as "pending re-inspection" to trigger manual review or supplementary inspection.

8. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 7, characterized in that: The step 5 specifically includes: The multimodal inspection system is modeled as a reinforcement learning environment. The state space is designed, the action space is defined, and the reward function is constructed. The state vector of the state space contains: the current performance of each modality, the modal conflict level, and the solder joint feature distribution. The current performance of each modality is the defect classification accuracy of the optical / X-Ray / thermal sub-detector in the past 10 inspections. The modal conflict level is the average BPA conflict probability between modalities in the last five fusions. The solder joint feature distribution is the static properties of the size and material type of the solder joints in the current batch. The action is defined as the adjustment amount to the trust weight of each modality. The core reward is the comprehensive defect classification accuracy after fusion. If the accuracy of the current round improves by 2% compared with the previous round, the reward is +0.

5. If the accuracy decreases by 2%, the penalty is -0.

3. If the conflict rate decreases by 5%, an additional reward of +0.2 is received. We collected production line inspection data from the past six months and divided it into training and validation sets. We then used the DQN architecture to build a reinforcement learning model, incorporating an experience replay mechanism to store the historical state-action-reward-new state quadruple. We also introduced dual DQN technology to separate the target network and evaluation network, and set training termination conditions. The trained reinforcement learning model is deployed to the production line for real-time weight optimization and continuous system improvement. After inspecting every 100 solder joints, the reinforcement learning model outputs an action based on the current state and applies the action to update the weights of each modality. Weight adjustment limits are pre-designed, and the single adjustment range does not exceed 20% of the initial weight. If the accuracy rate drops for three consecutive rounds after adjustment, it will automatically roll back to the previous round of weight configuration. At the same time, the historical database is updated monthly to include the latest inspection data and manual re-inspection results.

9. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 8, characterized in that: The construction process of the reinforcement learning model is as follows: Six months of inspection records were extracted from the production line database, including the original inspection results, manual re-inspection labels, and initial weight configurations. A state vector was constructed to analyze modal performance and conflict levels. Modal performance was calculated by calculating the average accuracy of each modality over the past 10 inspections, and conflict level was calculated by calculating the average BPA conflict probability between modalities in the most recent five fusions. The data was then divided into training and validation sets in chronological order, generating a 15-dimensional state vector and a complete reward calculation basis. A reinforcement learning model for learning the optimal weight adjustment strategy is constructed using the DQN structure. The network architecture is designed, including input layer, hidden layer, and output layer. The input layer is the state vector, the hidden layer is a 2-layer fully connected layer, and BatchNorm is introduced to accelerate convergence. The output layer is the Q value of 5 actions. Combined with the experience replay mechanism, 100,000 historical quadruple data are stored, and random sampling breaks the data correlation. The dual DQN mechanism is used, and the evaluation network is used to calculate the current Q value, and the target network is used to calculate the target Q value. The target network parameters are synchronized every 100 rounds to stabilize the training process. The fluctuation of the verification set reward for 10 consecutive rounds is set to <0.05 as the training termination condition, thereby obtaining a reinforcement learning model that has been trained.

10. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 9, characterized in that: The step 6 specifically includes: Comprehensively analyze the results of multimodal data fusion to identify the defect type and specific location of the solder joint. Generate a detailed visual report based on the fused data, presenting the inspection results in an intuitive manner while marking the location and type of the defect. Review the inspection results to identify false positives and missed detections. Record these cases, including the multimodal raw data of the solder joints, the model's inspection results, and the actual status after manual review. Conduct in-depth analysis of these false positives and missed detections to identify the causes. The data of false positive / missed positive cases are fed back into the training set of the model as new training samples for retraining and optimizing the model. By introducing false positive / missed positive cases, the model learns the feature patterns that it fails to correctly identify. Through continuous iterative optimization, a closed-loop system is formed from detection to feedback to optimization.

Images