A PCBA circuit board solder joint detection method based on multi-modal data fusion

Abstract

The application discloses a PCBA circuit board welding point detection method based on multi-modal data fusion, relates to the technical field of electronic manufacturing quality detection, constructs a distributed sensing network and collects multi-modal data, comprehensively obtains welding point information, and performs time and space alignment on the multi-modal data; feature extraction is performed on the multi-modal data, each modal feature is dynamically weighted through an attention mechanism, and key defect representation is highlighted; a welding point space topology graph is constructed by using a graph neural network, and the spatial relationship between welding points is modeled; by integrating multi-dimensional data such as optics, X-Ray, heat, mechanics and electricity, the information limitation of single-modal detection is broken through, the complementarity of different modal data is utilized, each modal feature is dynamically weighted in combination with the attention mechanism, complex defects are accurately identified, the welding point space topology relationship is modeled by using the graph neural network, associated defects are further captured, and the defect classification accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic manufacturing quality detection, and in particular to a PCBA circuit board solder joint detection method based on multi-modal data fusion. BACKGROUND

[0002] PCBA (Printed Circuit Board Assembly) circuit board solder joint detection is a key link in the electronic manufacturing process. The solder joint is the core of the electrical connection between electronic components and PCB boards, and its quality directly affects the long-term reliability of the product. Defects such as virtual welding, cold welding, and bridging can cause poor contact, signal interference, and even equipment failure. In high-reliability fields such as automotive electronics, aerospace, and medical devices, solder joint defects can cause serious safety hazards. Moreover, electronic products need to adapt to extreme environments such as high temperature, high humidity, and vibration, and the solder joint quality needs to meet long-term stability requirements to avoid failure due to environmental stress.

[0003] In the prior art, PCBA circuit board solder joint detection is usually performed using a single modal detection method. Due to the use of data from a single sensor, there is a lack of correlation analysis of multi-modal data, resulting in a high rate of defect misjudgment and poor adaptability to complex defects. Therefore, it is necessary to manually adjust the parameters. Therefore, how to use a distributed sensor network to obtain multi-modal data and extract complementary features of each modal data, and introduce a graph neural network to model the spatial relationship between solder joints and identify associated defects is a problem to be solved by the present application. To this end, a PCBA circuit board solder joint detection method based on multi-modal data fusion is proposed. SUMMARY

[0004] The present application aims to provide a PCBA circuit board solder joint detection method based on multi-modal data fusion to solve the problems raised in the background art.

[0005] To solve the above technical problems, the technical solution adopted by the present application is as follows:

[0006] A PCBA circuit board solder joint detection method based on multi-modal data fusion, comprising the following steps:

[0007] Step 1: Construct a distributed sensor network and collect multi-modal data to obtain solder joint information in all directions, and perform spatio-temporal alignment of multi-modal data through a Precision Time Protocol (PTP);

[0008] Step 2: Extract features from the multi-modal data, dynamically weight the features of each modal through an attention mechanism, and highlight key defect representations;

[0009] Step 3: Use a graph neural network to construct a solder joint spatial topology graph, model the spatial relationship between solder joints, input the graph neural network to learn the context features of associated defects, identify associated defects of the solder joints, and improve the detection accuracy;

[0010] Step 4, design independent sub-detectors for each mode, detect different modal data characteristics, output defect probability, and fuse the results of each sub-detector by D-S evidence theory to comprehensively judge the solder joint state;

[0011] Step 5, based on historical detection data, a reinforcement learning model is constructed, the defect classification accuracy is taken as the reward function, and the trust weight of each modal sub-detector is dynamically adjusted to realize adaptive optimization;

[0012] Step 6, the comprehensive fusion result outputs the defect type and position, and generates a visual report.

[0013] The further improvement of the technical scheme of the application is that the step 1 specifically comprises:

[0014] A distributed sensing network covering full-dimensional information of the solder joint is constructed, and multi-physical field synchronous sensing is performed, and a unique identifier is assigned to each sensor, wherein the distributed sensing network comprises an optical module, an X-Ray module, a thermal module, a mechanical module and an electrical module;

[0015] A star-bus hybrid architecture is adopted, the main controller is connected with the optical, X-Ray and thermal master nodes through Ethernet, each master node is further connected with the mechanical and electrical slave nodes through CAN bus, low-delay data transmission is ensured, a high-precision calibration board is used to calibrate the space coordinate systems of the optical and X-Ray, a pixel-physical coordinate conversion matrix is established through feature point matching, all sensors are triggered to collect full-modal data once by using a synchronous pulse signal, and a time and space alignment benchmark is verified;

[0016] Time synchronization of multi-modal data is performed based on PTP protocol, the main controller (PLC) is specified as a PTP master clock, the IEEE 1588-2008 protocol is used to send a synchronization message to each sensor slave node, each sensor slave node is a slave clock, the slave clock calculates the link delay according to the timestamp in the message and adjusts the local clock offset, and then a high-speed oscilloscope is used to synchronously capture the trigger signal of the optical camera and the frame synchronization signal of the X-Ray detector, and it is verified whether the time deviation is less than 1 μs;

[0017] Data of different angles and modes are mapped to a unified PCB coordinate system, and a 3D holographic model of the solder joint is constructed, wherein the feature points on the calibration board are matched through the ICP (Iterative Closest Point) algorithm, the rigid transformation matrix from the optical coordinate system to the X-Ray coordinate system is calculated, the solder joint profile detected by the optical detection is superimposed with the internal cavity of the solder ball detected by the X-Ray, and a "shell-core" composite model is generated, and then the temperature gradient of the thermal imaging, the stress distribution of the mechanical sensor and the impedance value of the electrical test are marked in the 3D model, and a thermal stress map of the solder joint health state is formed.

[0018] The further improvement of the technical scheme of the present application is that the step 2 specifically comprises:

[0019] Feature extraction is performed on multi-modal raw data including optics, X-Ray, thermology, mechanics and electricity, and each modal feature is aligned to a unified time axis through PTP synchronized timestamps, spatial scale differences are adopted by bilinear interpolation, all features are mapped to a PCB physical coordinate system (accuracy 0.01mm), and then five groups of modal feature vectors are generated , with dimensions of 512, 128, 64, 32 and 48 respectively;

[0020] The five groups of features are spliced into joint features , with a dimension of 784, and then the inter-modal correlation is analyzed through a multi-head self-attention mechanism (8 heads, each head with 64 dimensions), the 8 head attention outputs are spliced and linearly transformed to generate a modal weight vector , and then the weighted features are output to highlight the defect sensitive modal;

[0021] The Grad-CAM algorithm is applied to the weighted features, the defect classification loss is back propagated to the feature layer to generate a heat map, the heat map is superimposed on the original optical image to intuitively display the defect position, and the weighted features are input into a fully connected layer (256 dimensions) and a Softmax classifier to output the defect type and confidence.

[0022] The further improvement of the technical scheme of the present application is that the step 3 specifically comprises:

[0023] According to the physical layout of the PCB and the actual position of the solder joints, a spatial topology graph of the solder joints is constructed, each solder joint is regarded as a node in the spatial topology graph of the solder joints, the physical connection or spatial proximity relationship between the solder joints is regarded as an edge, and the coordinate information and connection relationship of the solder joints are extracted to generate an adjacency matrix and a node feature matrix of the graph;

[0024] The constructed spatial topology graph of the solder joints is input into a graph neural network, the neighborhood information of each solder joint node is aggregated, and the feature representation of the node is updated, wherein two layers of GCN are adopted, each layer includes graph convolution, batch normalization and ReLU activation, the first layer of GCN aggregates first-order neighborhood information, the second layer of GCN aggregates second-order neighborhood information, the output dimension of the context feature is 128, and a graph attention layer (GAT) is introduced after the GCN to dynamically calculate the neighborhood node weight, and then the solder joint feature is expanded from local single modal information to global context information;

[0025] Based on the context feature prediction of the welding point defect type and the correlation, a full connection layer (256 dimensions) is applied + Softmax output defect type and confidence, and a contrast learning loss (Contrastive Loss) is introduced to reduce the feature distance of the same type of defect welding points, so that the welding points with the same defect type are close in the feature space, and the feature distance of the different type of defect welding points is increased, so that the welding points belonging to different defect types are far away in the feature space, and then a Grad-CAM graph-level heat map is generated to highlight the key neighborhood welding points.

[0026] Further improvement of the technical scheme of the application is that the process of predicting the welding point defect type and the correlation is:

[0027] The context feature (128-dimensional feature extracted by the graph neural network) of each welding point is processed to predict the defect type and the correlation of the welding point, wherein the context feature is input into a full connection layer (256 dimensions) to perform nonlinear transformation on the feature to extract higher-level feature representation, and then the feature is mapped to different defect categories by a Softmax classifier to output the confidence of each category, and a probability distribution is generated for each welding point by the Softmax classifier to represent the probability of the welding point belonging to different defect types.

[0028] The contrast learning loss is used to reduce the feature distance of the same type of defect welding points and increase the feature distance of different type of welding points, so that the same type of defect welding points are gathered in the feature space, and the different type of welding points are scattered, which helps to improve the accuracy of defect classification and enhance the recognition ability of the model to the correlation of the welding points.

[0029] The Grad-CAM algorithm is used to generate a graph-level heat map, the classification loss is back-propagated to the feature layer, the contribution of each feature channel to the defect classification is calculated, and the generated graph-level heat map is used to highlight the neighborhood welding points that have the greatest impact on the defect classification, so that the correlation between the welding points is intuitively displayed.

[0030] Further improvement of the technical scheme of the application is that the step 4 specifically comprises:

[0031] Independent sub-detectors are designed according to the characteristics of different modal data, and each sub-detector independently outputs the defect probability of the welding point according to the characteristics and defect mode of the modal data thereof.

[0032] The defect probabilities output by the modal sub-detectors are regarded as evidence, the D-S (Dempster-Shafer) evidence theory is used for fusion, the defect probability output by each sub-detector is converted into a BPA (Basic Probability Assignment) by defining the BPA, the BPA is used as the evidence support for the welding point state, and the evidences of different modes are fused by the combination rule of the D-S evidence theory to calculate the comprehensive BPA and analyze the defect condition of the welding point.

[0033] According to the analysis result of the fusion BPA and the welding spot defect condition, a comprehensive state judgment result of the welding spot is output, including defect type, comprehensive defect probability and confidence information.

[0034] The further improvement of the technical scheme of the present application is that the process of fusing evidences of different modes is:

[0035] The 0-1 defect probability output by each modal sub-detector is converted into basic probability assignment (BPA) required by D-S theory to construct an evidence support framework, wherein a BPA structure is designed, and an identification framework of the welding spot state is defined, including defect, no defect and uncertainty, the defect probability output by the sub-detector is split into three parts, i.e. defect support , no defect support and uncertainty ;

[0036] From the BPA of any two modes, a new combined BPA is calculated, if a conflict occurs, the product of the BPA of the two modes is calculated to obtain a conflict probability, and the conflict probability is redistributed according to the original support ratio of the two modes to generate a new BPA, the result of the last step is combined with the BPA of the third mode again, and the conflict processing and redistribution are repeated until all modes are fused to generate a final comprehensive BPA.

[0037] Based on the fused comprehensive BPA, the value of the defect support in the comprehensive BPA is directly taken, and the confidence is calculated, i.e. 1-uncertainty, at the same time, double thresholds are set, i.e. defect threshold and no defect threshold, the defect threshold is defect support>0.8 and confidence>0.9, which is judged as having defect, the no defect threshold is defect support<0.3 or confidence<0.7, which is judged as no defect, and other cases are marked as “to be rechecked”, triggering manual review or supplementary detection.

[0038] The further improvement of the technical scheme of the present application is that the step 5 specifically includes:

[0039] The multi-modal detection system is modeled as a reinforcement learning environment, a state space is designed, an action space is defined, and a reward function is constructed, the state vector of the state space includes: current performance of each mode, mode conflict level and spot feature distribution, the current performance of each mode is the defect classification accuracy of the optical / X-ray / thermal sub-detector in the past 10 detections, the mode conflict level is the average of the BPA conflict probability between modes in the last 5 fusions, and the spot feature distribution is the static attribute of the size, material type and the like of the current batch of spots, the action is defined as the adjustment amount of the trust weight of each mode, and the comprehensive defect classification accuracy after fusion is taken as the core reward, if the accuracy in the current round is improved by 2% than the last round, the reward is +0.5, if the accuracy decreases by 2%, the punishment is-0.3, and if the conflict rate decreases by 5%, the additional reward is +0.2;

[0040] Collect the production line detection data in the past 6 months, and divide the data into a training set and a validation set, use the DQN (DeepQ-Network) structure to construct the reinforcement learning model, combine the experience replay mechanism, store the historical state-action-reward-new state four-tuple, at the same time, introduce the double DQN technology, separate the target network and the evaluation network, stabilize the Q value estimation, and set the training termination condition;

[0041] Deploy the trained reinforcement learning model to the production line for real-time weight optimization and system continuous improvement, after detecting 100 spots, the reinforcement learning model outputs the action according to the current state, and updates the weight of each mode by applying the action, the weight adjustment limit is designed in advance, the single adjustment amplitude does not exceed 20% of the initial weight, if the accuracy decreases continuously for 3 rounds after adjustment, it is automatically rolled back to the weight configuration of the last round, and at the same time, the historical database is updated every month, and the latest detection data and manual review results are added.

[0042] The further improvement of the technical scheme of the application is that the construction process of the reinforcement learning model is:

[0043] 6 months of detection records are extracted from the production line database, including: original detection results, manual review labels and initial weight configuration, and a state vector is constructed to analyze the performance of the mode and the conflict level, the performance of the mode is the average accuracy of the past 10 detections of each mode, and the conflict level is the average of the BPA conflict probability between modes in the last 5 fusions, and then the data is divided into a training set (the first 80%) and a validation set (the last 20%) in chronological order, a 15-dimensional state vector and a complete reward calculation basis are generated;

[0044] The DQN structure is adopted to construct a reinforcement learning model for learning an optimal weight adjustment strategy, a network architecture is designed, including an input layer, a hidden layer and an output layer, the input layer is a state vector, the hidden layer is 2 layers of full connection (128+64 neurons, ReLU activation), BatchNorm is introduced to accelerate convergence, the output layer is the Q value of 5 actions (corresponding to the weight adjustment amount of 5 modes), an experience replay mechanism is combined, 100,000 historical quadruples are stored, data correlation is broken by random sampling, and a double DQN mechanism is used, the current Q value is calculated by using an evaluation network, the target Q value is calculated by using a target network, the target network parameters are synchronized every 100 rounds to stabilize the training process, the training termination condition is set as the reward of the verification set fluctuating less than 0.05 for 10 consecutive rounds, and then the trained reinforcement learning model is obtained.

[0045] Further improvement of the technical scheme of the present application is that step 6 specifically includes:

[0046] The results of the multi-modal data fusion are comprehensively analyzed to determine the defect type and specific location of the welding point, a detailed visual report is generated based on the fused data to visually display the detection results, and the location and type of the defects are labeled;

[0047] The detection results are reviewed to identify misjudgment (judging normal welding points as defective) and missed cases (judging defective welding points as normal), the misjudgment / missed cases are recorded, including the multi-modal original data of the welding points, the detection results of the model and the actual state after manual review, and the misjudgment / missed cases are analyzed in depth to find out the causes of misjudgment;

[0048] The data of the misjudgment / missed cases are fed back to the training set of the model as new training samples for retraining and optimization of the model, the model learns the feature patterns that cannot be correctly identified by introducing the misjudgment / missed cases, the detection performance of the model is improved through continuous iteration and optimization, a closed-loop system from detection to feedback to optimization is formed to ensure the long-term stability and accuracy of the detection system.

[0049] Due to the adoption of the above technical scheme, the present application has the following technical progress compared with the prior art:

[0050] The present application provides a PCBA circuit board welding point detection method based on multi-modal data fusion, which integrates optical, X-Ray, thermal, mechanical, electrical and other multi-dimensional data, breaks through the information limitation of single-mode detection, dynamically weights each modal feature by using the complementarity of different modal data combined with the attention mechanism, accurately identifies complex defects, and further captures associated defects by modeling the spatial topological relationship of the welding points using a graph neural network, thereby improving the defect classification accuracy.

[0051] The present invention provides a PCBA circuit board solder joint detection method based on multimodal data fusion. By introducing a reinforcement learning model and using defect classification accuracy as a reward function, the trust weight of each modal sub-detector is dynamically adjusted. By analyzing the modal performance, conflict level, and solder joint feature distribution in historical detection data, the detection strategy is optimized in real time. This enables the detection system to automatically adjust its own detection strategy according to data changes and defect pattern changes in actual production, thereby always maintaining high detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0053] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0054] Figure 2 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] Example 1, as Figure 1 、 Figure 2 As shown, the present invention provides a PCBA circuit board solder joint detection method based on multimodal data fusion, comprising the following steps:

[0057] Step 1, construct a distributed sensing network and collect multi-modal data, obtain comprehensive information of the solder joint, and perform spatio-temporal alignment of multi-modal data through a time synchronization protocol (PTP), ensure synchronous capture of full-dimensional information of the solder joint, construct a distributed sensing network covering full-dimensional information of the solder joint, and perform synchronous sensing of multiple physical fields, assign a unique identifier to each sensor, wherein the distributed sensing network includes an optical module, an X-Ray module, a thermal module, a mechanical module, and an electrical module, for the optical module, deploy a high-resolution industrial camera (visible light + infrared dual-channel), install it above the SMT production line track, cover the PCB surface topography and thermal mark detection, for the X-Ray module, use a micro-focus X-Ray source and a CMOS flat panel detector, move to the key area of the PCB through a mechanical arm to realize local penetration imaging, for the thermal module, integrate an infrared thermal imager and a thermocouple array to monitor the dynamic temperature field and static thermal stress distribution during the welding process, for the mechanical module, embed a laser displacement sensor and a strain gauge in the PCB clamp to measure the height, coplanarity, and deformation caused by mechanical vibration of the solder joint in real time, for the electrical module, configure a multi-channel impedance analyzer, contact the PCB test points through probes to collect contact resistance, capacitance, and other electrical parameters, use a star-bus hybrid architecture, the main controller connects the optical, X-Ray, and thermal master nodes through Ethernet, and each master node extends the mechanical and electrical slave nodes through CAN bus to ensure low-delay data transmission, and use a high-precision calibration board to calibrate the spatial coordinate systems of the optical and X-Ray, establish a pixel-physical coordinate conversion matrix through feature point matching, trigger all sensors to collect full-modal data once using a synchronous pulse signal, verify the spatio-temporal alignment reference, synchronize the multi-modal data based on the PTP protocol, specify the main controller (PLC) as the PTP master clock, send synchronization messages to each sensor node through the IEEE 1588-2008 protocol, each sensor node is a slave clock, and the slave clock calculates the link delay according to the timestamp in the message and adjusts the local clock offset, then use a high-speed oscilloscope to synchronously capture the trigger signal of the optical camera and the frame synchronization signal of the X-Ray detector to verify whether the time deviation is less than 1 μs (satisfying the dynamic monitoring requirements of the welding process), eliminate the sampling clock deviation of the sensors, ensure the time consistency of dynamic defects, map the data of different perspectives and modalities to a unified PCB coordinate system, and construct a 3D holographic model of the solder joint, wherein the feature points on the calibration board are matched through the ICP (Iterative Closest Point) algorithm, the rigid transformation matrix from the optical coordinate system to the X-Ray coordinate system is calculated, the solder joint profile detected by the optical detection is superimposed with the internal cavity of the solder ball detected by the X-Ray detection to generate a "shell-core" composite model, and then the temperature gradient of thermal imaging, stress distribution of mechanical sensors, and impedance values of electrical testing are labeled in the 3D model to form a thermal stress map of the solder joint health status;

[0058] The expression of the rigid body transformation matrix from the optical coordinate system to the X-Ray coordinate system is:

[0059] ;

[0060] ;

[0061] Where, is the rigid body transformation matrix, is the homogeneous coordinate in the X-Ray coordinate system, is the homogeneous coordinate in the optical coordinate system, is the rotation matrix, a 2×2 submatrix, describing the rotation relationship between coordinate systems; is the rotation parameter, is the translation vector, a 2×1 vector, describing the offset of the coordinate system origin; is a 1×2 zero vector to ensure that the matrix multiplication dimensions match. The transformation process is: Align the coordinate axis direction of the optical coordinate system to the X-Ray coordinate system by Move the origin of the optical coordinate system to the origin of the X-Ray coordinate system. The 1 at the end allows for uniform representation of rotation and translation.

[0062] Step 2: Extract features from multimodal data. Dynamically weight each modal feature through the attention mechanism to highlight key defect representations. Extract features from multimodal raw data including optical, X-ray, thermal, mechanical, and electrical data. Align each modal feature to a unified time axis through PTP synchronized timestamps. Use bilinear interpolation for spatial scale differences and map all features to the PCB physical coordinate system (with an accuracy of 0.01mm) to generate five sets of modal feature vectors. The dimensions are 512, 128, 64, 32, and 48 respectively, eliminating the spatial / temporal scale differences between modes. Among them, the optical feature uses the ResNet-50 convolutional network to extract the surface morphology features of the solder joint and output a 512-dimensional feature vector. The X-Ray feature uses 3D U-Net to segment the internal voids of the solder ball and extract 128-dimensional features including void volume and shape irregularity. The thermal feature applies the LSTM network to the infrared thermal imager time series data to capture the 64-dimensional dynamic features of the welding process including the slope and peak of the temperature curve. The mechanical feature uses PCA to reduce the dimension of the laser displacement sensor and strain gauge data to extract 32-dimensional static features including solder joint height deviation and coplanarity error. The electrical feature performs Fourier transform on the frequency domain response (1kHz~1MHz) of the impedance analyzer to extract 48-dimensional features including contact resistance and capacitance resonance point. The five groups of features are spliced ​​into joint features. , Represents the optical characteristics, Represents X-Ray features, Represents thermal characteristics, Represents the mechanical characteristics, Represents electrical features with a dimension of 784. Then, the correlation between modalities is analyzed through a multi-head self-attention mechanism (8 heads, 64 dimensions per head). The 8-head attention output is concatenated and linearly transformed to generate a modal weight vector , represents the weight of the optical mode, Represents the weight of the X-Ray modality, represents the weight of the thermal mode, represents the weight of the mechanical mode, The weights representing the electrical modes are then output as weighted features to highlight the defect-sensitive modes. The Grad-CAM algorithm is applied to the weighted features, and the defect classification loss is back-propagated to the feature layer to generate a heat map. The heat map is superimposed on the original optical image to intuitively display the defect location. The weighted features are then input into the fully connected layer (256 dimensions) and the Softmax classifier to output the defect type and confidence level.

[0063] Step 3: Use graph neural network to construct solder joint spatial topology map, model the spatial relationship between solder joints, input graph neural network to learn contextual features of associated defects, identify associated defects of solder joints, improve detection accuracy, and construct solder joint spatial topology map based on the physical layout of PCB and the actual position of solder joints. Each solder joint is regarded as a node in the solder joint spatial topology map, and the physical connection or spatial proximity relationship between solder joints is regarded as an edge. By extracting the coordinate information and connection relationship of solder joints, the adjacency matrix and node feature matrix of the graph are generated. The constructed solder joint spatial topology map is input into the graph neural network, and the feature learning of solder joint nodes is performed through graph convolution operation to capture the local and global spatial relationships between solder joints. The system aggregates the neighborhood information of each solder joint node and updates the feature representation of the node. Two layers of GCN are used, each layer contains graph convolution, batch normalization and ReLU activation. The first layer of GCN aggregates the first-order neighborhood information, and the second layer of GCN aggregates the second-order neighborhood information. The output dimension is 128-dimensional context features. A graph attention layer (GAT) is introduced after GCN to dynamically calculate the weights of neighborhood nodes, thereby expanding the solder joint features from local unimodal information to global context information. Based on the context features, the solder joint defect type and correlation are predicted. A fully connected layer (256 dimensions) + Softmax is applied to output the defect type and confidence. Contrastive learning loss is introduced to shorten the feature distance of similar defective solder joints and push the features of different types of solder joints apart. Then, a graph-level heat map is generated through Grad-CAM to highlight key neighborhood solder joints.

[0064] In addition, the expression of the first-order neighborhood information aggregated by the first-layer GCN is as follows:

[0065] ;

[0066] In the formula, represents the output feature matrix of the first layer graph convolutional network, which is the node feature representation obtained after graph convolution operation and nonlinear activation function (ReLU) processing, is an activation function, is a degree matrix (DegreeMatrix), which is a diagonal matrix in the graph structure, is the square root of each diagonal element of the degree matrix , is the inverse of the square root of each diagonal element of the degree matrix, is the adjacency matrix with self-loop, which represents the original adjacency matrix describes the connection relationship between nodes in the spatial topology graph of the welding spot, and wherein is an identity matrix, and adding a self-loop means that each node also establishes a connection with itself, is an input feature matrix, each row of which corresponds to a feature vector of a node, is a trainable weight matrix of the first layer graph convolutional network, which performs linear transformation on the input features, and its dimension is usually , is the feature dimension of the first layer output;

[0067] The second layer GCN aggregates second-order neighborhood information, and the output dimension is 128-dimensional context features, whose expression is as follows:

[0068] ;

[0069] In the formula, is the output feature matrix of the second layer graph convolutional network, is a real set, indicating that the elements in the feature matrix are real numbers, is the number of nodes in the spatial topology graph of the welding spot, is the dimension of the output feature, indicating that after the second layer GCN processing, the feature of each node is mapped to a 128-dimensional space;

[0070] The expression for dynamically calculating the neighborhood node weight is as follows:

[0071] ;

[0072] ;

[0073] In the formula, represents an attention coefficient, which is used to measure the importance weight of node to node , is an activation function, is a small positive number, used to allow a small gradient when the input is negative, avoiding the "neuron death" problem caused by the gradient of ReLU function being zero in the negative region, is an attention parameter vector, belonging to trainable parameters, used to linearly transform the concatenated node features to calculate the attention score, denotes the transpose of the attention parameter vector , is a weight matrix, belonging to trainable parameters, used to linearly transform the node features to map the node features to a new feature space for better attention calculation, denotes the input value, when , the output of the LeakyReLU function is ; when , the output of the LeakyReLU function is , and are the feature vectors of the nodes and output by the 2nd layer GCN, respectively, and the symbol denotes the vector concatenation operation, i.e. is to concatenate the two vectors and into a new vector by column, is the set of neighbor nodes of the node ;

[0074] The process of predicting the type of welding defects and the correlation is as follows:

[0075] The context features (128-dimensional features extracted by the graph neural network) of each weld point are processed to predict the defect type and relevance of the weld point. The context features are input into a fully connected layer (256-dimensional) to perform non-linear transformation and extract higher-level feature representations. Then, the features are mapped to different defect categories by a Softmax classifier, outputting the confidence of each category. A probability distribution is generated for each weld point using the Softmax classifier, representing the probability of the weld point belonging to different defect types. Through contrastive learning loss, the feature distance of weld points with the same defect type is reduced, making weld points with the same defect type close in feature space. The feature distance of weld points with different defect types is increased, making weld points with different defect types far apart in feature space. Specifically, during training, for weld point pairs with the same defect type, the contrastive learning loss is used to punish the distance between features if it is too far apart. For weld point pairs with different defect types, the contrastive learning loss is used to punish the distance between features if it is too close. This helps to improve the accuracy of defect classification and enhance the model's ability to identify weld point relevance. Grad-CAM algorithm is used to generate a graph-level heat map. By backpropagating the classification loss to the feature layer, the contribution of each feature channel to defect classification is calculated. The generated graph-level heat map is used to highlight the most influential neighbor weld points for defect classification, providing a visual representation of the relevance between weld points.

[0076] The expression of contrastive learning loss is as follows:

[0077] ;

[0078] In the formula, is the contrastive learning loss, represents the sum operation on all sample pairs , is the same label, indicating whether sample and sample belong to the same category. If sample and sample belong to the same category, then , if sample and sample belong to different categories, then , are the attention feature vectors of sample and sample before passing through the fully connected layer (256-dimensional) and Softmax output, represents the Euclidean distance between feature vectors and , which measures the difference between two samples in feature space, represents when (i.e. sample and samples belong to different categories), ,when (i.e. sample and samples belong to the same category), , in conjunction with the maximum function, is used to impose losses on pairs of samples of different classes. is a pre-set threshold used to control the minimum distance between pairs of samples of different classes. and The characteristic distance When less than, As evidence, this item will cause loss, which will increase the feature distance of different types of samples. When the feature distance is greater than or equal to When , this item is 0, and no additional loss occurs;

[0079] Step 4: Design an independent sub-detector for each mode, perform detection based on the characteristics of different modal data, output the defect probability, and integrate the results of each sub-detector through DS evidence theory to comprehensively judge the solder joint status;

[0080] Step 5: Build a reinforcement learning model based on historical detection data, use defect classification accuracy as the reward function, and dynamically adjust the trust weight of each modality sub-detector to achieve adaptive optimization;

[0081] In step 6, the defect type and location are output based on the comprehensive fusion results, a visual report is generated, and the false detection / missed detection cases are fed back to the model training set. The feature extraction and fusion strategies are continuously optimized iteratively to form a closed loop of detection-feedback-optimization.

[0082] Example 2, as Figure 1 、 Figure 2 As shown, based on Example 1, the present invention provides a technical solution: preferably, step 4 specifically includes:

[0083] According to the characteristics of different modal data, independent sub-detectors are designed, and each sub-detector independently outputs the defect probability of the solder joint according to the characteristics and defect modes of its modal data, wherein the optical sub-detector uses a convolutional neural network (CNN) to analyze the surface topography characteristics of the solder joint and outputs the defect probability, the X-Ray sub-detector segments the internal structure of the solder ball through a 3D U-Net to identify internal voids and other defects and outputs the probability, the thermal sub-detector analyzes the temperature dynamic characteristics of the welding process based on an LSTM network and outputs the thermal defect probability, the mechanical sub-detector analyzes strain and displacement data to detect mechanical performance defects of the solder joint and outputs the probability, and the electrical sub-detector identifies electrical parameter abnormalities according to impedance analysis results and outputs the probability. The defect probabilities output by the sub-detectors of different modalities are regarded as evidence, and D-S (Dempster-Shafer) evidence theory is used for fusion. By defining basic probability assignment (BPA), the defect probability output by each sub-detector is converted into BPA as evidence supporting the state of the solder joint, and by using the combination rule of D-S evidence theory, the evidences of different modalities are fused to calculate the comprehensive BPA, and the solder joint defect condition is analyzed. According to the analysis results of the fused BPA and the solder joint defect condition, the comprehensive state judgment result of the solder joint is output, including the defect type, the comprehensive defect probability and the confidence information.

[0084] In addition, the process of fusing the evidences of different modalities is as follows:

[0085] The 0-1 defect probability output by each modal sub-detector is converted into the basic probability assignment (BPA) required by D-S theory to construct the evidence support framework, wherein the BPA structure is designed, and the recognition framework of the state of the solder joint is defined, including defects, no defects and uncertainty. The defect probability output by the sub-detector is divided into three parts, i.e. defect support: is the modal confidence coefficient, is the defect probability, no defect support: is the modal specificity coefficient, and uncertainty: (the remaining probability is assigned to uncertainty). Starting from the BPA of any two modalities, the new BPA after combination is calculated. If a conflict occurs, the product of the BPA of the two modalities is calculated to obtain the conflict probability, and the conflict probability is redistributed according to the original support ratio of the two modalities to generate a new BPA. The result of the last step is combined with the BPA of the third modality again, and the conflict processing and redistribution are repeated until all modalities are fused to generate the final comprehensive BPA, such as , , the conflict probability K=0.7×0.6=0.42, and K is redistributed to D and N according to the original support ratio (0.7:0.6) of the two modalities to generate a new BPA:​​ represents the new basic probability assignment for the defect class after fusing the new evidence, represents the new basic probability assignment for the non-defect class , directly taking the value of defect support in the fused BPA, calculating the confidence, which is 1- uncertainty, i.e. the degree of confidence in the weld state judgment, and the uncertainty refers to the uncertainty in the weld state due to insufficient or conflicting evidence, which is calculated by the D-S theory, represents the probability that is not explicitly assigned to any specific class during the fusion process, the higher the 1- uncertainty, the more confident the judgment of the weld state is, at the same time, a double threshold is set, which is a defect threshold and a non-defect threshold, the defect threshold is defect support > 0.8 and confidence > 0.9, which is determined as defective, the non-defect threshold is defect support < 0.3 or confidence < 0.7, which is determined as non-defective, and other cases are marked as “to be rechecked”, triggering manual review or supplementary detection;

[0086] Step 5 specifically includes:

[0087] ​​The multi-modal detection system is modeled as a reinforcement learning environment, the state space is designed, the action space is defined, and the reward function is constructed. The state vector of the state space includes: the current performance of each modality, the conflict level of the modalities, and the feature distribution of the welding spot. The current performance of each modality is the defect classification accuracy of the optical / X-ray / thermal sub-detector in the past 10 detections. The conflict level of the modalities is the average conflict probability between modalities in the last 5 fusion times. The feature distribution of the welding spot is the static attribute of the size and material type of the current batch of welding spots. The action is defined as the adjustment amount of the trust weight of each modality. The core reward is the comprehensive defect classification accuracy after fusion. If the accuracy of the current round is improved by 2% compared with the last round, the reward is +0.5. If the accuracy decreases by 2%, the punishment is -0.3. If the conflict rate decreases by 5%, the additional reward is +0.2. The production line detection data of the past 6 months is collected and divided into training set and validation set. The DQN (Deep Q-Network) structure is used to construct the reinforcement learning model. The experience replay mechanism is combined to store the historical state-action-reward-new state four-tuple. At the same time, the double DQN technology is introduced to separate the target network and the evaluation network to stabilize the Q value estimation. The training termination condition is set. The trained reinforcement learning model is deployed to the production line for real-time weight optimization and continuous improvement of the system. After detecting 100 welding spots, the reinforcement learning model outputs the action according to the current state and updates the weight of each modality. The weight adjustment limit is designed in advance. The single adjustment amplitude does not exceed 20% of the initial weight. If the accuracy decreases for 3 consecutive rounds after adjustment, it is automatically rolled back to the last weight configuration. At the same time, the historical database is updated every month to add the latest detection data and manual review results.

[0088] In addition, the construction process of the reinforcement learning model is as follows:

[0089] The 6-month detection records are extracted from the production line database, including: original detection results, artificial re-inspection labels and initial weight configurations, the original detection results are the defect probabilities of each solder joint in 5 modalities such as optical, X-ray, thermal, etc., the artificial re-inspection labels are the defect true values confirmed by metallographic microscope or X-ray perspective, and the initial weight configurations are the trust weights of each modality at each fusion, and a state vector is constructed to analyze the modality performance and conflict level, the modality performance is the average accuracy of each modality in the past 10 detections, and the conflict level is the average conflict probability of modalities in the last 5 fusions, and then the data is divided into training set (first 80%) and validation set (last 20%) in time sequence, 15-dimensional state vector and complete reward calculation basis are generated, DQN structure is used to construct a reinforcement learning model that learns the optimal weight adjustment strategy, the network architecture is designed, including input layer, hidden layer and output layer, the input layer is the state vector, the hidden layer is 2 layers of full connection (128+64 neurons, ReLU activation), BatchNorm is introduced to accelerate convergence, and the output layer is the Q value of 5 actions (corresponding to the weight adjustment amount of 5 modalities), combined with the experience replay mechanism, 100,000 historical quadruples are stored, random sampling is used to break the data correlation, and the double DQN mechanism is used, the evaluation network is used to calculate the current Q value, the target network is used to calculate the target Q value, the target network parameters are synchronized every 100 rounds to stabilize the training process, the training termination condition is set as the reward of the validation set fluctuating by less than 0.05 for 10 consecutive rounds, and then the trained reinforcement learning model is obtained.

[0090] Step 6 specifically includes:

[0091] The results of multi-modal data fusion are comprehensively analyzed to determine the defect type and specific location of the solder joint. Based on the fused data, a detailed visual report is generated to visually display the detection results. The visual report includes the superimposed view of the optical image, X-ray imaging, thermal imaging, and other multi-modal data of the solder joint, as well as the location and type of defects. Through the visual report, the detection personnel can quickly and accurately understand the quality status of the solder joint, review the detection results, identify misjudgment (incorrectly judging normal solder joints as defective) and missed detection (incorrectly judging defective solder joints as normal), record the misjudgment / missed detection cases, including the multi-modal original data of the solder joint, the detection results of the model, and the actual state after artificial re-inspection, and conduct in-depth analysis on the misjudgment / missed detection cases to find out the reasons for the misjudgment. The data of misjudgment / missed detection cases is fed back to the training set of the model as new training samples for retraining and optimization of the model. By introducing misjudgment / missed detection cases, the model learns the feature patterns that have not been correctly identified. Through continuous iteration and optimization, the detection performance of the model is improved, forming a closed-loop system from detection to feedback to optimization, ensuring the long-term stability and accuracy of the detection system.

[0092] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A PCBA circuit board solder joint detection method based on multimodal data fusion, characterized in that: The following steps are involved: Step 1: Build a distributed sensor network and collect multimodal data to obtain all-around solder joint information and perform spatiotemporal alignment of the multimodal data. Step 2: Extract features from multimodal data and dynamically weight the features of each modality through the attention mechanism to highlight key defect representations; Step 3: Use a graph neural network to construct a spatial topology map of solder joints, model the spatial relationship between solder joints, input the graph neural network to learn the contextual features of associated defects, identify the associated defects of solder joints, and construct a spatial topology map of solder joints based on the physical layout of the PCB and the actual location of the solder joints. Each solder joint is considered a node in the spatial topology map of solder joints, and the physical connections or spatial proximity relationships between solder joints are considered edges. By extracting the coordinate information and connection relationships of the solder joints, the adjacency matrix and node feature matrix of the graph are generated. The constructed solder joint spatial topology map is input into the graph neural network, which aggregates the neighborhood information of each solder joint node and updates the node feature representation. A two-layer GCN is used, each layer including graph convolution, batch normalization, and ReLU activation. The first layer of GCN aggregates first-order neighborhood information, and the second layer aggregates second-order neighborhood information, outputting 128-dimensional contextual features. A graph attention layer is introduced after the GCN to dynamically calculate the weights of neighboring nodes, thereby expanding the solder joint features from local unimodal information to global contextual information. Based on contextual features, the solder joint defect type and correlation are predicted. A fully connected layer + Softmax is applied to output the defect type and confidence level. Contrastive learning loss is introduced to reduce the feature distance between solder joints with similar defects and increase the feature distance between solder joints with different defect types. Grad-CAM is then used to generate a graph-level heat map to highlight key neighborhood solder joints. Step 4: Design an independent sub-detector for each mode, perform detection based on the characteristics of different modal data, output the defect probability, and integrate the results of each sub-detector through DS evidence theory to comprehensively judge the solder joint status; Step 5: Build a reinforcement learning model based on historical inspection data. Use defect classification accuracy as the reward function to dynamically adjust the trust weights of each modal sub-detector. Model the multimodal inspection system as a reinforcement learning environment. Design the state space, define the action space, and construct the reward function. The state vector of the state space includes the current performance of each modality, the modal conflict level, and the distribution of solder joint features. The data is divided into a training set and a validation set. A reinforcement learning model is constructed using the DQN structure. Combined with the experience replay mechanism, the historical state-action-reward-new state quadruple is stored. At the same time, the dual DQN technology is introduced to separate the target network and the evaluation network. The trained reinforcement learning model is deployed for continuous improvement. After every 100 welds are inspected, the reinforcement learning model outputs an action based on the current state and applies this action to update the weights of each modality. Weight adjustment limits are pre-set, and a single adjustment cannot exceed 20% of the initial weight. If the accuracy drops for three consecutive rounds after adjustment, it automatically rolls back to the previous round of weight configuration. Simultaneously, the historical database is updated monthly to include the latest inspection data and manual re-inspection results. Step 6: Output the defect type and location based on the comprehensive fusion results and generate a visual report.

2. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 1, characterized in that: The step 1 specifically includes: Build a distributed sensing network that covers all dimensions of solder joints, perform multi-physics field synchronous sensing, and assign a unique identifier to each sensor. The distributed sensing network includes optical modules, X-ray modules, thermal modules, mechanical modules, and electrical modules. Using a star-bus hybrid architecture, the main controller connects to the optical, X-ray, and thermal master nodes via Ethernet. Each master node then extends the mechanical and electrical sub-nodes via the CAN bus. High-precision calibration plates are used to calibrate the optical and X-ray spatial coordinate systems. A pixel-to-physical coordinate conversion matrix is ​​established through feature point matching. Synchronous pulse signals are used to trigger all sensors to collect full-modal data to verify the spatiotemporal alignment benchmark. Multimodal data time synchronization is performed based on the PTP protocol. The main controller is designated as the PTP master clock and sends synchronization messages to each sensor sub-node. Each sensor sub-node becomes a slave clock. The slave clock calculates the link delay based on the timestamp in the message and adjusts the local clock offset. A high-speed oscilloscope is then used to synchronously capture the trigger signal of the optical camera and the frame synchronization signal of the X-ray detector to verify that the time deviation is less than 1μs. Data from different perspectives and modalities are mapped to a unified PCB coordinate system to construct a 3D holographic model of the solder joint. The ICP algorithm is used to match feature points on the calibration board, calculate the rigid body transformation matrix from the optical coordinate system to the X-Ray coordinate system, and superimpose the solder joint contours from optical inspection with the internal voids of the solder balls from X-Ray inspection to generate a "shell-core" composite model. The temperature gradient from thermal imaging, the stress distribution from mechanical sensors, and the impedance value from electrical testing are then annotated in the 3D model to form a thermal map of the solder joint health status.

3. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 2, characterized in that: The step 2 specifically includes: Feature extraction is performed on multimodal raw data including optical, X-ray, thermal, mechanical, and electrical data. The modal features are aligned to a unified time axis using PTP synchronized timestamps. Bilinear interpolation is used to account for spatial scale differences, and all features are mapped to the PCB physical coordinate system. Five sets of modal feature vectors are generated, with dimensions of 512, 128, 64, 32, and 48, respectively. The five sets of features are spliced ​​into a joint feature with a dimension of 784. Then, the correlation between modalities is analyzed through a multi-head self-attention mechanism. The outputs of the eight attention heads are spliced ​​and linearly transformed to generate a modal weight vector, which is then output as a weighted feature to highlight defect-sensitive modalities. The Grad-CAM algorithm is applied to the weighted features, and the defect classification loss is back-propagated to the feature layer to generate a heat map. The heat map is superimposed on the original optical image to intuitively display the defect location. The weighted features are input into the fully connected layer and the Softmax classifier to output the defect type and confidence.

4. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 1, characterized in that: The process of predicting solder joint defect types and correlations is as follows: The contextual features of each solder joint are processed to predict the defect type and relevance of the solder joint. The contextual features are input into a fully connected layer, which performs a nonlinear transformation on the features. The features are then mapped to different defect categories using a Softmax classifier, and the confidence score for each category is output. The Softmax classifier is then used to generate a probability distribution for each solder joint, indicating the probability of the solder joint belonging to different defect types. By using contrastive learning loss, the feature distances between similar defective solder joints are shortened, while the feature distances between different types of solder joints are extended. This results in similar defective solder joints being clustered in the feature space, while different types of solder joints are dispersed. The Grad-CAM algorithm is used to generate a graph-level heat map. The classification loss is back-propagated to the feature layer, and the contribution of each feature channel to defect classification is calculated. The generated graph-level heat map is then used to highlight the neighboring solder joints that have the greatest impact on defect classification, visually demonstrating the correlation between solder joints.

5. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 2, characterized in that: The step 4 specifically includes: Based on the characteristics of different modal data, independent sub-detectors are designed. Each sub-detector independently outputs the defect probability of the solder joint according to the characteristics of its modal data and defect mode. The defect probability output by each modal sub-detector is regarded as evidence and fused using DS evidence theory. The defect probability output by each sub-detector is converted into BPA as evidence support for the solder joint status. The evidence of different modalities is fused through the combination rules of DS evidence theory to calculate the comprehensive BPA and analyze the solder joint defect situation at the same time. Based on the fused BPA and solder joint defect analysis results, the comprehensive status judgment results of the solder joint are output, including defect type, comprehensive defect probability and confidence information.

6. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 5, characterized in that: The process of fusing evidence from different modalities is as follows: The 0-1 defect probability output by each modal sub-detector is converted into the distribution results required by DS theory, and an evidence support framework is constructed. In this framework, a BPA structure is designed to define the identification framework of solder joint states, including defective, non-defective, and uncertain. The defect probability output by the sub-detector is split into three parts: defect support, non-defect support, and uncertainty. Starting from the BPA of any two modalities, calculate the new combined BPA. If a conflict occurs, calculate the product of the BPAs of the two modalities to obtain the conflict probability, and redistribute the conflict probability according to the original support ratio of the two modalities to generate a new BPA. Combine the result of the previous step with the BPA of the third modality, repeat the conflict processing and redistribution until all modalities are integrated to generate the final comprehensive BPA. Based on the integrated BPA after fusion, the defect support value in the integrated BPA is directly taken and the confidence level is calculated, that is, 1-uncertainty. At the same time, dual thresholds are set, namely the defect threshold and the non-defect threshold. When the defect support level is greater than 0.8 and the confidence level is greater than 0.9, the system is judged as defective. When the non-defect threshold level is less than 0.3 or the confidence level is less than 0.7, the system is judged as non-defective. Other cases are marked as "pending re-inspection" to trigger manual review or supplementary inspection.

7. A PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 6, characterized in that: The step 5 specifically includes: The multimodal inspection system is modeled as a reinforcement learning environment. The state space is designed, the action space is defined, and the reward function is constructed. The state vector of the state space contains: the current performance of each modality, the modal conflict level, and the solder joint feature distribution. The current performance of each modality is the defect classification accuracy of the optical / X-Ray / thermal detector in the past 10 inspections. The modal conflict level is the average BPA conflict probability between modalities in the last five fusions. The solder joint feature distribution is the static properties of the size and material type of the solder joints in the current batch. The action is defined as the adjustment amount to the trust weight of each modality. The core reward is the comprehensive defect classification accuracy after fusion. If the accuracy of the current round improves by 2% compared with the previous round, the reward is +0.

5. If the accuracy decreases by 2%, the penalty is -0.

3. If the conflict rate decreases by 5%, an additional reward of +0.2 is received. We collected production line inspection data from the past six months and divided it into training and validation sets. We then used the DQN architecture to build a reinforcement learning model, incorporating an experience replay mechanism to store the historical state-action-reward-new state quadruple. We also introduced dual DQN technology to separate the target network and evaluation network, and set training termination conditions. The trained reinforcement learning model is deployed to the production line for real-time weight optimization and continuous system improvement. After inspecting every 100 solder joints, the reinforcement learning model outputs an action based on the current state and applies the action to update the weights of each modality. Weight adjustment limits are pre-designed, and the single adjustment range does not exceed 20% of the initial weight. If the accuracy rate drops for three consecutive rounds after adjustment, it will automatically roll back to the previous round of weight configuration. At the same time, the historical database is updated monthly to include the latest inspection data and manual re-inspection results.

8. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 7, characterized in that: The construction process of the reinforcement learning model is as follows: Six months of inspection records were extracted from the production line database, including the original inspection results, manual re-inspection labels, and initial weight configurations. A state vector was constructed to analyze modal performance and conflict levels. Modal performance was calculated by calculating the average accuracy of each modality over the past 10 inspections, and conflict level was calculated by calculating the average BPA conflict probability between modalities in the most recent five fusions. The data was then divided into training and validation sets in chronological order, generating a 15-dimensional state vector and a complete reward calculation basis. A reinforcement learning model for learning the optimal weight adjustment strategy is constructed using the DQN structure. The network architecture is designed, including input layer, hidden layer, and output layer. The input layer is the state vector, the hidden layer is a 2-layer fully connected layer, and BatchNorm is introduced to accelerate convergence. The output layer is the Q value of 5 actions. Combined with the experience replay mechanism, 100,000 historical quadruple data are stored, and random sampling breaks the data correlation. The dual DQN mechanism is used, and the evaluation network is used to calculate the current Q value, and the target network is used to calculate the target Q value. The target network parameters are synchronized every 100 rounds to stabilize the training process. The fluctuation of the verification set reward for 10 consecutive rounds is set to <0.05 as the training termination condition, thereby obtaining a reinforcement learning model that has been trained.

9. The PCBA circuit board solder joint detection method based on multimodal data fusion according to claim 8, characterized in that: The step 6 specifically includes: Comprehensively analyze the results of multimodal data fusion to identify the defect type and specific location of the solder joint. Generate a detailed visual report based on the fused data, presenting the inspection results in an intuitive manner while marking the location and type of the defect. Review the inspection results to identify false positives and missed detections. Record these cases, including the multimodal raw data of the solder joints, the model's inspection results, and the actual status after manual review. Conduct in-depth analysis of these false positives and missed detections to identify the causes. The data of false positive / missed positive cases are fed back into the training set of the model as new training samples for retraining and optimizing the model. By introducing false positive / missed positive cases, the model learns the feature patterns that it fails to correctly identify. Through continuous iterative optimization, a closed-loop system is formed from detection to feedback to optimization.

Images