A Quality Control Method for Optical Module Solder Joints Based on Multi-Energy X-ray and Causal Inference

Abstract

This application discloses a quality control method for optical module solder joints based on multi-energy spectral X-rays and causal reasoning, comprising: using a multi-energy spectral X-ray source to perform online penetration imaging on the solder joints of an optical module that has undergone reflow soldering, generating an enhanced image of the solder joint physical information; inputting the enhanced image of the solder joint physical information into a large model of a vision converter for deep semantic reasoning, obtaining a solder joint defect segmentation map and a defect confidence heatmap; extracting defect connected components based on the defect segmentation map and generating a defect spatial coordinate set; constructing a causal graph of solder joint defects based on the defect segmentation map and the defect confidence heatmap, identifying process root causes and counting the frequency of occurrence of similar causal paths, generating process intervention instructions carrying root cause labels; transmitting the process intervention instructions, the defect spatial coordinate set, and the root cause labels to a surface mount digital twin controller to execute a hierarchical response strategy, realizing closed-loop control from defect detection, root cause localization to process self-healing, and improving defect identification rate and quality control efficiency.

Description

Technical Field

[0001] This application relates to the field of intelligent manufacturing technology, and in particular to a method for quality control of optical module solder joints based on multi-energy spectral X-rays and causal reasoning. Background Technology

[0002] With the rapid development of 5G communication and data centers, optical modules, as core components, are experiencing increasingly higher packaging densities. The quality of solder joints directly affects the integrity and reliability of signal transmission. Traditional automated optical inspection (AOI) technology cannot penetrate the package to image the internal solder joints, leading to missed defects such as cold solder joints and missing solder balls. While X-ray inspection technology can detect the internal condition of solder joints, conventional single-energy X-ray inspection has poor imaging contrast for solder joints of different materials and thicknesses, making it difficult to accurately identify minute defects. Furthermore, existing quality inspection data is usually independent of the production control system, and inspection results cannot be translated into specific process adjustment instructions in real time. When systemic quality drift occurs, a large number of defective products often accumulate before triggering a shutdown, resulting in significant material waste and production delays. Summary of the Invention

[0003] The main purpose of this application is to provide a quality control method for optical module solder joints based on multi-energy spectral X-rays and causal reasoning, which aims to solve the technical problems of poor defect identification rate and quality control efficiency.

[0004] To achieve the above objectives, this application proposes a quality control method for optical module solder joints based on multi-energy spectral X-rays and causal reasoning. The method includes: Online penetration imaging of the solder joints of an optical module that has undergone reflow soldering is performed using a multi-energy spectrum X-ray source to acquire the original X-ray projection image, and an enhanced image of the physical information of the solder joint is generated based on the original X-ray projection image. The enhanced image of the physical information of the solder joint is input into the large model of the vision transformer to perform deep semantic reasoning of the internal structure of the solder joint, and the solder joint defect segmentation map and defect confidence heat map are obtained. Based on the defect segmentation map, the connected components of each defect are extracted, the centroid of each connected component is determined in the image coordinate system, and coordinate transformation is performed by combining the camera calibration parameters and the conveyor encoder signal to generate a defect space coordinate set with the optical module reference point as the origin. Based on the defect segmentation map and the defect confidence heatmap, a cause-and-effect map of solder joint defects in the current production batch is constructed to identify the process root causes that lead to the defects. The frequency of occurrence of the same cause-and-effect path in multiple consecutive optical modules is counted, and process intervention instructions carrying root cause labels are generated. The process intervention command, the defect spatial coordinate set, and the root cause label are transmitted to the surface mount digital twin controller, so that the digital twin controller executes a hierarchical response strategy based on the root cause label to achieve optical module solder joint quality control.

[0005] In one embodiment, the step of using a multi-energy spectral X-ray source to perform online penetration imaging on the solder joints of the optical module that have undergone reflow soldering, acquiring the original X-ray projection image, and generating an enhanced image of the solder joint physical information based on the original X-ray projection image includes: When the solder joint type is a bottom ball grid array package solder joint, the multi-energy spectrum X-ray source is controlled to switch to dual-energy spectrum mode, and online penetration imaging is performed on the bottom ball grid array package solder joint that has completed reflow soldering to acquire high-energy spectrum X-ray projection images and low-energy spectrum X-ray projection images. A material-sensitive fusion image is generated based on the high-energy-spectrum X-ray projection image and the low-energy-spectrum X-ray projection image using an energy spectrum decomposition algorithm; When the solder joint type is chip-level packaged solder joint, the multi-energy spectrum X-ray source is controlled to switch to single-energy spectrum mode, and online penetration imaging is performed on the chip-level packaged solder joint that has completed reflow soldering to acquire single-energy spectrum X-ray projection images. Multimodal spatiotemporal synchronous sensing and physical field reconstruction are performed based on the single-energy spectrum X-ray projection image to generate a physical enhancement feature map; The material-sensitive fusion image and / or the physical enhancement feature map are used as the physical information enhancement image of the solder joint.

[0006] In one embodiment, generating a material-sensitive fused image based on the high-energy-spectrum X-ray projection image and the low-energy-spectrum X-ray projection image using an energy spectrum decomposition algorithm includes: A time-series image pair sequence is constructed based on multiple consecutive frames of the high-energy spectral X-ray projection image and the low-energy spectral X-ray projection image; The theoretical displacement between adjacent frames in the time sequence of image pairs is determined based on the conveyor belt encoder signal, and a subpixel-level affine transformation is performed on the subsequent frame images in the time sequence of image pairs according to the theoretical displacement to obtain the registered time sequence of image pairs. Determine the grayscale standard deviation of the same solder joint region in the registered temporal image pair sequence. If the grayscale standard deviation exceeds a preset noise threshold, the corresponding solder joint region is marked as a dynamic artifact region. Temporal median filtering is applied to the dynamic artifact region to obtain a filtered temporal image pair sequence. The filtered time-series image pairs in the sequence are weighted and summed according to the inverse of their corresponding signal-to-noise ratios to generate a high-energy spectral fusion image and a low-energy spectral fusion image. Based on the difference in effective atomic number between the solder region and the substrate material in the high-energy spectrum fusion image and the low-energy spectrum fusion image, the weight coefficients of the energy spectrum decomposition algorithm are dynamically adjusted to obtain the adjusted weight coefficients. The high-energy spectrum fused image and the low-energy spectrum fused image are weighted and fused based on the adjusted weight coefficients to generate a material-sensitive fused image.

[0007] In one embodiment, the step of performing multimodal spatiotemporal synchronous sensing and physical field reconstruction based on the single-energy spectral X-ray projection image to generate a physical enhancement feature map includes: A thermal infrared imaging module is deployed between the reflow oven outlet and the X-ray inspection station to collect the cooling curve of the chip-level package solder joint from the peak temperature to the ambient temperature, and generate a transient heat distribution sequence. An acoustic emission sensor array is embedded between the mounting station and the reflow oven to collect phase change shrinkage and gas escape signals during the solder paste curing process in real time, and generate stress wave signal sequences. The spatial offset compensation amount of the thermal infrared station and acoustic emission station relative to the X-ray station is determined based on the constant speed of the conveyor belt. Based on the spatial offset compensation, the transient thermal distribution sequence and the single-energy X-ray projection image are cross-correlation function registered, and the stress wave signal sequence is time-shifted and event-aligned to generate the registered transient thermal distribution sequence and the registered stress wave signal sequence. A physical information neural network is constructed, wherein the physical information neural network includes a heat conduction branch and a mechanics branch, the heat conduction branch adopts a three-dimensional convolutional encoder-decoder structure, and the mechanics branch adopts a graph neural network structure; The registered transient heat distribution sequence is input into the heat conduction branch of the physical information neural network to obtain a three-dimensional temperature field. The registered stress wave signal sequence is input into the mechanical branch of the physical information neural network to obtain the nodal stress characteristics; Based on the three-dimensional temperature field and the nodal stress characteristics, the three-dimensional residual stress field and micropore distribution field inside the chip-level package solder joint are reconstructed. The three-dimensional residual stress field is compressed into a two-dimensional residual stress map through maximum intensity projection, and the micropore distribution field is compressed into a two-dimensional pore density map through volume integral. The two-dimensional residual stress map, the two-dimensional void density map, and the single-energy spectral X-ray projection image are stitched together along the channel dimension to generate a physical enhancement feature map.

[0008] In one embodiment, the reconstructing of the three-dimensional residual stress field and micropore distribution field inside the chip-scale package solder joint based on the three-dimensional temperature field and the nodal stress characteristics includes: The temperature gradient at each spatial location is determined based on the three-dimensional temperature field, and the thermal stress tensor field is determined based on the temperature gradient and the thermal expansion coefficient of the solder material. The nodal stress characteristics are mapped to the plastic strain increments of each discrete node of the weld joint using a multilayer perceptron and accumulated along the time axis to obtain the equivalent plastic strain. The thermal stress tensor field and the equivalent plastic strain are input together into a preset viscoplastic constitutive model to obtain a modified stress field; The temperature threshold in the three-dimensional temperature field is monitored in real time. When the temperature threshold is lower than the solidus temperature of the solder material, it is determined that the solder joint has entered the solid state, the three-dimensional temperature field is frozen, and the static equilibrium equation is solved based on the modified stress field to obtain the preliminary residual stress field. Based on the preliminary residual stress field, the hydrostatic stress and equivalent stress at each location are determined. Based on the hydrostatic stress, equivalent stress, and stress-driven cavity nucleation model, the micropore nucleation rate per unit volume is determined. Based on the micropore nucleation rate per unit volume, the pore growth process is simulated using atomic diffusion control equations to generate a micropore density distribution field. Using the micropore density distribution field as a continuous damage variable, the stiffness reduction correction is applied to the preliminary residual stress field to obtain the three-dimensional residual stress field inside the chip-level package solder joint.

[0009] In one embodiment, the step of inputting the enhanced image of the solder joint physical information into a large model of a vision transformer to perform deep semantic reasoning of the internal structure of the solder joint, and obtaining a solder joint defect segmentation map and a defect confidence heatmap, includes: The enhanced image of the physical information of the solder joint is divided into a sequence of image blocks of fixed size. Linear projection and position encoding are performed on each image block to generate an image block embedding sequence. The image patch embedding sequence is input into the encoder stack structure of the large visual transformer model, wherein the large visual transformer model includes an encoder stack structure and a decoder, the first encoder layer at the front end of the encoder stack structure is composed of a multi-head self-attention mechanism and a feedforward neural network, and at least one second encoder layer at the end of the encoder stack structure inserts a physical attention bias module on the basis of the multi-head self-attention mechanism and the feedforward neural network. The first encoder layer extracts local texture features and edge contour features from the image patch embedding sequence to obtain a shallow feature sequence. The shallow feature sequence is input into the second encoder layer. The physical attention bias module is activated in the second encoder layer. The attention weight bias matrix is ​​determined according to the material sensitivity information or residual stress field information. The attention weight bias matrix is ​​then superimposed on the attention weight matrix of the multi-head self-attention mechanism to obtain the superimposed attention weight. The shallow feature sequence is obtained by weighting the superimposed attention weights; The deep feature sequence is input into the decoder of the large visual transformer model. The decoder upsamples the deep feature sequence to obtain a pixel-level classification probability map. The pixel-level classification probability map is post-processed using a conditional random field to generate a solder joint defect segmentation map. Based on the pixel-level classification probability map, the probability of each pixel belonging to each type of defect is determined, and a defect confidence heatmap is generated.

[0010] In one embodiment, the step of extracting the connected components of each defect based on the defect segmentation map, determining the pixel coordinates of the centroid of each connected component in the image coordinate system, and performing coordinate transformation by combining camera calibration parameters and conveyor encoder signals to generate a defect spatial coordinate set with the optical module reference point as the origin includes: The weld joint defect segmentation map is binarized, and the connected components of each defect are extracted using the eight-neighbor connected component labeling algorithm. The pixel coordinates of the centroid of each connected component in the image coordinate system are determined based on the zeroth and first moments of each connected component. Obtain camera calibration parameters, and perform distortion correction and normalization processing on the pixel coordinates based on the camera calibration parameters to convert them into three-dimensional coordinates in the camera coordinate system; The physical displacement of the current optical module relative to the imaging station is determined based on the conveyor belt encoder signal. Based on the physical displacement and the camera extrinsic calibration results, the three-dimensional coordinates in the camera coordinate system are converted into coordinates in the world coordinate system. Identify the reference marker point of the optical module, establish a local coordinate system with the reference point of the optical module as the origin, transform the coordinates in the world coordinate system to the local coordinate system, and generate a defect space coordinate set.

[0011] In one embodiment, the step of constructing a cause-and-effect graph of solder joint defects for the current production batch based on the defect segmentation graph and the defect confidence heatmap, identifying the process root causes leading to the defects, and counting the frequency of occurrence of similar cause-and-effect paths in multiple consecutive optical modules, and generating process intervention instructions carrying root cause tags, includes: Define a three-level cause-effect graph structure containing process parameter nodes, physical state nodes, and defect type nodes, initialize the node set, and set inter-level unidirectional cause-effect constraints; Collect time-series observation data of each node in historical production batches, and determine the conditional independence score of any two nodes under the given subset of other nodes; Based on the conditional independence score, a constrained causal discovery algorithm is used to remove statistically independent node connections, and the direction of the remaining edges is optimized through a scoring search strategy to generate an initial causal adjacency matrix that represents the causal connection relationship between nodes. The matrix element values ​​of the initial causal adjacency matrix represent the existence of causal edges. Traverse each pair of nodes marked as having directed edges in the initial causal adjacency matrix and construct a structural equation model. The structural equation model can predict the values ​​of child nodes based on the values ​​of parent nodes and quantify the uncertainty of prediction. The production process parameters of the current production batch are used as the observation values ​​of the process parameter node, the physical information of the solder joint is used as the observation values ​​of the physical state node, and the solder joint defect segmentation map and the defect confidence heat map are mapped to the observation values ​​of the defect type node to obtain the observation data of the current batch. Based on the initial causal adjacency matrix and the current batch observation data, a causal graph of solder joint defects for the current production batch is constructed: The cause-effect graph of the solder joint defect is input into the structural equation model to perform counterfactual causal intervention calculations to obtain the change in the probability of defect occurrence. The dominant causal path is identified based on the change in the probability of the defect occurring, and the starting process parameters of the dominant causal path are marked as the process root cause of the defect, and a root cause label is generated. The frequency of occurrence of the same root cause tag in a continuously preset number of optical modules is counted. If the frequency of occurrence exceeds the dynamic drift threshold, a process intervention instruction carrying the root cause tag is generated.

[0012] Furthermore, to achieve the above objectives, this application also proposes a quality control device for optical module solder joints based on multi-energy spectral X-rays and causal reasoning. The quality control device for optical module solder joints based on multi-energy spectral X-rays and causal reasoning includes: The generation module is used to perform online penetration imaging on the solder joints of the optical module that have completed reflow soldering using a multi-energy spectrum X-ray source, acquire the original X-ray projection image, and generate an enhanced image of the physical information of the solder joint based on the original X-ray projection image; The reasoning module is used to input the enhanced image of the physical information of the solder joint into the large model of the vision converter to perform deep semantic reasoning of the internal structure of the solder joint, and obtain the solder joint defect segmentation map and the defect confidence heat map. The conversion module is used to extract the connected components of each defect based on the defect segmentation map, determine the pixel coordinates of the centroid of each connected component in the image coordinate system, and perform coordinate transformation by combining the camera calibration parameters and the conveyor encoder signal to generate a defect space coordinate set with the optical module reference point as the origin. The construction module is used to construct a cause-effect graph of solder joint defects in the current production batch based on the defect segmentation graph and the defect confidence heatmap, identify the process root causes that lead to the defects, count the frequency of occurrence of the same cause-effect path in multiple consecutive optical modules, and generate process intervention instructions carrying root cause labels. The transmission module is used to transmit the process intervention command, the defect spatial coordinate set, and the root cause label to the surface mount digital twin controller, so that the digital twin controller can execute a hierarchical response strategy according to the root cause label to achieve optical module solder joint quality control.

[0013] One or more technical solutions proposed in this application utilize multi-energy spectral X-ray sources to adaptively switch imaging modes, and reconstruct the physical field of the solder joint through dual-energy spectral imaging and energy spectral decomposition algorithms, or by combining infrared thermography and acoustic emission signals to generate physically enhanced images. This effectively distinguishes between metal compounds and void defects in the solder joint, improving detection accuracy. By embedding a large-scale visual transformer model with physical attention bias, defect segmentation maps and confidence heatmaps are obtained through reasoning. Furthermore, a three-level causal graph containing process, physical, and defect nodes is constructed, and counterfactual intervention calculations are performed to identify the dominant causal path and process root causes. The frequency of root causes is statistically analyzed to generate intervention commands, driving the digital twin controller to execute hierarchical responses. This achieves closed-loop control from defect detection and root cause localization to process self-healing, improving defect identification rate and quality control efficiency. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating an embodiment of the optical module solder joint quality control method based on multi-energy spectral X-rays and causal reasoning provided in this application. Figure 2 This is a schematic diagram of the module structure of the optical module solder joint quality control device based on multi-energy spectral X-rays and causal reasoning, as described in an embodiment of this application.

[0017] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0019] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0020] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a quality control device for optical module solder joints based on multi-energy spectral X-rays and causal inference. The following description uses a quality control device for optical module solder joints based on multi-energy spectral X-rays and causal inference as an example to illustrate this embodiment and the subsequent embodiments.

[0021] Based on this, embodiments of this application provide a method for quality control of optical module solder joints based on multi-energy spectral X-rays and causal reasoning, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the optical module solder joint quality control method based on multi-energy spectral X-rays and causal reasoning in this application.

[0022] In this embodiment, the optical module solder joint quality control method based on multi-energy spectral X-rays and causal reasoning includes steps S10~S50: Step S10: Use a multi-energy X-ray source to perform online penetration imaging on the solder joints of the optical module that have completed reflow soldering, acquire the original X-ray projection image, and generate an enhanced image of the physical information of the solder joint based on the original X-ray projection image.

[0023] It should be noted that a multi-energy spectral microfocus X-ray source is deployed at the reflow oven exit on the high-speed optical module surface mount production line. A multi-energy spectral X-ray source is a device capable of emitting X-rays of different energies. The emitted X-rays can penetrate the solder joints of the optical module and interact differently with different materials within the solder joints, thereby acquiring richer information about the internal structure of the solder joints during imaging. In this embodiment, the multi-energy spectral X-ray source can adaptively switch imaging modes according to the type of solder joint.

[0024] Understandably, when the optical module that has completed reflow soldering is transferred to the X-ray inspection station, the X-ray source is triggered to perform online penetration imaging. Before performing online penetration imaging, the solder joint type needs to be detected, and the operating mode is switched according to the solder joint type. In this embodiment, the solder joint types include bottom ball grid array (BGA) package solder joints and chip-level package solder joints. For bottom BGA package solder joints, a dual-energy spectral imaging mode is used, acquiring projection images at low and high energy thresholds, such as 20-40keV and 40-60keV respectively. For chip-level package solder joints, a single-energy spectral imaging mode is used to acquire high-resolution projection images.

[0025] It is worth noting that the acquired raw X-ray projection images undergo physical information enhancement processing to generate physical information enhanced images of the solder joints. The specific enhancement method is determined according to the solder joint type and imaging mode: for dual-energy spectral mode, a material-sensitive fusion image is generated through an energy spectral decomposition algorithm, which separates information from different materials, thereby better distinguishing between metallic compounds and void defects in the solder joint; for single-energy spectral mode, a physical enhancement feature map is generated through physical field reconstruction by combining thermal infrared and acoustic emission multimodal data.

[0026] Step S20: Input the enhanced image of the physical information of the solder joint into the large model of the vision converter to perform deep semantic reasoning of the internal structure of the solder joint, and obtain the solder joint defect segmentation map and the defect confidence heat map.

[0027] It should be noted that the large-scale visual transformer model refers to a complete, end-to-end deep learning system architecture, employing an encoder-decoder architecture. The encoder is a stacked structure composed of multiple encoder layers, used for feature extraction from the input image. In this embodiment, the first encoder layer at the front of the encoder stack consists of a multi-head self-attention mechanism and a feedforward neural network. The second encoder layer at the rear inserts a physical attention bias module on top of the multi-head self-attention mechanism and feedforward neural network. That is, a special physical attention bias module is added to the last few encoder layers. This physical attention bias module can determine the attention weight bias matrix based on material sensitivity information or residual stress field information, thereby forcing the attention weights to focus on physically anomalous regions. The decoder adopts a progressive upsampling structure, gradually restoring the spatial resolution of the image through upsampling operations to generate pixel-level classification results.

[0028] Understandably, the large model output of the vision transformer produces a pixel-level solder joint defect segmentation map with the same size as the input image, along with a confidence heatmap indicating the defect category for each pixel. The defect segmentation map categorizes solder joint areas into types such as normal solder joints, cold solder joints, missing solder balls, bridging, and poor wetting.

[0029] It is understood that the large-scale visual transformer model in this embodiment is pre-trained, and the training data comes from a large number of enhanced image samples of solder joint physical information. These samples cover images of different types of solder joints under various imaging modes. During the pre-training process, the model continuously learns the features and patterns in the samples and adjusts its own parameters to improve its understanding of the internal structure of the solder joints and its defect recognition ability. The pre-training of the large-scale visual transformer model uses the AdamW optimizer with a weight decay of 0.05 to suppress overfitting, and an initial learning rate of lr. base =1×10 4 A cosine annealing strategy is employed, coupled with a warm-up mechanism. The warm-up steps account for 10% of the total steps, during which the learning rate is increased from 1×10⁻⁶. 7 linearly increasing to the initial learning rate lr base Then, following the cosine curve, it decreases to 1×10. 6 The batch size per run is set to 32, the total batch size is 256, 8 GPUs operate in parallel, and there are a total of training rounds. During training, the probability p is used to determine the training order. drop To enhance model robustness, a portion of the encoder layers is randomly dropped with a smoothing coefficient of 0.1. This prevents the model from overconfident in noisy labels. After training, the model is evaluated using an independent test set, with key metrics including mean Intersection over Union (mIoU), defect recall, and false positive rate (FPR). The target metrics are mIoU > 88%, small holes < 5 pixels, and recall > 92%. After training, the model weights are FP16 half-precision quantized, and the TensorRT engine is used to accelerate inference, compressing the single-frame processing latency to within 15ms to meet the requirements of online real-time detection.

[0030] In one feasible implementation, step S20 may include: segmenting the weld point physical information enhancement image into a sequence of image blocks of fixed size, performing linear projection and position encoding on each image block to generate an image block embedding sequence; inputting the image block embedding sequence into the encoder stack structure of the visual transformer large model, wherein the visual transformer large model includes an encoder stack structure and a decoder, the first encoder layer at the front end of the encoder stack structure is composed of a multi-head self-attention mechanism and a feedforward neural network, and at least one second encoder layer at the end of the encoder stack structure inserts a physical attention bias module on the basis of the multi-head self-attention mechanism and the feedforward neural network; extracting local texture features and edge contour features from the image block embedding sequence through the first encoder layer to obtain a shallow feature sequence; and inputting the image block embedding sequence into the encoder stack structure. The shallow feature sequence is input to the second encoder layer. Within the second encoder layer, a physical attention bias module is activated. An attention weight bias matrix is ​​determined based on material sensitivity information or residual stress field information. This matrix is ​​then superimposed on the attention weight matrix of the multi-head self-attention mechanism to obtain superimposed attention weights. The shallow feature sequence is weighted using these superimposed attention weights to obtain a deep feature sequence. This deep feature sequence is input to the decoder of the large visual transformer model. The decoder upsamples the deep feature sequence to obtain a pixel-level classification probability map. The pixel-level classification probability map is post-processed using a conditional random field to generate a weld joint defect segmentation map. Based on the pixel-level classification probability map, the probability of each pixel belonging to each type of defect is determined, generating a defect confidence heatmap.

[0031] It should be noted that in this embodiment, the encoder of the large-scale visual transformer model is described as consisting of 12 stacked encoder layers, with the first 9 layers being the first encoder layer and the last 3 layers being the second encoder layer. The specific number of encoder layers and the division between the first and second encoder layers can be adjusted according to specific needs and experimental results. The first encoder layer consists of a multi-head self-attention mechanism and a feedforward neural network, and is mainly responsible for extracting local texture features and edge contour features from the input image patch embedding sequence. The multi-head self-attention mechanism can capture long-distance dependencies between image patches, while the feedforward neural network performs nonlinear transformations and enhancements on the extracted features. The second encoder layer inserts a physical attention bias module on top of the multi-head self-attention mechanism and the feedforward neural network.

[0032] Understandably, the physically enhanced image is divided into P×P image blocks, linearly projected into vectors, and encoded with physical location information, including the relative coordinates of the solder joints on the optical module and material type identification information, to obtain an image block embedding sequence. In the first encoder layer, a multi-head self-attention mechanism determines the similarity between image blocks and assigns different weights to each image block, thereby capturing long-distance dependencies between image blocks. The feedforward neural network performs nonlinear transformations and enhancements on the extracted features, enabling the model to learn more complex feature representations. Capturing long-distance dependencies between image blocks includes the extraction of local texture features and edge contour features, based on the grayscale value changes and spatial distribution of pixels within the image block. By analyzing information such as the rate of change of pixel grayscale values ​​and gradient direction within the image block, local texture features of the solder joints, such as the surface roughness and graininess of the solder joints, can be extracted; while edge contour features can be determined by detecting abrupt changes in grayscale values ​​within the image block, which typically correspond to the edges of the solder joints.

[0033] When the shallow feature sequence enters the second encoder layer, the physical attention bias module is activated. This module determines the attention weight bias matrix based on material-sensitive information or residual stress field information, and also according to the solder joint type. For solder joints in the bottom ball grid array package of the optical module, the attention weight bias matrix is ​​determined based on material-sensitive information, which comes from the material-sensitive fusion image obtained by the energy spectrum decomposition algorithm. This matrix focuses the attention weights on metal compounds and void defect regions in the solder joint. For chip-level packaged solder joints, the attention weight bias matrix is ​​determined based on residual stress field information, which comes from the physical enhancement feature map generated by physical field reconstruction. Superimposing the attention weight bias matrix onto the attention weight matrix of the multi-head self-attention mechanism makes the model pay more attention to physically anomalous regions, thereby improving the ability to identify internal defects in the solder joint.

[0034] The shallow feature sequence is weighted by superimposed attention weights to obtain a deep feature sequence containing more physical information. This deep feature sequence is then input into the decoder of the large-scale visual transformer model. The decoder employs a progressive upsampling structure to gradually restore the spatial resolution of the image, generating a pixel-level classification probability map. To further optimize the classification results, a Conditional Random Field (CRF) is used for post-processing of the pixel-level classification probability map. The CRF considers the correlation between adjacent pixels and smooths and corrects the pixel-level classification results by establishing a joint probability distribution among pixels, thereby generating a more accurate solder joint defect segmentation map. This map clearly divides the solder joint region into different categories, such as normal solder joints, cold solder joints, missing solder balls, bridging, and poor wetting. Simultaneously, based on the pixel-level classification probability map, the probability of each pixel belonging to each defect category can be determined, thus generating a defect confidence heatmap. This heatmap uses different colors or grayscale values ​​to represent the probability of each pixel belonging to a defect category; the higher the probability, the darker the color.

[0035] Traditional visual transformers rely solely on data-driven learning of attention. This embodiment explicitly injects physical laws, such as the fact that high-stress areas are more prone to cracking, through an attention weight bias matrix, forcing the model to focus on key areas and improving generalization ability and reasoning interpretability under small sample sizes.

[0036] Step S30: Based on the defect segmentation map, extract the connected components of each defect, determine the pixel coordinates of the centroid of each connected component in the image coordinate system, and perform coordinate transformation by combining the camera calibration parameters and the conveyor encoder signal to generate a defect spatial coordinate set with the optical module reference point as the origin.

[0037] It should be noted that, based on the defect segmentation map, an eight-neighbor connected component labeling algorithm is used to extract the connected components of each defect instance. The pixel coordinates (u, v) of the centroid in the image coordinate system are determined based on the zero-order and first-order geometric moments of each connected component. Pre-calibrated camera calibration parameters, including the intrinsic parameter matrix K and distortion coefficients, are obtained, and distortion correction is performed on the pixel coordinates to obtain the three-dimensional coordinates in the camera coordinate system.

[0038] The conveyor belt encoder pulse count is read to obtain the conveyor belt encoder signal, which represents the displacement information of the optical module on the conveyor belt. Combining the camera calibration parameters and the conveyor belt encoder signal, the three-dimensional coordinates in the camera coordinate system are converted into a defect spatial coordinate set with the optical module's reference point as the origin using a coordinate transformation formula. This defect spatial coordinate set can accurately locate the actual spatial position of each solder joint defect within the optical module, providing accurate location information for subsequent quality control and repair work.

[0039] In one feasible implementation, step S30 may include: binarizing the weld joint defect segmentation map and extracting the connected components of each defect using an eight-neighbor connected component labeling algorithm; determining the pixel coordinates of the centroid of each connected component in the image coordinate system based on the zero-order moment and first-order moment of each connected component; obtaining camera calibration parameters and performing distortion correction and normalization processing on the pixel coordinates based on the camera calibration parameters to convert them into three-dimensional coordinates in the camera coordinate system; determining the physical displacement of the current optical module relative to the imaging station based on the conveyor encoder signal, and converting the three-dimensional coordinates in the camera coordinate system into coordinates in the world coordinate system according to the physical displacement and the camera extrinsic parameter calibration results; identifying the optical module reference marker point, establishing a local coordinate system with the optical module reference point as the origin, and converting the coordinates in the world coordinate system to the local coordinate system to generate a defect space coordinate set.

[0040] It should be noted that binarization of the solder joint defect segmentation image is performed to convert it into a form suitable for connected component labeling algorithms. By setting an appropriate threshold, the pixel values ​​in the image are divided into two categories: foreground, representing the defect area, and background, representing the non-defect area. Pixel values ​​in the defect area are set to 1, and pixel values ​​in the background area are set to 0, which clearly highlights the boundaries of the defects.

[0041] The eight-neighbor connected component labeling algorithm is used to identify sets of interconnected pixels in an image. In a binarized weld joint defect segmentation map, this algorithm iterates through each pixel, labeling interconnected foreground pixels as belonging to the same connected component. Specifically, for each foreground pixel, the algorithm checks its eight neighboring pixels; if a neighboring pixel is also a foreground pixel, it labels them as belonging to the same connected component. In this way, individual defect instances in the weld joint defect segmentation map can be separated, and each defect instance can be assigned a unique label.

[0042] Determining the zeroth and first moments of each connected component is necessary to calculate the pixel coordinates of the centroid of each component in the image coordinate system. The zeroth moment represents the total number of pixels within the connected component, while the first moment reflects the distribution of pixels in the x and y directions. The pixel coordinates (u, v) of the centroid of the connected component can be obtained using the zeroth and first moments, as shown in the following equation: u=M 10 / M 00 v=M 01 / M 00 Among them, M 00 It is the zeroth moment, M 10 and M 01 These are the first moments in the x and y directions, respectively.

[0043] Obtaining camera calibration parameters is crucial for converting pixel coordinates in the image coordinate system to 3D coordinates in the camera coordinate system. These parameters include the intrinsic parameter matrix K and distortion coefficients. K describes the camera's internal parameters, such as focal length and principal point coordinates, while the distortion coefficients correct for lens distortion. By performing distortion correction on the pixel coordinates, more accurate 3D coordinates (x, y) in the camera coordinate system can be obtained.

[0044] The physical displacement of the optical module relative to the imaging station is determined based on the conveyor belt encoder signal. The conveyor belt encoder records the distance the optical module moves on the conveyor belt. By reading the encoder's pulse count, i.e., the conveyor belt encoder signal, the displacement information of the optical module can be obtained. Based on the physical displacement and the camera extrinsic parameter calibration results, the three-dimensional coordinates in the camera coordinate system are converted to coordinates in the world coordinate system. The camera extrinsic parameter calibration results, i.e., the hand-eye calibration matrix, describe the relative positional relationship between the camera and the robot hand. This matrix can be used to transform the coordinates in the camera coordinate system to the world coordinate system, as shown in the following equation: in, The coordinate vector in the world coordinate system. The hand-eye calibration matrix is ​​obtained from camera extrinsic calibration, where (x, y) are the coordinates in the camera coordinate system, and Z is the depth value. This refers to the physical displacement of the optical module relative to the imaging station, determined based on the conveyor encoder signal.

[0045] Identifying the reference markers of the optical module and establishing a local coordinate system with the reference point as its origin is to transform the coordinates from the world coordinate system to a local coordinate system that better suits the needs of practical applications. The reference markers of the optical module are typically points on the module with specific characteristics. Image processing algorithms are used to identify the positions of these markers, thereby determining the origin and coordinate axis directions of the local coordinate system. Transforming the coordinates from the world coordinate system to the local coordinate system generates a defect spatial coordinate set with the reference point of the optical module as its origin. This defect spatial coordinate set eliminates errors caused by camera distortion and conveyor belt movement, achieving precise location of the defect in the physical world and providing accurate spatial guidance for subsequent rework robots or process adjustments.

[0046] Step S40: Based on the defect segmentation map and the defect confidence heatmap, construct the cause-effect map of solder joint defects in the current production batch, identify the process root causes that lead to the defects, and count the frequency of occurrence of the same cause-effect path in multiple consecutive optical modules, and generate process intervention instructions carrying root cause labels.

[0047] It should be noted that, based on the defect segmentation map and defect confidence heatmap, combined with process parameters obtained in real time from the Manufacturing Execution System (MES), a cause-and-effect graph of solder joint defects for the current production batch is constructed. The solder joint defect cause-and-effect graph contains three layers of nodes: a process parameter layer, such as stencil tension, scraper speed, and reflow peak temperature; a physical state layer, such as cooling rate, residual stress peak, and micropore density; and a defect type layer, such as cold solder joints and cracks. Through counterfactual intervention calculations, the dominant causal path leading to the current defect is identified, and the process parameter at the beginning of the path is marked as the root cause, generating a root cause label. The frequency of occurrence of similar causal paths in multiple consecutive optical modules is statistically analyzed; a high frequency indicates a relatively stable and significant causal relationship. Based on the root cause label and frequency of occurrence, process intervention instructions carrying the root cause label are generated. For example, if insufficient stencil tension is found to be the root cause of cold solder joint defects, and this causal path frequently occurs in multiple optical modules, then the process intervention instruction could be to appropriately increase the stencil tension to a reasonable range.

[0048] In one feasible implementation, step S40 may include: defining a three-level causal graph structure containing process parameter nodes, physical state nodes, and defect type nodes; initializing the node set and setting inter-layer unidirectional causal constraints; collecting time-series observation data of each node in historical production batches and determining the conditional independence score of any two nodes under the condition of a given subset of other nodes; based on the conditional independence score, removing statistically independent node connections using a constraint-based causal discovery algorithm and optimizing the direction of the remaining connections through a scoring search strategy to generate an initial causal adjacency matrix representing the causal connection relationship between nodes, wherein the matrix element values ​​of the initial causal adjacency matrix represent the existence of causal edges; traversing each pair of nodes marked as having directed edges in the initial causal adjacency matrix and constructing a structural equation model, wherein the structural equation model can predict the values ​​of child nodes based on the values ​​of parent nodes and quantify the prediction uncertainty. The production process parameters of the current production batch are used as the observation values ​​of the process parameter node, and the physical information of the solder joint is used as the observation value of the physical state node. The solder joint defect segmentation map and the defect confidence heatmap are mapped to the observation values ​​of the defect type node to obtain the observation data of the current batch. Based on the initial causal adjacency matrix and the observation data of the current batch, a solder joint defect causal graph of the current production batch is constructed. The solder joint defect causal graph is input into the structural equation model to perform counterfactual causal intervention calculation to obtain the change in the probability of defect occurrence. The dominant causal path is identified according to the change in the probability of defect occurrence, and the starting process parameter of the dominant causal path is marked as the process root cause of the defect, and a root cause label is generated. The frequency of occurrence of the same root cause label in a consecutive preset number of optical modules is counted. If the frequency of occurrence exceeds the dynamic drift threshold, a process intervention instruction carrying the root cause label is generated.

[0049] It's important to note that a three-level cause-effect graph is a graphical model used to describe the causal relationships between process, physics, and defects. Process nodes represent various process parameters in the production of optical module solder joints, such as stencil tension, scraper speed, and reflow peak temperature; physics state nodes represent the physical states of the solder joints during production, such as cooling rate, residual stress peak, and micropore density; and defect type nodes represent various possible defect types of the solder joints, such as cold solder joints and cracks. By constructing such a three-level cause-effect graph, the causal relationships between process parameters, physical states, and defect types can be clearly demonstrated.

[0050] Understandably, all potential node sets are initialized, and unidirectional causal prior constraints are set from the process layer to the physical layer and from the physical layer to the defect layer, prohibiting the existence of reverse edges and cross-layer jump edges, in order to ensure the logical rationality of the causal graph.

[0051] When collecting time-series observation data for each node in historical production batches, it is essential to ensure the accuracy and completeness of the data. To determine the conditional independence score of any two nodes under given subsets of other nodes, the chi-square test can be used. This involves calculating the partial correlation coefficient of any node i and j under a given condition set S using historical data, and then determining the chi-square statistic. If the chi-square statistic is less than a preset threshold, then i and j are considered conditionally independent under the given S. For example, when determining the conditional independence of the stencil tension node and the weld defect node under given scraper speed and reflow peak temperature, if their chi-square statistics are less than the preset threshold, it indicates that the stencil tension and weld defect are conditionally independent under given scraper speed and reflow peak temperature, as shown in the following formula: in, It is the partial correlation coefficient after Fisher transformation. It is the partial correlation coefficient between nodes i and j under a given condition set S. This is the corresponding chi-square statistic, where n is the sample size. It is the number of elements in the condition set S.

[0052] By comparing the chi-square statistic with a preset critical value, it can be determined whether node pairs i and j are conditionally independent under a given condition set S.

[0053] Based on conditional independence scores, a constrained causal discovery algorithm is used to remove statistically independent node connections. This algorithm removes connections between data points that satisfy conditional independence based on the calculated conditional independence scores, thus simplifying the causal graph structure and reducing unnecessary interference. For example, if the steel mesh tension node and the weld defect node are calculated to be conditionally independent under given scraper speed and reflow peak temperature conditions, the connection between these two nodes can be removed. Then, a scoring search strategy is used to optimize the direction of the remaining edges. A suitable scoring function, such as the Bayesian information criterion, can be selected to find the optimal edge direction, generating an initial causal adjacency matrix representing the causal connections between nodes. The elements of this matrix are either 0 or 1, where 1 indicates the existence of a causal edge and 0 indicates its absence. The scoring function is as follows: in, The value of the scoring function. n is the number of nodes, and n is the number of samples. It is the residual variance of node i. It is the number of parent nodes of node i. It is the observation value of node i in the j-th sample. Let i be the parent node observation of node i in the j-th sample. These are the estimated values ​​of the structural equation model parameters, obtained through the least squares method or the maximum likelihood method. This is the function used to fit the structural equation.

[0054] When constructing a structural equation model (SEM), each pair of nodes marked with directed edges in the initial causal adjacency matrix must be traversed. This requires considering the causal relationships between nodes and the characteristics of the data. SEM not only predicts child node values ​​based on parent node values ​​but also quantifies the uncertainty of the prediction, which is crucial for accurately analyzing causal relationships. For example, regarding the causal relationship between stencil tension and weld residual stress, a SEM can predict the value of weld residual stress based on the stencil tension value, while simultaneously providing the confidence interval to quantify the uncertainty. When constructing the SEM, an appropriate functional form should be chosen to describe the relationships between nodes, such as linear or nonlinear functions. Different functions should be used for fitting different types of node relationships to ensure the model accurately reflects the causal connections between nodes.

[0055] When mapping the production process parameters, solder joint physical information, solder joint defect segmentation map, and defect confidence heatmap of the current production batch to the observation values ​​of the corresponding nodes, the accuracy and consistency of the data must be ensured, and the data should be used as the observation data of the current batch.

[0056] When constructing the causal graph of solder joint defects for the current production batch based on the initial causal adjacency matrix and the current batch observation data, the edges corresponding to the currently active nodes in the initial causal adjacency matrix are retained and assigned posterior weights based on the observation data, while the edges of inactive nodes are set to zero or weakened, generating a weighted causal adjacency matrix specific to the current batch. Combining the node observation state and the weighted causal adjacency matrix, a complete causal graph of solder joint defects for the current batch is formed.

[0057] When the constructed cause-effect graph of weld joint defects is input into the structural equation model for counterfactual causal intervention calculation, a virtual intervention operation is performed on a specific process parameter node, fixing its value and disconnecting its parent node connection, deriving the change in the probability of defect occurrence, and generating counterfactual prediction results.

[0058] The natural direct and indirect effects of each process parameter on the defect are determined based on the change in the probability of defect occurrence. The natural direct effect is the direct influence that bypasses the physical state node, and the natural indirect effect is the indirect influence transmitted through the physical state node. The dominant causal path with the highest contribution whose effect value exceeds a preset threshold is identified, and the starting process parameter node of the dominant causal path is marked as the process root cause. A root cause label is generated, which includes the root cause name, the quantitative value of the degree of influence, the ratio of natural direct effect to natural indirect effect, and the recommended adjustment direction. A sliding window queue is maintained in memory, and the frequency of the same root cause label in a consecutive preset number of optical modules is counted. If the frequency exceeds the dynamic drift threshold calculated based on the historical mean and variance, a process intervention instruction carrying the root cause label, urgency indicator, suggested correction amount, and confidence interval is generated.

[0059] Step S50: Transmit the process intervention command, the defect spatial coordinate set, and the root cause label to the surface mount digital twin controller, so that the digital twin controller executes a hierarchical response strategy according to the root cause label to achieve optical module solder joint quality control.

[0060] It should be noted that the process intervention command defect spatial coordinate set and root cause label are transmitted to the surface mount digital twin controller in real time. The surface mount digital twin controller is an intelligent system used to simulate and control the surface mount process. After receiving this information, it will activate the graded response strategy based on the detailed content contained in the root cause label, such as the root cause name, the quantitative value of the degree of impact, and the proportion of natural direct effects and natural indirect effects.

[0061] During initialization, the digital twin controller loads a safety state machine, defining three states: normal, warning, and meltdown, along with their transition conditions. When the maximum value of the defect confidence heatmap exceeds a preset threshold in a single inspection and the root cause label indicates complete missing solder balls, the controller immediately switches to the meltdown state. In the meltdown state, the controller sends an emergency power-off signal to the reflow oven, locks the conveyor motor, and activates nitrogen inertia protection. Complete process parameters, image hashes, and operation logs at the meltdown moment are packaged, encrypted, and uploaded to the security audit server. Exiting the meltdown state is only permitted after two authorized engineers have respectively entered biometric credentials and a dynamic password. The meltdown event is included in the equipment reliability index, and after system recovery, a full re-inspection of all solder joints on the last multiple optical modules before the meltdown is automatically initiated.

[0062] When in alert mode, the digital twin controller fine-tunes production process parameters based on the specific information in the root cause label. For example, if the root cause label indicates that stencil tension is the root cause of defects and its quantified impact is high, indicating a significant direct effect, the controller will adjust the direction accordingly, appropriately increasing or decreasing the stencil tension. Simultaneously, the controller monitors weld defects in real time; if the defects do not improve or even worsen, it will further intensify adjustments.

[0063] Under normal conditions, the digital twin controller continuously receives process intervention commands, defect spatial coordinate sets, and root cause labels, and monitors the production process in real time. It analyzes this data to determine if production anomalies are likely. If it detects that certain process parameters are trending towards potential defects, it will issue an early warning to alert operators.

[0064] For the set of spatial coordinates of defects, the digital twin controller will match them with the actual location of the solder joint to accurately locate the location of the defect, quickly find the problematic solder joint, and carry out targeted treatment.

[0065] This embodiment provides a quality control method for optical module solder joints based on multi-energy spectral X-rays and causal reasoning. By utilizing a multi-energy spectral X-ray source to adaptively switch imaging modes, and reconstructing the physical field of the solder joint through dual-energy spectral imaging and energy spectral decomposition algorithms, or by combining infrared thermography and acoustic emission signals, a physically enhanced image is generated. This effectively distinguishes between metal compounds and void defects in the solder joint, improving detection accuracy. By embedding a large-scale visual transformer model with physical attention bias, a defect segmentation map and a confidence heatmap are obtained through reasoning. Then, a three-level causal graph containing process, physical, and defect nodes is constructed, and counterfactual intervention calculations are performed to identify the dominant causal path and process root causes. The frequency of root causes is statistically analyzed to generate intervention commands, driving a digital twin controller to execute hierarchical responses. This achieves closed-loop control from defect detection and root cause localization to process self-healing, improving defect identification rate and quality control efficiency.

[0066] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, step S50 includes steps S101 to S105: Step S101: When the solder joint type is bottom ball grid array package solder joint, control the multi-energy spectrum X-ray source to switch to dual-energy spectrum mode, and perform online penetration imaging on the bottom ball grid array package solder joint that has completed reflow soldering, and acquire high-energy spectrum X-ray projection images and low-energy spectrum X-ray projection images.

[0067] It should be noted that the bottom ball grid array (BGA) package has solder joints distributed at the bottom of the package in a spherical arrangement. With this type of package, traditional inspection methods may struggle to effectively detect defects within the solder joints, while dual-energy X-ray spectroscopy offers unique advantages. High-energy X-rays have strong penetrating power, capable of penetrating the solder joints and surrounding packaging material to obtain general structural information about the solder joints; low-energy X-rays are more sensitive to absorption differences in different substances, allowing for a clearer display of the distribution of different components within the solder joint.

[0068] Understandably, in bottom ball grid array (BGA) packaging scenarios, the X-ray source is controlled to emit X-rays of two different energies, such as 80kV and 120kV. Since the solder, such as a tin-silver-copper alloy, and the substrate, such as FR4 or ceramic, exhibit significantly different attenuation coefficients at different energies, dual-energy spectral image pairs are acquired, namely high-energy spectral X-ray projection images and low-energy spectral X-ray projection images. In this embodiment, multiple consecutive frames of high-energy and low-energy spectral images are acquired to improve the signal-to-noise ratio and detection accuracy.

[0069] Step S102: Generate a material-sensitive fusion image based on the high-energy-spectrum X-ray projection image and the low-energy-spectrum X-ray projection image using an energy spectrum decomposition algorithm.

[0070] It should be noted that the energy spectrum decomposition algorithm is an algorithm that can effectively integrate information from dual-energy spectrum images based on the differences in material attenuation characteristics in X-ray projection images of different energies. When processing high-energy and low-energy X-ray projection images using this algorithm, the attenuation coefficients of materials such as solder and substrate are analyzed at different energies. Since solder and substrate exhibit different attenuation characteristics at high and low energies, these characteristics can be quantified and extracted using the energy spectrum decomposition algorithm. Information about the general internal structure of the solder joint from the high-energy spectrum image is fused with detailed information about the distribution of different components from the low-energy spectrum image to generate a material-sensitive fused image. This fused image can more clearly and accurately reflect the distribution and structural characteristics of different materials inside the solder joint of the bottom ball grid array package, providing a richer and more reliable data foundation for subsequent defect detection and analysis.

[0071] In one feasible implementation, step S102 may include: constructing a time-series image pair sequence based on multiple consecutive frames of the high-energy-spectrum X-ray projection image and the low-energy-spectrum X-ray projection image; determining the theoretical displacement between adjacent frames in the time-series image pair sequence based on the conveyor belt encoder signal, and performing a sub-pixel-level affine transformation on the subsequent frame image in the time-series image pair sequence according to the theoretical displacement to obtain a registered time-series image pair sequence; determining the grayscale standard deviation of the same solder joint area in the registered time-series image pair sequence, and if the grayscale standard deviation exceeds a preset noise threshold, marking the corresponding solder joint area as dynamic. Artifact regions; perform temporal median filtering on the dynamic artifact regions to obtain a filtered temporal image pair sequence; sum the temporal image pairs in the filtered temporal image pair sequence by weighting according to the inverse of the corresponding signal-to-noise ratio to generate a high-energy spectral fusion image and a low-energy spectral fusion image; dynamically adjust the weight coefficients of the energy spectrum decomposition algorithm based on the difference in effective atomic numbers between the solder region and the substrate material in the high-energy spectral fusion image and the low-energy spectral fusion image to obtain adjusted weight coefficients; perform weighted fusion of the high-energy spectral fusion image and the low-energy spectral fusion image based on the adjusted weight coefficients to generate a material-sensitive fusion image.

[0072] It should be noted that when the multi-energy X-ray source is switched to dual-energy mode, high-energy and low-energy projection images of the optical module solder joints are continuously acquired at a frame rate of 30fps to construct a time-series image pair. The conveyor belt encoder signal is read to determine the theoretical displacement between adjacent frames. The conveyor belt encoder signal is the encoder's pulse count; the theoretical displacement between adjacent frames can be determined based on the encoder's pulse count and resolution. Based on this theoretical displacement, a sub-pixel-level affine transformation is performed on the subsequent frames in the sequence. This sub-pixel-level affine transformation can perform translation, rotation, and scaling transformations on the image at the sub-pixel level to ensure precise alignment between adjacent frames. This transformation effectively eliminates image offset and rotation problems caused by conveyor belt movement, resulting in a registered time-series image pair.

[0073] The standard deviation of grayscale values ​​for the same solder joint region in the registered time-series image pairs is calculated. This standard deviation reflects the fluctuation of grayscale values ​​in that region. If the standard deviation exceeds a preset noise threshold, it indicates the presence of dynamic artifacts in that region, such as foreign object obstruction or mechanical vibration, and is marked as a dynamic artifact region. Temporal median filtering is then applied to the dynamic artifact regions. Temporal median filtering is an effective method for removing dynamic artifacts; it suppresses abnormal grayscale fluctuations by calculating the median of the image data over time, resulting in a filtered time-series image pair sequence. The time-series image pairs in the filtered sequence are then weighted and summed according to the reciprocal of their respective signal-to-noise ratios (SNRs). The SNR reflects the ratio of image signal to noise; weighted summation according to the reciprocal of the SNR highlights image information with stronger signals and lower noise, thereby generating high-energy spectral fusion images and low-energy spectral fusion images.

[0074] Based on the difference in effective atomic numbers between the solder region and the substrate material in the high-energy spectral fusion image and the low-energy spectral fusion image, the weighting coefficients of the energy spectrum decomposition algorithm are dynamically adjusted. Different materials have different effective atomic numbers and exhibit different attenuation characteristics under X-ray irradiation at different energies. By dynamically adjusting the weighting coefficients, the characteristics of different materials can be reflected more accurately. Based on the adjusted weighting coefficients, the high-energy spectral fusion image and the low-energy spectral fusion image are weighted and fused. The resulting material-sensitive fusion image can more clearly show the distribution and structure of different materials inside the solder joints of the bottom ball grid array package.

[0075] Motion artifacts were eliminated by temporal filtering, and the problem of poor decomposition effect under different material combinations was solved by adaptive weighting, which significantly improved the contrast of small holes.

[0076] Step S103: When the solder joint type is chip-level package solder joint, control the multi-energy spectrum X-ray source to switch to single-energy spectrum mode, and perform online penetration imaging on the chip-level package solder joint that has completed reflow soldering to acquire single-energy spectrum X-ray projection images.

[0077] It should be noted that in chip-level packaging scenarios, due to the small size and susceptibility of solder joints to thermal stress, single-energy spectral imaging is insufficient to reflect the internal stress state. Switching a multi-energy X-ray source to single-energy spectral mode, such as 100kV, can reduce X-ray damage to the chip, while the acquired single-energy X-ray projection images can still meet the requirements for detecting the basic morphology and structure of chip-level package solder joints. Single-energy X-rays have relatively uniform energy; when penetrating chip-level package solder joints, their interaction with the solder joint material and surrounding medium mainly manifests as the photoelectric effect and Compton scattering. Single-energy X-ray projection images can clearly show the outline and location of the solder joints, as well as whether there are obvious defects such as fractures or displacements. When acquiring single-energy X-ray projection images, it is necessary to precisely control the emission parameters of the X-ray source, such as tube current and tube voltage, to ensure image quality and stability.

[0078] Step S104: Perform multimodal spatiotemporal synchronous sensing and physical field reconstruction based on the single-energy spectrum X-ray projection image to generate a physical enhancement feature map.

[0079] It should be noted that multimodal spatiotemporal synchronous sensing and physical field reconstruction combine information from various physical signals to provide a more comprehensive understanding of the physical state of chip-level packaged solder joints. In addition to single-energy X-ray projection images, infrared thermography and acoustic emission signals are introduced. Infrared thermography can reflect the temperature distribution on the solder joint surface; abnormal temperature changes are often related to defects or stress concentrations within the solder joint. Acoustic emission signals can capture the elastic waves generated by the solder joint under stress or undergoing minor deformation. By analyzing the characteristics of the acoustic emission signals, it is possible to infer whether potential problems such as crack propagation exist within the solder joint.

[0080] By synchronizing single-energy X-ray projection images with infrared thermography and acoustic emission signals in time and space, a comprehensive analysis of the weld joint's state is ensured across the same temporal and spatial dimensions. A physical field reconstruction algorithm is then used to fuse information from these multimodal signals, generating a physically enhanced feature map.

[0081] In one feasible implementation, step S104 may include: deploying a thermal infrared imaging module between the reflow oven outlet and the X-ray inspection station to acquire the cooling curve of the chip-level package solder joint from peak temperature to ambient temperature, generating a transient thermal distribution sequence; embedding an acoustic emission sensor array between the mounting station and the reflow oven to acquire phase change shrinkage and gas escape signals during the solder paste curing process in real time, generating a stress wave signal sequence; determining the spatial offset compensation amount of the thermal infrared station and acoustic emission station relative to the X-ray station based on the constant speed of the conveyor belt; performing cross-correlation function registration on the transient thermal distribution sequence and the single-energy X-ray projection image based on the spatial offset compensation amount, and performing time translation and event alignment on the stress wave signal sequence to generate a registered transient thermal distribution sequence and a registered stress wave signal sequence; constructing a physical information neural network, wherein the physical information... The physical information neural network includes a thermal conduction branch and a mechanical branch. The thermal conduction branch adopts a three-dimensional convolutional encoder-decoder structure, and the mechanical branch adopts a graph neural network structure. The registered transient thermal distribution sequence is input into the thermal conduction branch of the physical information neural network to obtain a three-dimensional temperature field. The registered stress wave signal sequence is input into the mechanical branch of the physical information neural network to obtain node stress features. Based on the three-dimensional temperature field and the node stress features, the three-dimensional residual stress field and micropore distribution field inside the chip-level package solder joint are reconstructed. The three-dimensional residual stress field is compressed into a two-dimensional residual stress map through maximum intensity projection, and the micropore distribution field is compressed into a two-dimensional pore density map through volume integration. The two-dimensional residual stress map, the two-dimensional pore density map, and the single-energy spectral X-ray projection image are stitched together along the channel dimension to generate a physical enhancement feature map.

[0082] It should be noted that the purpose of deploying a thermal infrared imaging module between the reflow oven exit and the X-ray inspection station is to capture the temperature changes of the chip-level packaged solder joints as they cool from peak temperature to ambient temperature. As the solder joint temperature gradually decreases, its internal thermal stress also changes accordingly. By acquiring the cooling curves and generating a transient heat distribution sequence, the thermal state evolution of the solder joints during the cooling process can be intuitively reflected.

[0083] An acoustic emission sensor array is embedded between the placement station and the reflow oven to monitor the physical changes during the solder paste curing process in real time. During curing, the solder paste undergoes a phase transition and shrinkage, while gases escape, generating stress wave signals. The acoustic emission sensor can sensitively capture these signals, generating a stress wave signal sequence. These stress wave signals contain information about the internal structural changes of the solder joint. By analyzing them, it is possible to infer whether there are potential defects in the solder joint during curing, such as the initiation or propagation of cracks.

[0084] Each optical module is tagged with an RFID tag, and readers are deployed at the entrances of the thermal infrared, acoustic emission, and X-ray stations to record the precise timestamps of data acquisition for each mode. The spatial offset compensation amount between the thermal infrared and acoustic emission stations and the X-ray station is determined based on the constant conveyor belt speed to ensure accurate spatiotemporal synchronization of multimodal signals. Because there is a certain spatial distance between different stations, the time and location of signal acquisition will differ. By determining the spatial offset compensation amount, the transient thermal distribution sequence and stress wave signal sequence can be adjusted to match the single-energy spectral X-ray projection image in the same spatiotemporal dimension.

[0085] Cross-correlation registration of the transient thermal distribution sequence and single-energy X-ray projection image, as well as time shifting and event alignment of the stress wave signal sequence, are performed to eliminate the spatiotemporal differences between different signals, enabling them to be analyzed in the same coordinate system. Cross-correlation registration involves determining the cross-correlation function between two signals to find their optimal matching position. Time shifting and event alignment align the events of different signals on the time axis based on their temporal characteristics, ensuring they reflect the state of the weld joint at the same point in time. The registered transient thermal distribution sequence and stress wave signal sequence can more accurately reflect the physical state of the weld joint. The physical information neural network adopts a dual-branch coupled architecture, comprising a heat conduction branch and a mechanical branch, achieving field reconstruction under physical consistency constraints through joint training. The heat conduction branch employs a 3D convolutional encoder-decoder structure. The encoder contains four downsampling layers, each using 3×3×3 convolutions, instance normalization, and LeakyReLU activation; the decoder contains four upsampling layers, using transposed convolutions and skip connections. The mechanical branch uses a graph neural network, discretizing the weld joint region into a hexahedral mesh to construct a graph structure. Node features include temperature T and temperature gradient. T represents the material label. Edges connect to 26 neighboring nodes, with edge weights determined by temperature difference and spatial distance. The graph neural network updates node features through a message passing mechanism.

[0086] The physical information neural network also requires training before it can be used. A large number of chip-level packaged solder joint samples are collected. For each sample, single-energy spectral X-ray projection images, infrared thermographic signals, and acoustic emission signals are simultaneously acquired, and the corresponding real three-dimensional residual stress field and micropore distribution field are obtained as training label data. Then, the registered transient thermal distribution sequence, the registered stress wave signal sequence, and the single-energy spectral X-ray projection image are input into the physical information neural network for forward propagation. In the heat conduction branch, a 3D convolutional encoder extracts and downsamples the input transient thermal distribution sequence to capture the spatial and temporal features of the temperature field; then, a decoder upsamples the sequence to reconstruct the three-dimensional temperature field. In the mechanics branch, the graph neural network uses the input stress wave signal sequence and the graph structure of the solder joint area... The process involves updating node features to obtain node stress features. Then, the obtained 3D temperature field and node stress features are used to reconstruct the 3D residual stress field and micropore distribution field inside the chip-scale package solder joints. During training, an appropriate loss function is used to measure the difference between the predicted results and the actual labeled data. The parameters of the physical information neural network are continuously adjusted through backpropagation, making the predicted results gradually approach the true values. After multiple iterations of training, the physical information neural network can learn the complex mapping relationship between multimodal signals and the internal physical state of the solder joints, thereby improving the accurate perception and analysis capability of the physical state of chip-scale package solder joints. In the training process of the physical information neural network, the Adam optimizer is also used. It converges quickly in the initial stages, and in the later stages, such as the last 20% of iterations, it switches to the L-BFGS optimizer to improve accuracy. Adam stage: The initial learning rate is lr = 1 × 10⁻⁶. 3 The ReduceLROnPlateau strategy is adopted, and the learning rate is reduced to 0.5 times the original value when the validation loss does not decrease after 10 epochs. L-BFGS stage: learning rate is fixed at lr = 1 × 10⁻⁶. 5The training process uses residual point sampling; every 100 training steps, Nf=5000 collocation points are randomly sampled again in the spatiotemporal domain to calculate the residual loss of the physical equations, preventing the network from overfitting to a fixed collocation point distribution. Due to the need to calculate higher-order derivatives, which consumes a large amount of GPU memory, the batch size is set to 16, and all collocation points participate in the calculation. The total training time is 50,000 steps, approximately 200 epochs. All hidden layers use the Swish activation function, which offers better smoothness than ReLU and facilitates automatic differentiation calculation of higher-order derivatives.

[0087] After the physical information neural network is trained, the registered transient heat distribution sequence is input into the heat conduction branch to solve the transient heat conduction equation and reconstruct the three-dimensional temperature field. The registered stress wave signal sequence is input into the mechanics branch. The mechanics branch will quickly update the node features through a message passing mechanism based on the graph structure of the weld joint area and the input stress wave signal sequence, and accurately obtain the node stress features.

[0088] By utilizing the obtained three-dimensional temperature field and nodal stress characteristics, the three-dimensional residual stress field and micropore distribution field inside the chip-scale package solder joint can be rapidly and accurately reconstructed. The three-dimensional residual stress field describes the residual stress state at each point in the three-dimensional space inside the chip-scale package solder joint, and is the self-balancing stress distribution remaining inside the solder joint after the thermo-mechanical coupling process. The micropore distribution field describes the volume fraction distribution of micropores, such as microcrack precursors, in the three-dimensional space inside the solder joint, reflecting the degree of material damage and reliability risk.

[0089] The three-dimensional residual stress field is compressed into a two-dimensional residual stress map through maximum intensity projection. The two-dimensional representation obtained by compressing the three-dimensional residual stress field along the X-ray projection direction is used for channel stitching with the X-ray image.

[0090] The micropore distribution field is compressed into a two-dimensional pore density map through volume integration. The two-dimensional pore density map is a two-dimensional representation obtained by integrating the three-dimensional micropore distribution field along the X-ray projection direction, reflecting the cumulative damage in the projection direction.

[0091] The two-dimensional residual stress map, the two-dimensional void density map, and the single-energy spectral X-ray projection image are stitched together along the channel dimension to generate a physical enhancement feature map.

[0092] In one feasible implementation, the reconstructing of the three-dimensional residual stress field and micropore distribution field inside the chip-scale package solder joint based on the three-dimensional temperature field and the nodal stress characteristics includes: determining the temperature gradient at each spatial location based on the three-dimensional temperature field; determining the thermal stress tensor field according to the temperature gradient and the thermal expansion coefficient of the solder material; mapping the nodal stress characteristics to the plastic strain increment of each discrete node of the solder joint using a multilayer perceptron and accumulating it along the time axis to obtain the equivalent plastic strain; inputting the thermal stress tensor field and the equivalent plastic strain into a preset viscoplastic constitutive model to obtain a corrected stress field; and monitoring the temperature threshold in the three-dimensional temperature field in real time. When the temperature threshold is lower than the solder material temperature, the corrected stress field is determined accordingly. When the solidus temperature of the material is reached, it is determined that the solder joint has entered the solid state. The three-dimensional temperature field is frozen, and the static equilibrium equation is solved based on the modified stress field to obtain the preliminary residual stress field. Based on the preliminary residual stress field, the hydrostatic stress and equivalent stress at each location are determined. Based on the hydrostatic stress, equivalent stress, and stress-driven hole nucleation model, the micropore nucleation rate per unit volume is determined. According to the micropore nucleation rate per unit volume, the hole growth process is simulated through the atomic diffusion control equation to generate a micropore density distribution field. The micropore density distribution field is used as a continuous damage variable to perform stiffness reduction correction on the preliminary residual stress field to obtain the three-dimensional residual stress field inside the chip-level package solder joint.

[0093] It should be noted that the temperature gradient describes the rate of temperature change with spatial location at each point in a three-dimensional temperature field and is the fundamental cause of thermal stress. The thermal stress tensor field is a transient stress state generated due to non-uniform temperature distribution and constrained thermal expansion / contraction of materials. The spatial partial derivatives of the three-dimensional temperature field T(x,y,z,t) are determined using the finite difference method to obtain the temperature gradient vector. Based on the temperature gradient vector and the thermal expansion coefficient of the solder material, combined with Hooke's law and thermoelastic constitutive relations, the thermal stress tensor is determined. The thermal expansion coefficient of the solder material is an inherent property of the material, reflecting the degree of expansion or contraction of the material when temperature changes.

[0094] A multilayer perceptron (MLP) is a nonlinear mapping network that decodes nodal stress features into physically interpretable plastic strain increments. Equivalent plastic strain, a scalar index describing the degree of cumulative plastic deformation, is a key state variable in viscoplastic constitutive models. A trained graph neural network is used to extract nodal stress features, which are then mapped to plastic strain increment tensors via a three-layer perceptron. These increments are accumulated along the time axis to obtain the equivalent plastic strain.

[0095] The viscoplastic constitutive model is a mathematical model describing the plastic behavior of materials under high temperature conditions, including creep and stress relaxation. The modified stress field considers the actual stress state after plastic deformation and stress relaxation, and is distinct from purely elastic thermal stress. Using the Perzyna viscoplastic model combined with the Garofalo creep equation, the stress update is iteratively solved using a radial backtracking algorithm to obtain the modified stress field.

[0096] The solidus temperature is the critical temperature at which the solder alloy completely solidifies; below this temperature, the material behaves as a purely elastic solid. Temperature threshold monitoring is used to determine in real time whether the solder joint has entered the solid state. Freezing the temperature field is used to stop the heat conduction solution and fix the temperature distribution. The static equilibrium equation is used to solve for the self-equilibrium residual stress distribution. Monitoring the minimum value of the three-dimensional temperature field triggers a locking mechanism when it falls below the solidus temperature, allowing the solution of the linear elastic static equation to obtain the preliminary residual stress field.

[0097] Hydrostatic stress, the spherical component of the stress tensor, determines volumetric deformation and influences the driving force of void nucleation. The equivalent stress, von Mises equivalent stress, determines shape change and is the primary driving force for plastic flow and void growth. The void nucleation rate is the number of new voids formed per unit volume per unit time. Based on the preliminary residual stress field, hydrostatic stress and equivalent stress are determined, and then substituted into the stress-driven void nucleation model to obtain the nucleation rate. The stress-driven void nucleation model is based on the theories of materials mechanics and thermodynamics; it describes the relationship between the nucleation process of micropores within a material and the stress state under stress. This model considers the influence of hydrostatic equivalent stress on void nucleation, and by analyzing these stress parameters, the nucleation rate of micropores within the material can be predicted.

[0098] The atomic diffusion governing equations describe the physical process of pore growth through atomic diffusion. Pore growth refers to the process by which a nucleated pore absorbs vacancies and increases in volume under high-temperature stress. A continuum model of pore evolution is established, considering diffusion, stress-driven processes, and aggregation effects, and the micropore density distribution field is numerically solved. The micropore density distribution field is a physical quantity describing the distribution of micropores within a solder joint in three-dimensional space, reflecting the degree of damage to the material inside the solder joint and the reliability risk.

[0099] Treating the micropore density distribution field as a continuous damage variable to correct the stiffness of the initial residual stress field is based on the consideration that the presence of micropores reduces the stiffness and load-bearing capacity of the material. By introducing a damage variable, the stress state and deformation behavior inside the weld joint can be described more accurately.

[0100] The progressive damage variable is an intrinsic variable describing the degree of material degradation, with 0 representing intact and 1 representing complete failure. Stiffness reduction refers to the decrease in the effective load-bearing area and elastic modulus caused by voids. The final three-dimensional residual stress field considers the actual residual stress state after damage. By using the Lemaitre progressive damage mechanics model, mapping the void volume fraction to a damage variable, and reducing and correcting the initial residual stress field, the final three-dimensional residual stress field can be obtained.

[0101] Step S105: Use the material-sensitive fusion image and / or the physical enhancement feature map as the solder joint physical information enhancement image.

[0102] It should be noted that the material-sensitive fusion image is obtained by fusing images of different modalities. It integrates information from multiple images and can more comprehensively reflect the physical characteristics of the solder joint. The physical enhancement feature map, on the other hand, is composed of a two-dimensional residual stress map, a two-dimensional void density map, and a single-energy X-ray projection image. It contains important information such as the residual stress and microvoid distribution within the solder joint. Using the material-sensitive fusion image and / or the physical enhancement feature map as enhanced images of the solder joint's physical information can provide richer and more accurate data for the assessment and control of solder joint quality.

[0103] Depending on the specific needs and scenarios, you can choose to use material-sensitive fusion images, physical enhancement feature maps, or a combination of both. If the solder joint type is a bottom ball grid array (BGA) package solder joint, the physical information enhancement image is a material-sensitive fusion image. Due to the characteristics of the BGA package, the material-sensitive fusion image can better capture information such as the interaction between the solder joint and the surrounding materials, as well as changes in material properties, thus more accurately assessing the quality of this type of solder joint. If the solder joint type is a chip-scale package solder joint, the physical information enhancement image is a physical enhancement feature map. Chip-scale package solder joints are small in size and have more intricate and complex internal structures. The stitched information from the two-dimensional residual stress map, two-dimensional void density map, and single-energy X-ray projection image contained in the physical enhancement feature map can clearly present the residual stress distribution and micropore situation inside the chip-scale package solder joint, which is of great significance for assessing the internal microstructure and potential defects of this type of solder joint. By analyzing the physical enhancement feature map, you can more accurately determine whether there are problems such as residual stress concentration and excessive micropores in the chip-scale package solder joint, thus providing a strong basis for improving solder joint quality.

[0104] In this embodiment, by utilizing a multi-energy X-ray source to adaptively switch imaging modes, employing dual-energy spectral imaging and energy spectral decomposition algorithms, and combining infrared thermography and acoustic emission signals to reconstruct the physical field of the solder joint, a physically enhanced image is generated. This effectively distinguishes between metallic compounds and void defects in the solder joint, thereby improving detection accuracy.

[0105] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the optical module solder joint quality control method based on multi-energy spectral X-rays and causal reasoning. Any simple modifications based on this technical concept are within the protection scope of this application.

[0106] This application also provides a quality control device for optical module solder joints based on multi-energy spectral X-rays and causal reasoning. Please refer to [link / reference]. Figure 2 The optical module solder joint quality control device based on multi-energy spectral X-rays and causal reasoning includes: The preprocessing module 10 is used to deploy edge computing nodes at key process equipment in the printed circuit board assembly line, build a device communication topology mapping table, collect the operating data of key process equipment in real time, and perform preprocessing to obtain the original state vector.

[0107] The prediction module 20 is used to construct a spatiotemporal predictive digital twin based on the device communication topology mapping table and the original state vector, and to obtain an enhanced state vector by predicting production line data through the spatiotemporal predictive digital twin.

[0108] The calibration module 30 is used to send a detection command to the physical entity of the key process equipment based on the enhanced state vector and observe the response difference, and perform physical virtual bidirectional calibration to obtain the calibrated state vector.

[0109] The aggregation module 40 is used to train a local scheduling strategy model based on the calibrated state vector stored locally on each edge computing node, and upload the model gradient parameters and contribution proof to the central server based on contract theory, so that the central server can perform weighted federated aggregation based on contract weights and reputation values ​​to obtain a global scheduling strategy model and distribute it to each edge node.

[0110] The scheduling module 50 is used to construct a multi-time-scale hierarchical scheduling controller based on the calibrated state vector and the global scheduling strategy model, solve the drift plus penalty plus risk optimization problem, generate collaborative scheduling instructions across the strategic layer, tactical layer and execution layer, and issue the collaborative scheduling instructions to the production line execution unit for collaborative scheduling.

[0111] The optical module solder joint quality control device based on multi-energy X-ray and causal reasoning provided in this application adopts the optical module solder joint quality control method based on multi-energy X-ray and causal reasoning in the above embodiments, which can solve the technical problems of poor defect identification rate and quality control efficiency. Compared with the prior art, the beneficial effects of the optical module solder joint quality control device based on multi-energy X-ray and causal reasoning provided in this application are the same as the beneficial effects of the optical module solder joint quality control method based on multi-energy X-ray and causal reasoning provided in the above embodiments, and other technical features in the optical module solder joint quality control device based on multi-energy X-ray and causal reasoning are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0112] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the optical module solder joint quality control method based on multi-energy spectral X-rays and causal inference as described above.

[0113] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for quality control of optical module solder joints based on multi-energy spectral X-rays and causal reasoning, characterized in that, The method includes: Online penetration imaging of the solder joints of an optical module that has undergone reflow soldering is performed using a multi-energy spectrum X-ray source to acquire the original X-ray projection image, and an enhanced image of the physical information of the solder joint is generated based on the original X-ray projection image. The enhanced image of the physical information of the solder joint is input into the large model of the vision transformer to perform deep semantic reasoning of the internal structure of the solder joint, and the solder joint defect segmentation map and defect confidence heat map are obtained. Based on the defect segmentation map, the connected components of each defect are extracted, the centroid of each connected component is determined in the image coordinate system, and coordinate transformation is performed by combining the camera calibration parameters and the conveyor encoder signal to generate a defect space coordinate set with the optical module reference point as the origin. Based on the defect segmentation map and the defect confidence heatmap, a cause-and-effect map of solder joint defects in the current production batch is constructed to identify the process root causes that lead to the defects. The frequency of occurrence of the same cause-and-effect path in multiple consecutive optical modules is counted, and process intervention instructions carrying root cause labels are generated. The process intervention command, the defect spatial coordinate set, and the root cause label are transmitted to the surface mount digital twin controller, so that the digital twin controller executes a hierarchical response strategy based on the root cause label to achieve optical module solder joint quality control.

2. The method as described in claim 1, characterized in that, The process of using a multi-energy spectral X-ray source to perform online penetration imaging on the solder joints of the optical module that have undergone reflow soldering, acquiring original X-ray projection images, and generating enhanced images of the solder joint physical information based on the original X-ray projection images includes: When the solder joint type is a bottom ball grid array package solder joint, the multi-energy spectrum X-ray source is controlled to switch to dual-energy spectrum mode, and online penetration imaging is performed on the bottom ball grid array package solder joint that has completed reflow soldering to acquire high-energy spectrum X-ray projection images and low-energy spectrum X-ray projection images. A material-sensitive fusion image is generated based on the high-energy-spectrum X-ray projection image and the low-energy-spectrum X-ray projection image using an energy spectrum decomposition algorithm; When the solder joint type is chip-level packaged solder joint, the multi-energy spectrum X-ray source is controlled to switch to single-energy spectrum mode, and online penetration imaging is performed on the chip-level packaged solder joint that has completed reflow soldering to acquire single-energy spectrum X-ray projection images. Multimodal spatiotemporal synchronous sensing and physical field reconstruction are performed based on the single-energy spectrum X-ray projection image to generate a physical enhancement feature map; The material-sensitive fusion image and / or the physical enhancement feature map are used as the physical information enhancement image of the solder joint.

3. The method as described in claim 2, characterized in that, The process of generating a material-sensitive fused image based on the high-energy-spectrum X-ray projection image and the low-energy-spectrum X-ray projection image using an energy spectrum decomposition algorithm includes: A time-series image pair sequence is constructed based on multiple consecutive frames of the high-energy spectral X-ray projection image and the low-energy spectral X-ray projection image; The theoretical displacement between adjacent frames in the time sequence of image pairs is determined based on the conveyor belt encoder signal, and a subpixel-level affine transformation is performed on the subsequent frame images in the time sequence of image pairs according to the theoretical displacement to obtain the registered time sequence of image pairs. Determine the grayscale standard deviation of the same solder joint region in the registered temporal image pair sequence. If the grayscale standard deviation exceeds a preset noise threshold, the corresponding solder joint region is marked as a dynamic artifact region. Temporal median filtering is applied to the dynamic artifact region to obtain a filtered temporal image pair sequence. The filtered time-series image pairs in the sequence are weighted and summed according to the inverse of their corresponding signal-to-noise ratios to generate a high-energy spectral fusion image and a low-energy spectral fusion image. Based on the difference in effective atomic number between the solder region and the substrate material in the high-energy spectrum fusion image and the low-energy spectrum fusion image, the weight coefficients of the energy spectrum decomposition algorithm are dynamically adjusted to obtain the adjusted weight coefficients. The high-energy spectrum fused image and the low-energy spectrum fused image are weighted and fused based on the adjusted weight coefficients to generate a material-sensitive fused image.

4. The method as described in claim 2, characterized in that, The process of performing multimodal spatiotemporal synchronous sensing and physical field reconstruction based on the single-energy spectral X-ray projection image to generate a physically enhanced feature map includes: A thermal infrared imaging module is deployed between the reflow oven outlet and the X-ray inspection station to collect the cooling curve of the chip-level package solder joint from the peak temperature to the ambient temperature, and generate a transient heat distribution sequence. An acoustic emission sensor array is embedded between the mounting station and the reflow oven to collect phase change shrinkage and gas escape signals during the solder paste curing process in real time, and generate stress wave signal sequences. The spatial offset compensation amount of the thermal infrared station and acoustic emission station relative to the X-ray station is determined based on the constant speed of the conveyor belt. Based on the spatial offset compensation, the transient thermal distribution sequence and the single-energy X-ray projection image are cross-correlation function registered, and the stress wave signal sequence is time-shifted and event-aligned to generate the registered transient thermal distribution sequence and the registered stress wave signal sequence. A physical information neural network is constructed, wherein the physical information neural network includes a heat conduction branch and a mechanics branch, the heat conduction branch adopts a three-dimensional convolutional encoder-decoder structure, and the mechanics branch adopts a graph neural network structure; The registered transient heat distribution sequence is input into the heat conduction branch of the physical information neural network to obtain a three-dimensional temperature field. The registered stress wave signal sequence is input into the mechanical branch of the physical information neural network to obtain the nodal stress characteristics; Based on the three-dimensional temperature field and the nodal stress characteristics, the three-dimensional residual stress field and micropore distribution field inside the chip-level package solder joint are reconstructed. The three-dimensional residual stress field is compressed into a two-dimensional residual stress map through maximum intensity projection, and the micropore distribution field is compressed into a two-dimensional pore density map through volume integral. The two-dimensional residual stress map, the two-dimensional void density map, and the single-energy spectral X-ray projection image are stitched together along the channel dimension to generate a physical enhancement feature map.

5. The method as described in claim 4, characterized in that, The reconstruction of the three-dimensional residual stress field and micropore distribution field inside the chip-level package solder joint based on the three-dimensional temperature field and the nodal stress characteristics includes: The temperature gradient at each spatial location is determined based on the three-dimensional temperature field, and the thermal stress tensor field is determined based on the temperature gradient and the thermal expansion coefficient of the solder material. The nodal stress characteristics are mapped to the plastic strain increments of each discrete node of the weld joint using a multilayer perceptron and accumulated along the time axis to obtain the equivalent plastic strain. The thermal stress tensor field and the equivalent plastic strain are input together into a preset viscoplastic constitutive model to obtain a modified stress field; The temperature threshold in the three-dimensional temperature field is monitored in real time. When the temperature threshold is lower than the solidus temperature of the solder material, it is determined that the solder joint has entered the solid state, the three-dimensional temperature field is frozen, and the static equilibrium equation is solved based on the modified stress field to obtain the preliminary residual stress field. Based on the preliminary residual stress field, the hydrostatic stress and equivalent stress at each location are determined. Based on the hydrostatic stress, equivalent stress, and stress-driven cavity nucleation model, the micropore nucleation rate per unit volume is determined. Based on the micropore nucleation rate per unit volume, the pore growth process is simulated using atomic diffusion control equations to generate a micropore density distribution field. Using the micropore density distribution field as a continuous damage variable, the stiffness reduction correction is applied to the preliminary residual stress field to obtain the three-dimensional residual stress field inside the chip-level package solder joint.

6. The method as described in claim 1, characterized in that, The step of inputting the enhanced image of the solder joint physical information into the large model of the vision transformer to perform deep semantic reasoning of the internal structure of the solder joint, and obtaining a solder joint defect segmentation map and a defect confidence heatmap, includes: The enhanced image of the physical information of the solder joint is divided into a sequence of image blocks of fixed size. Linear projection and position encoding are performed on each image block to generate an image block embedding sequence. The image patch embedding sequence is input into the encoder stack structure of the large visual transformer model, wherein the large visual transformer model includes an encoder stack structure and a decoder, the first encoder layer at the front end of the encoder stack structure is composed of a multi-head self-attention mechanism and a feedforward neural network, and at least one second encoder layer at the end of the encoder stack structure inserts a physical attention bias module on the basis of the multi-head self-attention mechanism and the feedforward neural network. The first encoder layer extracts local texture features and edge contour features from the image patch embedding sequence to obtain a shallow feature sequence. The shallow feature sequence is input into the second encoder layer. The physical attention bias module is activated in the second encoder layer. The attention weight bias matrix is ​​determined according to the material sensitivity information or residual stress field information. The attention weight bias matrix is ​​then superimposed on the attention weight matrix of the multi-head self-attention mechanism to obtain the superimposed attention weight. The shallow feature sequence is obtained by weighting the superimposed attention weights; The deep feature sequence is input into the decoder of the large visual transformer model. The decoder upsamples the deep feature sequence to obtain a pixel-level classification probability map. The pixel-level classification probability map is post-processed using a conditional random field to generate a solder joint defect segmentation map. Based on the pixel-level classification probability map, the probability of each pixel belonging to each type of defect is determined, and a defect confidence heatmap is generated.

7. The method as described in claim 1, characterized in that, The process involves extracting the connected components of each defect based on the defect segmentation map, determining the pixel coordinates of the centroid of each connected component in the image coordinate system, and performing coordinate transformation by combining camera calibration parameters and conveyor encoder signals to generate a defect spatial coordinate set with the optical module reference point as the origin. This includes: The weld joint defect segmentation map is binarized, and the connected components of each defect are extracted using the eight-neighbor connected component labeling algorithm. The pixel coordinates of the centroid of each connected component in the image coordinate system are determined based on the zeroth and first moments of each connected component. Obtain camera calibration parameters, and perform distortion correction and normalization processing on the pixel coordinates based on the camera calibration parameters to convert them into three-dimensional coordinates in the camera coordinate system; The physical displacement of the current optical module relative to the imaging station is determined based on the conveyor belt encoder signal. Based on the physical displacement and the camera extrinsic calibration results, the three-dimensional coordinates in the camera coordinate system are converted into coordinates in the world coordinate system. Identify the reference marker point of the optical module, establish a local coordinate system with the reference point of the optical module as the origin, transform the coordinates in the world coordinate system to the local coordinate system, and generate a defect space coordinate set.

8. The method as described in claim 1, characterized in that, Based on the defect segmentation map and the defect confidence heatmap, a cause-and-effect graph of solder joint defects for the current production batch is constructed to identify the root causes of the defects. The frequency of similar cause-and-effect paths occurring in multiple consecutive optical modules is counted, and process intervention instructions carrying root cause tags are generated, including: Define a three-level cause-effect graph structure containing process parameter nodes, physical state nodes, and defect type nodes, initialize the node set, and set inter-level unidirectional cause-effect constraints; Collect time-series observation data of each node in historical production batches, and determine the conditional independence score of any two nodes under the given subset of other nodes; Based on the conditional independence score, a constrained causal discovery algorithm is used to remove statistically independent node connections, and the direction of the remaining edges is optimized through a scoring search strategy to generate an initial causal adjacency matrix that represents the causal connection relationship between nodes. The matrix element values ​​of the initial causal adjacency matrix represent the existence of causal edges. Traverse each pair of nodes marked as having directed edges in the initial causal adjacency matrix and construct a structural equation model. The structural equation model can predict the values ​​of child nodes based on the values ​​of parent nodes and quantify the uncertainty of prediction. The production process parameters of the current production batch are used as the observation values ​​of the process parameter node, the physical information of the solder joint is used as the observation values ​​of the physical state node, and the solder joint defect segmentation map and the defect confidence heat map are mapped to the observation values ​​of the defect type node to obtain the observation data of the current batch. Based on the initial causal adjacency matrix and the current batch observation data, a causal graph of solder joint defects for the current production batch is constructed: The cause-effect graph of the solder joint defect is input into the structural equation model to perform counterfactual causal intervention calculations to obtain the change in the probability of defect occurrence. The dominant causal path is identified based on the change in the probability of the defect occurring, and the starting process parameters of the dominant causal path are marked as the process root cause of the defect, and a root cause label is generated. The frequency of occurrence of the same root cause tag in a continuously preset number of optical modules is counted. If the frequency of occurrence exceeds the dynamic drift threshold, a process intervention instruction carrying the root cause tag is generated.

9. A quality control device for optical module solder joints based on multi-energy spectral X-rays and causal reasoning, characterized in that, The device includes: The generation module is used to perform online penetration imaging on the solder joints of the optical module that have completed reflow soldering using a multi-energy spectrum X-ray source, acquire the original X-ray projection image, and generate an enhanced image of the physical information of the solder joint based on the original X-ray projection image; The reasoning module is used to input the enhanced image of the physical information of the solder joint into the large model of the vision converter to perform deep semantic reasoning of the internal structure of the solder joint, and obtain the solder joint defect segmentation map and the defect confidence heat map. The conversion module is used to extract the connected components of each defect based on the defect segmentation map, determine the pixel coordinates of the centroid of each connected component in the image coordinate system, and perform coordinate transformation by combining the camera calibration parameters and the conveyor encoder signal to generate a defect space coordinate set with the optical module reference point as the origin. The construction module is used to construct a cause-effect graph of solder joint defects in the current production batch based on the defect segmentation graph and the defect confidence heatmap, identify the process root causes that lead to the defects, count the frequency of occurrence of the same cause-effect path in multiple consecutive optical modules, and generate process intervention instructions carrying root cause labels. The transmission module is used to transmit the process intervention command, the defect spatial coordinate set, and the root cause label to the surface mount digital twin controller, so that the digital twin controller can execute a hierarchical response strategy according to the root cause label to achieve optical module solder joint quality control.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the optical module solder joint quality control method based on multi-energy spectral X-rays and causal reasoning as described in any one of claims 1 to 8.

Images