PCB solder paste printing three-dimensional defect detection system based on multispectral imaging

Abstract

The application discloses a PCB solder paste printing three-dimensional defect detection system based on multispectral imaging, and relates to the field of machine vision detection.The technical scheme points of the application include a multispectral image acquisition module, a four-channel industrial camera based on RGB and near-infrared, and a tunable LED light source array; the reflection characteristic data under different penetration depths are acquired through a wavelength switching mechanism; a motion control trigger module is based on a servo motor driven XYZ three-axis object table, and positioning is realized in cooperation with an encoder feedback; a pulse width modulation signal is adopted to coordinate the camera shutter and the platform moving speed; the application realizes the dual verification of material composition quantitative analysis and three-dimensional topography reconstruction through the combination of the four-channel imaging of RGB and NIR and the Beer-Lambert law modeling, and avoids the limitations of monocular vision; the cross-frame data alignment is realized through the projection of a structured light and a binocular stereo matching technology in cooperation with an ICP algorithm, and the details defects can be better detected.

Description

Technical Field

[0001] This invention relates to the field of machine vision inspection technology, and more specifically, to a 3D defect detection system for PCB solder paste printing based on multispectral imaging. Background Technology

[0002] In the electronics manufacturing industry, the solder paste printing quality of printed circuit boards (PCBs) directly affects the reliability of subsequent soldering processes and the overall performance of electronic products. Traditional inspection methods mainly rely on two-dimensional vision systems, which analyze planar images acquired using a single wavelength light source. This approach has the following problems:

[0003] It is impossible to accurately quantify three-dimensional morphological parameters (such as height deviation and volume error), making it difficult to detect hidden defects (such as microcracks and voids).

[0004] Monochromatic light has difficulty distinguishing different material components in multilayer structures and is poorly adaptable to special alloys or composite materials.

[0005] Therefore, there is an urgent need for a 3D defect detection system for PCB solder paste printing based on multispectral imaging to overcome the above problems. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a three-dimensional defect detection system for PCB solder paste printing based on multispectral imaging.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A 3D defect detection system for PCB solder paste printing based on multispectral imaging includes:

[0009] The multispectral image acquisition module is based on a four-channel industrial camera with RGB and near-infrared capabilities and a tunable LED light source array; it acquires reflection characteristic data at different penetration depths through a wavelength switching mechanism.

[0010] The motion control trigger module is based on a servo motor-driven XYZ three-axis stage, which uses encoder feedback to achieve positioning; it uses pulse width modulation signals to coordinate the camera shutter speed and the platform movement speed.

[0011] The 3D point cloud generation module, based on structured light projection, projects laser stripes of known patterns onto the target area, calculates depth information using a binocular stereo vision algorithm, aligns multi-view data through an iterative nearest point registration algorithm, and constructs a dense point cloud model.

[0012] The defect feature analysis module, based on an adaptive threshold segmentation algorithm, defines a comprehensive scoring function to perform defect feature analysis.

[0013] The machine learning classification module constructs a convolutional neural network model, with normalized multi-channel feature maps as input. The network parameters are optimized through backpropagation to improve the accuracy of defect identification.

[0014] The human-computer interaction interface module displays real-time detection results and provides human-computer interaction functions.

[0015] Preferably, the defect detection method of this system is as follows:

[0016] Initialization phase: Load the process file to set the allowable error range, calibrate the equipment coordinate system and mechanical origin; perform white balance correction after preheating the light source to a stable state;

[0017] Data acquisition loop, for each testing station:

[0018] Step 1: Move the platform to the target location according to the CAD path plan;

[0019] Step 2: Turn on the light source of each channel in sequence and capture the corresponding frame image;

[0020] Step 3: Trigger structured light projection and record the projection deformation pattern;

[0021] Step 4: After completing the multimodal data collection for the current area, proceed to the next station;

[0022] Real-time processing threads execute the following tasks in parallel:

[0023] Thread A: Performs background subtraction and coarse localization of the region of interest;

[0024] Thread B: Runs a 3D reconstruction algorithm to generate a point cloud model;

[0025] Thread C: Invokes the pre-trained model to perform preliminary defect screening;

[0026] The main thread integrates the results and triggers an exception interruption mechanism.

[0027] Preferably, the defect detection system further includes:

[0028] The multi-scale feature enhancement module downsamples the original image to three scales (1 / 2, 1 / 4, and 1 / 8) using bilinear interpolation, forming a multi-layered representation of a pyramid structure. Each layer extracts local features through independent convolutional blocks, with shallow layers preserving edge details and deep layers capturing large-scale topological anomalies. A combination of upsampling paths and skip connections is used to inject high-level semantic information into the low-resolution feature map.

[0029] The displacement compensation calibration module performs phased corrections for image misalignment caused by mechanical motion errors.

[0030] The simulation verification module establishes a virtual detection environment for algorithm pre-running and parameter optimization.

[0031] The edge computing module breaks down computationally intensive operations into a cluster of parallelizable subtasks; it leverages the bandwidth advantage of DDR4 memory to build a multi-level buffer pool; and it preloads hot data blocks to on-chip memory based on an access pattern prediction algorithm.

[0032] The self-supervised domain adaptation training module automatically collects normal samples as a benchmark dataset; it adopts a progressive learning strategy to gradually expand the category boundaries, updating only the parameters of the last two network layers in each iteration; it constructs a game model between the domain discriminator and the feature extractor; and it forces the generator to produce domain-invariant feature representations through a gradient reversal layer, so that the data distributions of the source domain and the target domain are aligned in the latent space.

[0033] Preferably, the displacement compensation calibration module performs staged correction in the following manner:

[0034] The Hough transform based on pre-embedded markers is used to detect the center of the circular contour, and the deviation vector between the actual coordinates and the theoretical position is calculated. A rigid affine transformation model is used to correct the overall offset. Four reference points at the corner of the PCB are selected as the control set, and a nonlinear deformation field is generated using a thin plate spline interpolation algorithm. The rate of change of the feature point trajectory is monitored in real time during continuous scanning, and a prediction compensation mechanism is triggered when the detected acceleration exceeds the threshold. The position deviation of the next frame is estimated using a Kalman filter, and the acquisition timing parameters are adjusted in advance.

[0035] Preferably, the simulation verification module performs algorithm pre-running and parameter tuning, specifically as follows:

[0036] A ray tracing model was built using Zemax software to simulate the scattering distribution characteristics under different incident angles; the light source uniformity index was optimized; the Navier-Stokes equations were solved using COMSOL to reproduce the solder paste flow process and quantify the curing shrinkage rate; the morphology evolution law under different coating speeds was predicted by combining the material viscosity curve database; and steady-state heat conduction analysis was performed using ANSYS Workbench to evaluate the influence of the temperature gradient during the solder reflow stage on the substrate warpage.

[0037] Preferably, the 3D point cloud generation module includes:

[0038] The multimodal data acquisition unit adopts a composite projection mode of Gray code and phase-shifted fringe, and realizes encoding through four sets of structured light patterns in orthogonal directions; each frame of projection contains unique spatiotemporal marker information; equipped with a dual-frequency laser, it dynamically adjusts the fringe density to adapt to surfaces with different curvatures; FPGA-based timestamp management ensures that the camera exposure time is synchronized with the laser pulse; 21 preset standard calibration target points form a three-dimensional mesh framework, and establish a homography mapping matrix from the world coordinate system to the image coordinate system;

[0039] The depth map calculation unit performs a two-dimensional Fourier transform on the captured structured light image to extract the dominant frequency components and adjusts the bandpass filter parameters in real time; it uses a Wiener filter to suppress random noise interference and retain effective periodic signal components; it monitors the changing trend of spectral energy distribution in real time and dynamically adjusts the sampling window size; it uses B-spline curve fitting to estimate gray values ​​at non-integer positions; and it combines bidirectional cubic convolution kernels to perform interpolation operations to generate continuous and smooth height field data.

[0040] The point cloud fusion processing unit constructs a two-level matching architecture from coarse registration to fine registration. In the initial stage, the RANSAC algorithm is used to remove the influence of outliers and quickly lock the transformation relationship. In the fine stage, the LM optimization algorithm is used to minimize the residual error function. Based on the local curvature estimation results, the smoothing intensity coefficient is adaptively adjusted to enhance the denoising effect in flat areas while protecting the integrity of sharp-edged structures.

[0041] Preferably, the defect feature analysis module includes:

[0042] Multi-scale segmentation units are implemented using a local dynamic thresholding algorithm improved from the Otsu method. K-means clustering is used to determine the initial segmentation threshold, and a serpentine algorithm is used to iteratively optimize the boundary positions. Different color space conversion channel weight allocation schemes are designed for different background materials. A hierarchical connected component analysis process is established, first merging adjacent small-area fragments to form a potential object candidate set, and then filtering effective targets based on shape factors and grayscale statistics. A minimum acceptable area threshold is set to filter false positive detection results. The Chan-Vese level set method is used to refine rough edge contours. Boundary localization is achieved by tracking the gradient direction evolution path based on an active contour model.

[0043] The three-dimensional morphological analysis unit projects a two-dimensional binary image along the normal direction to construct a height histogram, integrates it to obtain the volume measurement value, and performs compensation calculations based on the perspective transformation correction factor; it uses the power spectral density function to analyze the height fluctuation characteristics; it decomposes the surface morphology into the contribution ratio of different wavelength components and calculates the roughness parameters; it fits the best reference plane as the datum plane and calculates the distance deviation distribution from each measurement point to this plane; it uses the least squares method to solve the coefficients of the ideal plane equation, statistically analyzes the maximum positive and negative deviation values ​​and their location coordinates, and generates a heat map of the out-of-tolerance area distribution;

[0044] Pareto chart generation unit: summarizes the frequency of occurrence by defect type and displays it in descending order; automatically identifies influencing factors;

[0045] Correlation matrix construction unit: Analyze the interrelationships between various quality characteristics; display the results using a scatter plot matrix.

[0046] Preferably, the machine learning classification module includes:

[0047] The data augmentation unit simulates the deformation that occurs during PCB manufacturing. It generates synthetic samples by applying nonlinear transformations to the original images, expanding the data coverage of rare operating conditions in the training set. It also trains the CycleGAN network to achieve a unified transformation of the appearance of data across suppliers.

[0048] The model architecture unit inserts an SENet module into ResNet to capture the importance difference information between channels; it also caches a negative sample pool formed by misclassified samples within a set time period.

[0049] The inference deployment unit pre-simulates the impact of numerical truncation errors caused by low-precision quantization and adjusts the scaling factor of the batch normalization layer to reduce quantization loss; it adopts mixed-precision training to balance speed and accuracy and finds the most suitable quantization bit width configuration scheme for the target hardware platform; and iteratively removes redundant connection edges based on the weight importance scoring index.

[0050] Preferably, the multi-scale feature enhancement module includes:

[0051] The multi-resolution input preprocessing unit performs dynamic histogram stretching on the original images under different lighting conditions and uses a block-based local normalized Gamma correction algorithm to eliminate the effects of non-uniform illumination. It calculates the mean and standard deviation of brightness for each region using a sliding window and independently adjusts the contrast parameter to enhance details in dark areas. It constructs channel and spatial domains. The spatial domain is based on an improved Squeeze-and-Excitation structure, obtaining a two-dimensional spatial weight map through global average pooling. The channel domain uses a bottleneck-style fully connected layer to compress the feature dimension. The two domains are then weighted and fused to generate an adaptive feature enhancement mask.

[0052] The cross-level feature fusion unit adopts a hybrid strategy of deconvolution and bilinear interpolation to achieve resolution alignment, and dynamically adjusts the contribution ratio of each level through a gating mechanism.

[0053] An adaptive output adaptation layer automatically adjusts the loss weight coefficients of the classification / regression branches based on the downstream task type; the contribution of hard sample loss is reweighted using the focal loss function to improve the recall rate of extreme-sized targets; multiple independent decoders are set up in parallel to process feature maps of different scales; the optimal configuration scheme is determined through self-supervised learning under the supervision of cross-entropy loss; Monte Carlo Dropout technology is integrated to predict the model's prediction variance and output the quality score and confidence interval of each candidate box.

[0054] Preferably, the displacement compensation calibration module includes:

[0055] The motion trajectory acquisition unit integrates a miniature interferometer composed of diffractive optical elements as a position sensor. It employs a differential signal receiving scheme to suppress common-mode interference and uses a lock-in amplifier to extract phase difference information. Based on the angular velocity and accelerometer data provided by the IMU and the visual odometry estimation results, a multi-source fusion motion state observer is constructed. A Kalman filter framework is used to fuse motion information from different frequency bands to compensate for high-frequency noise interference caused by mechanical vibration.

[0056] The dynamic registration optimization unit employs a GPU-parallelized semi-global matching algorithm to optimize disparity map generation efficiency and eliminates mismatched point pairs. The final output is a 3D point cloud dataset as the registration benchmark. The PCB is treated as an anisotropic hyperelastic material, constitutive equations are constructed, and the static equilibrium state under contact mechanical boundary conditions is solved using the finite element method. The local deformation effect caused by fixture pressure is simulated. The solution is obtained until the residual stress is less than a preset threshold, thus obtaining a true characterization of the workpiece deformation state. An optimization objective function is established based on the workpiece deformation state characterization, and the Levenberg-Marquardt algorithm is used to achieve fast convergence.

[0057] The real-time correction execution unit is based on a piezoelectric actuator to form a six-degree-of-freedom fine-tuning platform, and uses a capacitive displacement sensor to form a closed-loop control system to achieve displacement control resolution.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] This invention achieves dual verification of quantitative analysis of material composition and three-dimensional morphology reconstruction by combining RGB and NIR four-channel imaging with Beer-Lambert's law modeling, thus avoiding the limitations of monocular vision. It also employs structured light projection and binocular stereo matching technology, combined with the ICP algorithm, to achieve cross-frame data alignment, enabling better detection of detailed defects.

[0060] This invention uses a marker tracking and Kalman prediction model based on Hough transform to correct mechanical motion errors in real time and ensure coordinate system consistency during continuous scanning.

[0061] This invention constructs a pyramid-shaped feature extraction network, which achieves the organic integration of shallow edge details and deep semantic information through skip connections, and maintains high sensitivity to filament breakage and glue overflow areas.

[0062] This invention utilizes generative adversarial networks to align data distribution between laboratory environments and workshop settings, enabling models to be deployed across different scenarios without the need to re-label datasets.

[0063] This invention maintains the universality of the basic feature extractor by updating the parameters of the last layer of the network, avoiding overfitting and accelerating the convergence process. Attached Figure Description

[0064] Figure 1 The present invention presents a framework diagram of a 3D defect detection system for PCB solder paste printing based on multispectral imaging.

[0065] Figure 2 The flowchart of the PCB solder paste printing three-dimensional defect detection system based on multispectral imaging is presented in this invention. Detailed Implementation

[0066] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0067] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0068] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places throughout this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments.

[0069] Reference Figures 1-2 As shown.

[0070] The embodiments further illustrate the PCB solder paste printing three-dimensional defect detection system based on multispectral imaging proposed in this invention.

[0071] Example 1:

[0072] A 3D defect detection system for PCB solder paste printing based on multispectral imaging includes:

[0073] The multispectral image acquisition module is based on a four-channel industrial camera (RGB and near-infrared (NIR)) and a tunable LED light source array; it acquires reflection characteristic data at different penetration depths through a wavelength switching mechanism.

[0074] Based on Beer-Lambert's law describing the light intensity attenuation, the relationship between absorbance and material concentration is established:

[0075] ;

[0076] in, The absorption coefficient is wavelength-dependent. For substance concentration, This is the path length;

[0077] The motion control trigger module is based on a servo motor-driven XYZ three-axis stage, which uses encoder feedback to achieve positioning; it uses pulse width modulation (PWM) signals to coordinate the camera shutter speed and the platform movement speed.

[0078] The 3D point cloud generation module, based on structured light projection, projects laser stripes of a known pattern onto the target area. It then calculates depth information using a binocular stereo vision algorithm and aligns multi-view data through an iterative nearest point (ICP) registration algorithm to construct a dense point cloud model. Details are as follows:

[0079] ;

[0080] By minimizing the energy function Solving the rigid transformation matrix This enables cross-frame point cloud fusion.

[0081] The defect feature analysis module, based on an adaptive threshold segmentation algorithm, extracts the mean height, standard deviation, tilt angle, and surface area, and defines a comprehensive scoring function.

[0082] ;

[0083] in, This is the absolute value of the height deviation. R represents the proportion of the area exceeding the design range, and is a surface roughness index (such as Ra value). These are the weighting coefficients.

[0084] The machine learning classification module constructs a convolutional neural network model. The input is a normalized multi-channel feature map, and the output layer uses a softmax activation function to predict defect type probabilities. The loss function is cross-entropy.

[0085] ;

[0086] in, This refers to the batch sample size. For real labels, The probability distribution vector predicted by the model is used to optimize the network parameters through backpropagation, thereby improving the accuracy of defect identification.

[0087] The human-computer interaction interface module displays real-time detection results and provides human-computer interaction functions.

[0088] The PCB solder paste printing three-dimensional defect detection system based on multispectral imaging detects defects in the following way:

[0089] Initialization phase: Load the process file to set the allowable error range, calibrate the equipment coordinate system and mechanical origin; perform white balance correction after preheating the light source to a stable state;

[0090] Data acquisition loop, for each testing station:

[0091] Step 1: Move the platform to the target location according to the CAD path plan;

[0092] Step 2: Turn on the light source of each channel in sequence and capture the corresponding frame image;

[0093] Step 3: Trigger structured light projection and record the projection deformation pattern;

[0094] Step 4: After completing the multimodal data collection for the current area, proceed to the next station.

[0095] Real-time processing threads execute the following tasks in parallel:

[0096] Thread A: Performs background subtraction and coarse localization of the region of interest (ROI);

[0097] Thread B: Runs a 3D reconstruction algorithm to generate a point cloud model;

[0098] Thread C: Invokes the pre-trained model to perform preliminary defect screening;

[0099] The main thread integrates the results and triggers an exception interruption mechanism.

[0100] Example 2:

[0101] This embodiment of the PCB solder paste printing three-dimensional defect detection system based on multispectral imaging, building upon Embodiment 1, further includes:

[0102] The multi-scale feature enhancement module downsamples the original image to three scales (1 / 2, 1 / 4, and 1 / 8) using bilinear interpolation, forming a multi-layered pyramid structure. Each layer extracts local features through independent convolutional blocks, with shallow layers preserving edge details (such as filament breaks with a width <5μm) and deep layers capturing large-scale topological anomalies (such as areas of glue overflow with an area >2mm²). A combination of upsampling paths and skip connections is used to inject high-level semantic information into the low-resolution feature map.

[0103] For example, during the second-level fusion, after restoring the spatial dimension using nearest neighbor interpolation, the number of channels is adjusted to a unified dimension through 1×1 convolution, and then element-wise weighted summation is performed with the output of the same-level backbone network; feature weights are assigned to each frequency band; and considering the dense distribution of pad areas, the gain coefficient of the high-frequency response branch is automatically enhanced when high-density connected components are detected.

[0104] The displacement compensation calibration module performs phased corrections for image misalignment caused by mechanical motion errors.

[0105] The Hough transform based on pre-embedded markers is used to detect the center of the circular contour, and the deviation vector between the actual coordinates and the theoretical position is calculated. A rigid affine transformation model is used to correct the overall offset. Four reference points at the corner of the PCB are selected as the control set, and a nonlinear deformation field is generated using a thin plate spline interpolation algorithm. The rate of change of the feature point trajectory is monitored in real time during continuous scanning, and a prediction compensation mechanism is triggered when the detected acceleration exceeds the threshold. The position deviation of the next frame is estimated using a Kalman filter, and the acquisition timing parameters are adjusted in advance.

[0106] The simulation verification module establishes a virtual detection environment for algorithm pre-running and parameter tuning.

[0107] A ray tracing model was built using Zemax software to simulate the scattering distribution characteristics under different incident angles; the light source uniformity index was optimized, and the illuminance fluctuation was controlled within ±3% through non-serialized design; the Navier-Stokes equations were solved using COMSOL to reproduce the solder paste flow process and quantify the curing shrinkage rate; the morphology evolution law under different coating speeds was predicted by combining the material viscosity curve database; and steady-state heat conduction analysis was performed using ANSYS Workbench to evaluate the influence of the temperature gradient during the soldering reflow stage on the substrate warpage.

[0108] The edge computing module breaks down computationally intensive operations into a cluster of parallelizable subtasks; for example, it distributes histogram equalization in image preprocessing to FPGA DSP slices for execution, while complex matrix operations are handled by the GPU CUDA core; it utilizes the bandwidth advantage of DDR4 memory to build a multi-level buffer pool; and it preloads hot data blocks to on-chip memory based on access pattern prediction algorithms.

[0109] The self-supervised domain adaptation training module automatically collects a certain number of normal samples as a benchmark dataset. A progressive learning strategy is employed to gradually expand the class boundaries, updating only the parameters of the last two network layers in each iteration to maintain the generality of the feature extractor. A game model is constructed between the domain discriminator and the feature extractor. A gradient inversion layer forces the generator to produce domain-invariant feature representations, aligning the data distributions of the source domain (laboratory environment) and the target domain (workshop environment) in the latent space.

[0110] The three-dimensional point cloud generation module includes:

[0111] The multimodal data acquisition unit employs a composite projection mode of Gray code and phase-shifting fringe, achieving high-density encoding through four sets of orthogonal structured light patterns. Each projection frame contains unique spatiotemporal marker information, supporting rapid phase unwrapping on complex curved surfaces. Equipped with a dual-frequency laser (480Hz main frequency / 240Hz auxiliary frequency), it dynamically adjusts the fringe density to adapt to surfaces with different curvatures. FPGA-based timestamp management ensures synchronization between camera exposure and laser pulses. A three-dimensional mesh framework is constructed using 21 preset standard calibration target points, establishing a homography mapping matrix from the world coordinate system to the image coordinate system. A self-test procedure is periodically executed to verify the calibration effectiveness.

[0112] The depth map calculation unit performs a two-dimensional Fourier transform on the captured structured light image to extract the dominant frequency components and adjusts the bandpass filter parameters in real time. It uses a Wiener filter to suppress random noise interference and retain effective periodic signal components. It monitors the trend of spectral energy distribution changes in real time and dynamically adjusts the sampling window size. It uses B-spline curve fitting to estimate gray values ​​at non-integer positions. It combines bidirectional cubic convolution kernels to perform interpolation operations to generate continuous and smooth height field data.

[0113] The point cloud fusion processing unit constructs a two-level matching architecture from coarse registration to fine registration. In the initial stage, the RANSAC algorithm is used to remove the influence of outliers and quickly lock the transformation relationship. In the fine stage, the LM optimization algorithm is used to minimize the residual error function. The smoothing intensity coefficient is adaptively adjusted according to the local curvature estimation results to enhance the denoising effect in flat areas while protecting the integrity of sharp corner structures. For data missing areas caused by occlusion, Kriging interpolation, radial basis function interpolation, and Poisson reconstruction methods are used in sequence for repair.

[0114] The defect feature analysis module includes:

[0115] Multi-scale segmentation units are implemented using a local dynamic thresholding algorithm improved from the Otsu method. Initial segmentation thresholds are determined with the aid of K-means clustering, and boundary positions are iteratively optimized using a serpentine algorithm. Different color space conversion channel weight allocation schemes are designed for different background materials. A hierarchical connected component analysis process is established, first merging adjacent small-area fragments to form a potential object candidate set, and then filtering effective targets based on shape factors and grayscale statistics. A minimum acceptable area threshold is set to filter false positive detection results. The Chan-Vese level set method is used to refine rough edge contours. Boundary localization is achieved by tracking the gradient direction evolution path based on an active contour model.

[0116] The three-dimensional morphological analysis unit projects a two-dimensional binary image along the normal direction to construct a height histogram, integrates it to obtain the volume measurement value, and performs compensation calculations based on the perspective transformation correction factor; it uses the power spectral density function to analyze the height fluctuation characteristics; it decomposes the surface morphology into the contribution ratio of different wavelength components and calculates the roughness parameters; it fits the best reference plane as the datum plane and calculates the distance deviation distribution from each measurement point to this plane; it uses the least squares method to solve the coefficients of the ideal plane equation, statistically analyzes the maximum positive and negative deviation values ​​and their location coordinates, and generates a heat map of the out-of-tolerance area distribution.

[0117] Pareto chart generation unit: summarizes the frequency of occurrence by defect type and displays it in descending order; automatically identifies the main influencing factors (usually items that account for more than 80% of the cumulative percentage);

[0118] Correlation matrix construction unit: Analyze the interrelationships between various quality characteristics; display the results using a scatter plot matrix.

[0119] The machine learning classification module includes:

[0120] The data augmentation unit simulates various deformations that may occur during PCB manufacturing, including effects such as thermal expansion and contraction, and bending and twisting caused by mechanical stress. Synthetic samples are generated by applying a nonlinear transformation to the original images, expanding the data coverage of rare operating conditions in the training set. The CycleGAN network is trained to achieve a unified appearance transformation across supplier data, eliminating the impact of texture inconsistencies caused by batch differences in raw materials on classification performance.

[0121] The model architecture unit inserts an SENet module on top of ResNet to capture the importance difference information between channels; it caches the negative sample pool formed by misclassified samples within a set time period and periodically retrains the parameters of the last few layers of the network to correct erroneous memory patterns.

[0122] The inference deployment unit pre-simulates the impact of numerical truncation errors caused by low-precision quantization and adjusts the scaling factor of the batch normalization layer to mitigate quantization loss. Mixed-precision training is employed to balance speed and accuracy, finding the most suitable quantization bit width configuration for the target hardware platform. Redundant connections are iteratively removed based on a weight importance scoring metric. Multiple similar operation instruction sequences are executed in parallel using SIMD vector instructions.

[0123] The multi-scale feature enhancement module includes:

[0124] The multi-resolution input preprocessing unit performs dynamic histogram stretching on the original images under different lighting conditions and employs a block-based local normalized Gamma correction algorithm to eliminate the effects of non-uniform illumination. It uses a sliding window to statistically analyze the mean and standard deviation of brightness in each region and independently adjusts the contrast parameter to enhance details in dark areas. Channel and spatial domains are constructed. The spatial domain is based on an improved Squeeze-and-Excitation structure, obtaining a two-dimensional spatial weight map through global average pooling. The channel domain uses a bottleneck-style fully connected layer to compress the feature dimension. The two domains are then weighted and fused to generate an adaptive feature enhancement mask that highlights the response intensity of key regions.

[0125] The cross-level feature fusion unit adopts a hybrid strategy of deconvolution and bilinear interpolation to achieve resolution alignment, and dynamically adjusts the contribution ratio of each level through a gating mechanism.

[0126] An adaptive output adaptation layer automatically adjusts the loss weight coefficients of the classification / regression branches based on the downstream task type; the contribution of hard sample loss is reweighted using the focal loss function to improve the recall rate of extreme-sized targets; multiple independent decoders are set up in parallel to process feature maps of different scales; the optimal configuration scheme is determined through self-supervised learning under the supervision of cross-entropy loss; Monte Carlo Dropout technology is integrated to predict the model's prediction variance and output the quality score and confidence interval of each candidate box.

[0127] The displacement compensation calibration module includes:

[0128] The motion trajectory acquisition unit integrates a miniature interferometer composed of diffractive optical elements as a position sensor. It employs a differential signal receiving scheme to suppress common-mode interference and uses a lock-in amplifier to extract phase difference information. Based on the angular velocity and accelerometer data provided by the IMU and the visual odometry estimation results, a multi-source fusion motion state observer is constructed. A Kalman filter framework is used to fuse motion information from different frequency bands to compensate for high-frequency noise interference caused by mechanical vibration.

[0129] The dynamic registration optimization unit employs a GPU-parallelized semi-global matching algorithm to optimize disparity map generation efficiency and eliminates mismatched point pairs. The final output is a 3D point cloud dataset as the registration benchmark. The PCB is treated as an anisotropic hyperelastic material, constitutive equations are constructed, and the static equilibrium state under contact mechanical boundary conditions is solved using the finite element method. The local deformation effect caused by fixture pressure is simulated. The solution is obtained until the residual stress is less than a preset threshold, thus obtaining a true characterization of the workpiece deformation state. An optimization objective function is established based on the workpiece deformation state characterization, and the Levenberg-Marquardt algorithm is used to achieve fast convergence.

[0130] The real-time correction execution unit is based on a piezoelectric actuator to form a six-degree-of-freedom fine-tuning platform, and uses a capacitive displacement sensor to form a closed-loop control system to achieve displacement control resolution.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D defect detection system for PCB solder paste printing based on multispectral imaging, characterized in that, include: A multispectral image acquisition module based on an RGB and near-infrared four-channel industrial camera and a tunable LED light source array; The system acquires reflection characteristic data at different penetration depths through a wavelength switching mechanism; the motion control trigger module, based on a servo motor-driven XYZ three-axis stage, achieves positioning with encoder feedback; pulse width modulation signals are used to coordinate the camera shutter and platform movement speed; the 3D point cloud generation module, based on structured light projection, projects laser stripes of known patterns onto the target area, calculates depth information using a binocular stereo vision algorithm, aligns multi-view data through an iterative nearest-point registration algorithm, and constructs a dense point cloud model. The defect feature analysis module, based on an adaptive threshold segmentation algorithm, defines a comprehensive scoring function to perform defect feature analysis. The machine learning classification module constructs a convolutional neural network model, taking normalized multi-channel feature maps as input, and optimizes network parameters through backpropagation to improve the accuracy of defect identification; the human-computer interaction interface module displays real-time detection results and provides human-computer interaction functions. The multi-scale feature enhancement module downsamples the original image to three scales (1 / 2, 1 / 4, and 1 / 8) using bilinear interpolation, forming a multi-layered representation of a pyramid structure. Each layer extracts local features through independent convolutional blocks, with shallow layers preserving edge details and deep layers capturing large-scale topological anomalies. A combination of upsampling paths and skip connections is used to inject high-level semantic information into the low-resolution feature map. The displacement compensation calibration module performs phased corrections for image misalignment caused by mechanical motion errors. The simulation verification module establishes a virtual detection environment for algorithm pre-running and parameter optimization. The edge computing module decomposes computationally intensive operations into parallelizable subtask clusters; it leverages the bandwidth advantage of DDR4 memory to construct a multi-level buffer pool; and it preloads hot data blocks into on-chip memory based on an access pattern prediction algorithm. The self-supervised domain adaptation training module automatically collects normal samples as a benchmark dataset; it employs a progressive learning strategy to gradually expand class boundaries, updating only the parameters of the last two network layers in each iteration; it constructs a game model between the domain discriminator and the feature extractor; and it forces the generator to produce domain-invariant feature representations through a gradient inversion layer, aligning the data distributions of the source and target domains in the latent space. The center of the circular contour is detected by Hough transform based on pre-installed markers, and the deviation vector between the actual coordinates and the theoretical position is calculated; a rigid affine transformation model is used to correct the overall offset. Four reference points at the corner of the PCB are selected as the control set, and a nonlinear deformation field is generated using a thin plate spline interpolation algorithm. The rate of change of the feature point trajectory is monitored in real time during continuous scanning. When the acceleration exceeds the threshold, a prediction compensation mechanism is triggered. The position deviation of the next frame is estimated using a Kalman filter, and the acquisition timing parameters are adjusted in advance. The 3D point cloud generation module includes: a multimodal data acquisition unit, employing a Gray code and phase-shifting fringe composite projection mode, and encoding through four sets of orthogonal structured light patterns; each projection frame contains unique spatiotemporal marker information; equipped with a dual-frequency laser, dynamically adjusting the fringe density to adapt to surfaces with different curvatures; FPGA-based timestamp management to ensure that the camera exposure time is synchronized with the laser pulse; 21 preset standard calibration target points form a 3D mesh framework, establishing a homography mapping matrix from the world coordinate system to the image coordinate system; a depth map calculation unit, performing a two-dimensional Fourier transform on the captured structured light image, extracting the dominant frequency components, and adjusting the bandpass filter parameters in real time; using a Wiener filter to suppress random noise interference and retain effective periodic signal components; monitoring the trend of spectral energy distribution changes in real time and dynamically adjusting the sampling window size; using B-spline curve fitting to estimate gray values ​​at non-integer positions; and combining bidirectional cubic convolution kernels for interpolation operations to generate continuous and smooth height field data; The point cloud fusion processing unit constructs a two-level matching architecture from coarse registration to fine registration. In the initial stage, the RANSAC algorithm is used to remove the influence of outliers and quickly lock the transformation relationship. In the fine stage, the LM optimization algorithm is used to minimize the residual error function. The smoothing intensity coefficient is adaptively adjusted according to the local curvature estimation results to enhance the denoising effect in flat areas while protecting the integrity of sharp-edged structures. The defect feature analysis module includes: a multi-scale segmentation unit, based on a local dynamic threshold algorithm improved from the Otsu method, which uses K-means clustering to help determine the initial segmentation threshold and a snake algorithm to iteratively optimize the boundary position; a differentiated color space conversion channel weight allocation scheme designed for different material backgrounds; a hierarchical connected component analysis process, which first merges adjacent small-area fragments to form a potential object candidate set, and then filters effective targets based on the attributes of shape factor and grayscale statistics; sets a minimum acceptable area threshold to filter false positive detection results; uses the Chan-Vese level set method to refine rough edge contours; and tracks the gradient direction evolution path based on an active contour model to achieve boundary localization. The three-dimensional morphological analysis unit projects a two-dimensional binary image along the normal direction to construct a height histogram, integrates to obtain the volume measurement value, and performs compensation calculations based on the perspective transformation correction factor; it analyzes height fluctuation characteristics using the power spectral density function; it decomposes the surface morphology into the contribution ratio of different wavelength components and calculates roughness parameters; it fits the optimal reference plane as the datum plane and calculates the distance deviation distribution from each measurement point to this plane; it solves the coefficients of the ideal plane equation using the least squares method, statistically analyzes the maximum positive and negative deviation values ​​and their occurrence coordinates, and generates a heat map of the out-of-tolerance area distribution; the Pareto chart generation unit summarizes the frequency of occurrence by defect type and displays it in descending order; it automatically identifies influencing factors; the correlation matrix construction unit analyzes the interrelationships between various quality characteristics and displays them using a scatter plot matrix.

2. The PCB solder paste printing three-dimensional defect detection system based on multispectral imaging according to claim 1, characterized in that, The defect detection method of this system is as follows: Initialization phase: Load the process file to set the allowable error range, calibrate the equipment coordinate system and mechanical origin; perform white balance correction after preheating the light source to a stable state; Data acquisition loop, for each inspection station: Step 1: Move the stage to the target position according to the CAD path plan; Step 2: Turn on the light source of each channel in sequence and capture the corresponding frame image; Step 3: Trigger structured light projection and record the projection deformation mode; Step 4: After completing the multimodal data acquisition of the current area, move to the next station; The real-time processing threads execute the following tasks in parallel: Thread A: performs background subtraction and coarse localization of the region of interest; Thread B: runs a 3D reconstruction algorithm to generate a point cloud model; Thread C: calls a pre-trained model to perform preliminary defect screening; The main thread integrates the results and triggers an exception interruption mechanism.

3. The PCB solder paste printing three-dimensional defect detection system based on multispectral imaging according to claim 1, characterized in that, The simulation verification module performs algorithm pre-runs and parameter optimization, specifically: It builds a ray tracing model based on Zemax software to simulate the scattering distribution characteristics under different incident angles; optimizes the light source uniformity index; uses COMSOL to solve the Navier-Stokes equations to reproduce the solder paste flow process and quantify the curing shrinkage rate; combines a material viscosity curve database to predict the morphology evolution law under different coating speeds; and uses ANSYS Workbench to perform steady-state heat conduction analysis to evaluate the influence of the temperature gradient during the solder reflow stage on substrate warpage.

4. The PCB solder paste printing three-dimensional defect detection system based on multispectral imaging according to claim 1, characterized in that, The machine learning classification module includes: a data augmentation unit that simulates deformation during PCB manufacturing, generates synthetic samples by applying nonlinear transformations to the original images, and expands the data coverage under rare conditions in the training set; trains the CycleGAN network to achieve a unified appearance transformation of cross-supplier data; a model architecture unit that inserts an SENet module on top of ResNet to capture importance differences between channels; caches a negative sample pool formed by misclassified samples within a set time period; and an inference deployment unit that pre-simulates the impact of numerical truncation errors caused by low-precision quantization, adjusts the scaling factor of the batch normalization layer to reduce quantization loss, uses mixed-precision training to balance speed and accuracy, and finds the most suitable quantization bit width configuration scheme for the target hardware platform; and iteratively removes redundant connection edges based on the weight importance scoring index.

5. The PCB solder paste printing three-dimensional defect detection system based on multispectral imaging according to claim 1, characterized in that, The multi-scale feature enhancement module includes: a multi-resolution input preprocessing unit, which performs dynamic histogram stretching on the original image under different lighting conditions, and uses a block-based local normalized Gamma correction algorithm to eliminate the influence of non-uniform illumination; it calculates the mean and standard deviation of brightness in each region through a sliding window, and independently adjusts the contrast parameter to enhance details in dark areas; it constructs a channel domain and a spatial domain, the spatial domain is based on an improved Squeeze-and-Excitation structure, and obtains a two-dimensional spatial weight map through global average pooling; the channel domain uses a bottleneck-type fully connected layer to compress the feature dimension, and the two are weighted and fused to generate an adaptive feature enhancement mask; and a cross-level feature fusion unit, which uses a hybrid strategy of deconvolution and bilinear interpolation to achieve resolution alignment, and dynamically adjusts the contribution ratio of each level through a gating mechanism. An adaptive output adaptation layer automatically adjusts the loss weight coefficients of the classification / regression branches based on the downstream task type; the contribution of hard sample loss is reweighted using the focal loss function to improve the recall rate of extreme-sized targets; multiple independent decoders are set up in parallel to process feature maps of different scales; the optimal configuration scheme is determined through self-supervised learning under the supervision of cross-entropy loss; Monte Carlo Dropout technology is integrated to predict the model's prediction variance and output the quality score and confidence interval of each candidate box.

6. The PCB solder paste printing three-dimensional defect detection system based on multispectral imaging according to claim 1, characterized in that, The displacement compensation calibration module includes: a motion trajectory capture unit, which integrates a micro-interferometer composed of diffractive optical elements as a position sensor, uses a differential signal receiving scheme to suppress common-mode interference, and extracts phase difference information in conjunction with a lock-in amplifier; it constructs a multi-source fusion motion state observer based on the angular velocity and accelerometer data provided by the IMU and the visual odometry estimation results; and it uses a Kalman filter framework to fuse motion information of different frequency bands to compensate for high-frequency noise interference caused by mechanical vibration. The dynamic registration optimization unit employs a GPU-parallelized semi-global matching algorithm to optimize disparity map generation efficiency and eliminates mismatched point pairs. It ultimately outputs a 3D point cloud dataset as the registration benchmark. The PCB is treated as an anisotropic hyperelastic material, and constitutive equations are constructed. The static equilibrium state under contact mechanical boundary conditions is solved using the finite element method. Local deformation effects caused by fixture pressure are simulated. The solution is obtained until the residual stress is less than a preset threshold, thus obtaining a true representation of the workpiece deformation state. An optimization objective function is established based on the workpiece deformation state representation, and the Levenberg-Marquardt algorithm is used to achieve fast convergence. The real-time correction execution unit uses a piezoelectric actuator to construct a six-degree-of-freedom fine-tuning platform and a capacitive displacement sensor to form a closed-loop control system, achieving high displacement control resolution.

Images