A multi-modal imaging fusion-based circuit board defect detection and reconstruction technique

Abstract

The application discloses a kind of multi-modal imaging fusion's circuit board defect detection and reconstruction technology, it is intended to by in-depth analysis the polarization characteristic of circuit board, utilize the distinguishing characteristics of polarization degree and polarization phase to different materials, study the determination of optimal exposure time or the encoding mode of stripe, improve the speed and accuracy of measurement.Combining 2D image, 3D image and polarization image three kinds of modal images, as model multi-channel input, after extracting different modal features, fusion image reconstruction is carried out, and the quality of fusion image is promoted through joint loss, finally, fusion optical flow neural network realizes the accurate prediction of target space-time optical flow motion information, to realize the accurate detection of abnormal event.

Description

Technical Field

[0001] This invention relates to the field of electronic information technology, and in particular to a multimodal imaging fusion technology for circuit board defect detection and reconstruction. Background Technology

[0002] Printed circuit boards (PCBs), as core components of electronic products, directly determine product reliability through their manufacturing quality. With the miniaturization and high-density development of electronic products, the line width and spacing of PCBs are constantly shrinking, and the variety of components and surface materials presents significant challenges to automated optical inspection. Currently, mainstream inspection technologies rely primarily on high-resolution two-dimensional imaging, but this method is highly susceptible to lighting conditions. Areas with strong surface reflections are prone to overexposure, masking true defects; contaminants or micro-scratches with colors and textures similar to the substrate are difficult to identify due to insufficient contrast. While three-dimensional contour measurement technology can acquire height information, effectively supplementing the deficiencies of two-dimensional data, in high dynamic range scenarios, multi-exposure fusion technology is required to simultaneously and clearly capture highly reflective areas and dark details. This leads to a sharp increase in the number of acquisitions, severely limiting inspection efficiency and failing to meet the real-time requirements of online inspection. Polarization imaging technology can acquire the degree of polarization and polarization phase information of an object's surface. This information is directly related to the material's microstructure, roughness, and other physical properties, providing a new dimension for distinguishing different materials and identifying surface anomalies. However, effectively fusing polarization information with traditional two-dimensional and three-dimensional images to form a more comprehensive data representation that facilitates defect identification, and then achieving rapid and accurate automated detection based on this, remains a technical bottleneck for those skilled in the art. Therefore, the industry urgently needs an innovative detection solution that can overcome these limitations, significantly improving detection speed while ensuring high accuracy and robustness, to meet the demands of modern electronics manufacturing. Summary of the Invention

[0003] A multimodal imaging fusion technology for circuit board defect detection and reconstruction includes the following steps; S1. Polarization characteristic measurement: Using polarization measurement equipment, circuit board samples of different materials, processes and structures are measured to obtain their polarization degree and polarization phase data; S2. Modeling and Optimization: Based on the polarization degree and polarization phase data, establish the correspondence between material type and polarization characteristics, and determine the optimal exposure time and adaptive stripe coding strategy for different material regions; S3. Multimodal image acquisition and feature fusion: Simultaneously acquire two-dimensional images, three-dimensional topographic images and polarization images of the circuit board, extract the texture features of the two-dimensional images, the shape features of the three-dimensional topographic images and the polarization features of the polarization images respectively, and perform weighted fusion and dimensionality reduction optimization. S4. Image Reconstruction and Optimization: The fused features are input into an image reconstruction model based on a deep convolutional neural network, and a joint loss function including reconstruction error, texture consistency, shape consistency and polarization consistency is used to constrain the reconstruction process to generate a high-quality fused image. S5. Spatiotemporal optical flow detection: The serialized fused image is input into an optical flow neural network to predict the spatiotemporal motion information on the circuit board surface; S6. Defect Identification: Based on spatiotemporal motion information and preset anomaly rules, identify and locate defects on the circuit board.

[0004] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S2, the optimal exposure time and adaptive stripe coding strategy for different material regions are determined. The specific steps are as follows: S21. Material Region Division and Characteristic Analysis: Based on the acquired polarization degree and polarization phase data, analyze the differences in polarization characteristics of different materials, and establish a correspondence model between polarization characteristics and material types; use clustering algorithms to divide the circuit board surface into k regions with similar polarization characteristics; S22. Highlight Area Identification and Processing: Based on the Fresnel reflection principle, calculate the reflection characteristics of each area; based on the relationship between the vertical polarization component and the horizontal polarization component, identify the specular or soft specular areas that are prone to producing highlights, and mark them at the pixel level. S23. Adaptive Imaging Parameter Determination: For the k regions, determine their optimal exposure time to avoid overexposure or underexposure; for the identified highlight regions, formulate a specific stripe coding strategy, and generate an adaptive projection stripe pattern that can suppress highlights by reducing the corresponding maximum input gray value of the projection. A sinusoidal stripe projection with a maximum input gray value of 120 can be used, and the intensity mapping coefficient is optimized by the Levenberg-Marquardt algorithm to achieve high-quality image acquisition.

[0005] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S3, multimodal feature fusion and dimensionality reduction optimization are performed. The specific steps are as follows: S31. Construction of Multimodal Feature Extraction Network: Construct a feature extraction network with three parallel branches. Each branch consists of a common convolutional layer and two local residual learning modules, which are used to process two-dimensional images, three-dimensional topographic images and polarization images, respectively. S32. Feature Extraction and Preliminary Fusion: In each branch, shallow and deep features are extracted using convolution kernels and parametric rectified linear unit activation functions; subsequently, a weighted average algorithm is used to perform preliminary fusion of the feature vectors from the three branches; S33. Feature Dimensionality Reduction and Optimization: Principal Component Analysis (PCA) is used to reduce the dimensionality of the high-dimensional features after initial fusion. This extracts the main components that best represent the original information, removes redundant information, improves the robustness of the features, and prepares for subsequent reconstruction.

[0006] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S4, an image reconstruction model is used and optimized using a joint loss function. The specific steps are as follows: S41. Forward propagation of the reconstruction network: The fused and dimensionality-reduced features are input into the image reconstruction model. This model contains three cascaded convolutional layers and one 1×1 convolutional layer, which generates the reconstructed fused image through the forward computation of the network; S42. Joint Loss Calculation and Backpropagation: Calculate the joint loss function value of the reconstructed image. This function is a weighted average of reconstruction error, texture consistency loss, shape consistency loss, and polarization consistency loss. The Adam optimization algorithm is used, and the model parameters are updated through backpropagation based on the loss values. S43. Model Training and Weight Tuning: Iteratively train the model until the loss converges. During this process, dynamically adjust the weights of each component loss in the joint loss function based on the performance on the validation set to obtain the best reconstruction effect.

[0007] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. The specific steps for predicting the spatiotemporal motion information of the circuit board surface in step S5 are as follows: S51. Forward prediction of optical flow network: The sequential fused image is input into a trained optical flow neural network, which consists of a feature pyramid extraction subnetwork and an optical flow estimation subnetwork. S52. Coarse-to-fine optical flow estimation: The NetC subnetwork first extracts multi-scale feature pyramids for image pairs. Based on these features, the NetE subnetwork, through cascaded flow inference and regularization modules, achieves coarse-to-fine optical flow field estimation from low resolution to high resolution, outputting accurate spatiotemporal motion information.

[0008] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S6, defects are identified and located based on spatiotemporal motion information and preset anomaly rules. The specific steps are as follows: S61. Anomaly rule base construction: Based on historical defect data and production process knowledge, a variety of anomaly rules are preset. These rules include, but are not limited to: threshold rules based on the magnitude and direction of optical flow vectors, rules based on the continuity of optical flow field, and discrimination rules based on machine learning classifiers. S62. Spatiotemporal motion information analysis: Analyze the predicted spatiotemporal optical flow field and extract key features, such as the average optical flow vector, optical flow field gradient, and optical flow statistical characteristics of a specific region; S63. Defect Identification and Localization: The analyzed motion features are matched and compared with the constructed anomaly rule base. When the motion information meets a specific anomaly rule, it is determined that there is a defect in the area. At the same time, based on the image spatial location of the abnormal optical flow site, the defect is located at the pixel level or region level, and the type, location and confidence level of the defect are output. S64. Results Output and Feedback: Output the results of defect identification and location for subsequent alarm, sorting or maintenance guidance. At the same time, the detection results can be used as feedback data to update and optimize the anomaly rule base or machine learning model, so as to realize the self-iteration and improvement of the detection system.

[0009] A system for circuit board defect detection and reconstruction technology based on multimodal imaging fusion is provided. This system is used to implement any one of the aforementioned multimodal imaging fusion circuit board defect detection and reconstruction technologies. The system includes: a polarization characteristic measurement module, a modeling optimization module, a multimodal image acquisition and feature fusion module, an image reconstruction and optimization module, a spatiotemporal optical flow prediction module, and a defect identification module. The polarization characteristic measurement module uses polarization light measurement equipment to measure circuit board samples of different materials, processes and structures to obtain their polarization degree and polarization phase data. Modeling and optimization module: Based on the polarization degree and polarization phase data, establish the correspondence between material type and polarization characteristics, and determine the optimal exposure time and adaptive stripe coding strategy for different material regions; Multimodal image acquisition and feature fusion module: Based on the optimal exposure time and adaptive stripe coding strategy, it simultaneously acquires two-dimensional images, three-dimensional topographic images and polarization images of the circuit board; extracts texture features from the two-dimensional images, shape features from the three-dimensional topographic images and polarization features from the polarization images, and performs feature optimization using weighted average fusion and principal component analysis dimensionality reduction methods; Image Reconstruction and Optimization Module: Inputs the fused features into the image reconstruction model based on a deep convolutional neural network, integrates relevant joint loss optimization units, and uses a joint loss function that includes reconstruction error, texture consistency, shape consistency and polarization consistency to constrain the reconstruction process in order to generate a high-quality fused image; Spatiotemporal optical flow prediction module: The serialized fused image is input into the optical flow neural network, which includes a feature pyramid extraction subnetwork NetC and an optical flow estimation subnetwork NetE, for predicting the spatiotemporal motion information of the circuit board surface; Defect identification module: Based on the predicted spatiotemporal motion information and preset anomaly rules, identify and locate defects on the circuit board.

[0010] The beneficial effects of this invention are: 1. Improved Detection Efficiency: By utilizing polarization characteristics to pre-distinguish materials and guide adaptive exposure, rapid imaging of high dynamic range circuit board surfaces is achieved. This reduces the dozens of exposures required in traditional 3D measurement to just two, significantly improving data acquisition speed and laying the foundation for online real-time inspection. 2. Enhanced detection accuracy and robustness: By deeply fusing multimodal information such as two-dimensional texture, three-dimensional morphology and polarization physical characteristics, the generated fused image can complement the shortcomings of a single mode, effectively suppress specular interference, and enhance the contrast of weak defect features, thereby significantly improving the accuracy of complex defect identification and anti-interference ability. 3. Expand the dimensions of defect identification: By introducing optical flow neural network analysis to analyze the spatiotemporal motion information of sequential fused images, the system can not only detect static defects, but also capture dynamic anomalies related to the production process, thus achieving a more comprehensive perception and diagnosis of defects. 4. Enhanced automation and intelligence: From data acquisition and feature fusion to image reconstruction and defect identification, the entire process is driven by optimized algorithms and deep learning models, reducing reliance on manual experience and improving the consistency and repeatability of the detection process. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a multimodal imaging fusion technology for circuit board defect detection and reconstruction. Detailed Implementation

[0012] Example 1: A multimodal imaging fusion technology for circuit board defect detection and reconstruction includes the following steps; S1. Polarization characteristic measurement: Using polarization measurement equipment, circuit board samples of different materials, processes and structures are measured to obtain their polarization degree and polarization phase data; S2. Modeling and Optimization: Based on the polarization degree and polarization phase data, establish the correspondence between material type and polarization characteristics, and determine the optimal exposure time and adaptive stripe coding strategy for different material regions; S3. Multimodal image acquisition and feature fusion: Simultaneously acquire two-dimensional images, three-dimensional topographic images and polarization images of the circuit board, extract the texture features of the two-dimensional images, the shape features of the three-dimensional topographic images and the polarization features of the polarization images respectively, and perform weighted fusion and dimensionality reduction optimization. S4. Image Reconstruction and Optimization: The fused features are input into an image reconstruction model based on a deep convolutional neural network, and a joint loss function including reconstruction error, texture consistency, shape consistency and polarization consistency is used to constrain the reconstruction process to generate a high-quality fused image. S5. Spatiotemporal optical flow detection: The serialized fused image is input into an optical flow neural network to predict the spatiotemporal motion information on the circuit board surface; S6. Defect Identification: Based on spatiotemporal motion information and preset anomaly rules, identify and locate defects on the circuit board.

[0013] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S2, the optimal exposure time and adaptive stripe coding strategy for different material regions are determined. The specific steps are as follows: S21. Material Region Division and Characteristic Analysis: Based on the acquired polarization degree and polarization phase data, analyze the differences in polarization characteristics of different materials, and establish a correspondence model between polarization characteristics and material types; use clustering algorithms to divide the circuit board surface into k regions with similar polarization characteristics; S22. Highlight Area Identification and Processing: Based on the Fresnel reflection principle, calculate the reflection characteristics of each area; based on the relationship between the vertical polarization component and the horizontal polarization component, identify the specular or soft specular areas that are prone to producing highlights, and mark them at the pixel level. S23. Adaptive Imaging Parameter Determination: For each of the k divided regions, the optimal exposure time is determined to avoid overexposure or underexposure. For the identified highlight regions, a specific stripe coding strategy is formulated. By reducing the corresponding maximum input gray value of the projection, an adaptive projection stripe pattern that can suppress highlights is generated. A sinusoidal stripe projection with a maximum input gray value of 120 can be used, and the intensity mapping coefficient is optimized through the Levenberg-Marquardt algorithm to achieve high-quality image acquisition.

[0014] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S3, multimodal feature fusion and dimensionality reduction optimization are performed. The specific steps are as follows: S31. Construction of Multimodal Feature Extraction Network: Construct a feature extraction network with three parallel branches. Each branch consists of a common convolutional layer and two local residual learning modules, which are used to process two-dimensional images, three-dimensional topographic images and polarization images, respectively. S32. Feature Extraction and Preliminary Fusion: In each branch, shallow and deep features are extracted using convolution kernels and parametric rectified linear unit activation functions; subsequently, a weighted average algorithm is used to perform preliminary fusion of the feature vectors from the three branches; S33. Feature Dimensionality Reduction and Optimization: Principal Component Analysis (PCA) is used to reduce the dimensionality of the high-dimensional features after initial fusion. This extracts the main components that best represent the original information, removes redundant information, improves the robustness of the features, and prepares for subsequent reconstruction.

[0015] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S4, an image reconstruction model is used and optimized using a joint loss function. The specific steps are as follows: S41. Forward propagation of the reconstruction network: The fused and dimensionality-reduced features are input into the image reconstruction model. This model contains three cascaded convolutional layers and one 1×1 convolutional layer, which generates the reconstructed fused image through the forward computation of the network; S42. Joint Loss Calculation and Backpropagation: Calculate the joint loss function value of the reconstructed image. This function is a weighted average of reconstruction error, texture consistency loss, shape consistency loss, and polarization consistency loss. The Adam optimization algorithm is used, and the model parameters are updated through backpropagation based on the loss values. S43. Model Training and Weight Tuning: Iteratively train the model until the loss converges. During this process, dynamically adjust the weights of each component loss in the joint loss function based on the performance on the validation set to obtain the best reconstruction effect.

[0016] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. The specific steps for predicting the spatiotemporal motion information of the circuit board surface in step S5 are as follows: S51. Forward prediction of optical flow network: The sequential fused image is input into a trained optical flow neural network, which consists of a feature pyramid extraction subnetwork and an optical flow estimation subnetwork. S52. Coarse-to-fine optical flow estimation: The NetC subnetwork first extracts multi-scale feature pyramids of image pairs. Based on these features, the NetE subnetwork achieves coarse-to-fine optical flow field estimation from low resolution to high resolution through cascaded flow inference and regularization modules, outputting accurate spatiotemporal motion information.

[0017] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S6, defects are identified and located based on spatiotemporal motion information and preset anomaly rules. The specific steps are as follows: S61. Anomaly rule base construction: Based on historical defect data and production process knowledge, a variety of anomaly rules are preset. These rules include, but are not limited to: threshold rules based on the magnitude and direction of optical flow vectors, rules based on the continuity of optical flow field, and discrimination rules based on machine learning classifiers. S62. Spatiotemporal motion information analysis: Analyze the predicted spatiotemporal optical flow field and extract key features, such as the average optical flow vector, optical flow field gradient, and optical flow statistical characteristics of a specific region; S63. Defect Identification and Localization: The analyzed motion features are matched and compared with the constructed anomaly rule base. When the motion information meets a specific anomaly rule, it is determined that there is a defect in the area. At the same time, based on the image spatial location of the abnormal optical flow site, the defect is located at the pixel level or region level, and the type, location and confidence level of the defect are output. S64. Results Output and Feedback: Output the results of defect identification and location for subsequent alarm, sorting or maintenance guidance. At the same time, the detection results can be used as feedback data to update and optimize the anomaly rule base or machine learning model, so as to realize the self-iteration and improvement of the detection system.

[0018] A system for circuit board defect detection and reconstruction technology based on multimodal imaging fusion is provided. This system is used to implement any one of the aforementioned multimodal imaging fusion circuit board defect detection and reconstruction technologies. The system includes: a polarization characteristic measurement module, a modeling optimization module, a multimodal image acquisition and feature fusion module, an image reconstruction and optimization module, a spatiotemporal optical flow prediction module, and a defect identification module. The polarization characteristic measurement module uses polarization light measurement equipment to measure circuit board samples of different materials, processes and structures to obtain their polarization degree and polarization phase data. Modeling and optimization module: Based on the polarization degree and polarization phase data, establish the correspondence between material type and polarization characteristics, and determine the optimal exposure time and adaptive stripe coding strategy for different material regions; Multimodal image acquisition and feature fusion module: Based on the optimal exposure time and adaptive stripe coding strategy, it simultaneously acquires two-dimensional images, three-dimensional topographic images and polarization images of the circuit board; extracts texture features from the two-dimensional images, shape features from the three-dimensional topographic images and polarization features from the polarization images, and performs feature optimization using weighted average fusion and principal component analysis dimensionality reduction methods; Image Reconstruction and Optimization Module: Inputs the fused features into the image reconstruction model based on a deep convolutional neural network, integrates relevant joint loss optimization units, and uses a joint loss function that includes reconstruction error, texture consistency, shape consistency and polarization consistency to constrain the reconstruction process in order to generate a high-quality fused image; Spatiotemporal optical flow prediction module: The serialized fused image is input into the optical flow neural network, which includes a feature pyramid extraction subnetwork NetC and an optical flow estimation subnetwork NetE, for predicting the spatiotemporal motion information of the circuit board surface; Defect identification module: Based on the predicted spatiotemporal motion information and preset anomaly rules, identify and locate defects on the circuit board.

[0019] Example 2: This embodiment is applied to a hybrid circuit board containing gold-plated pads, an FR-4 epoxy glass cloth substrate, and black plastic connectors. Under traditional optical inspection, the board exhibits a high dynamic range due to the strong reflection of metallic areas and the light absorption of non-metallic areas, resulting in significant loss of detail and making it difficult to detect connectivity defects such as cold solder joints.

[0020] A multimodal imaging fusion technology for circuit board defect detection and reconstruction includes the following steps; S1. Polarization characteristics measurement: Using a polarization camera under the illumination of a ring LED polarization light source, a sample of a mixed material circuit board containing gold-plated pads, an FR-4 epoxy glass cloth substrate and a black plastic connector is measured to obtain its Stokes vectors S0, S1 and S2, and the polarization degree and polarization phase data of the entire field of view are calculated accordingly. S2. Modeling and Optimization: Based on the acquired polarization data, establish the mapping relationship between material type and polarization characteristics, and determine the optimal exposure time and adaptive stripe coding strategy for different material regions accordingly. S3. Multimodal image acquisition and feature fusion: Based on adaptive imaging parameters, simultaneously acquire two-dimensional brightness images, three-dimensional topographic images based on the phase deflection principle, and polarization angle images of the circuit board; S4. Image Reconstruction and Optimization: The 128-dimensional feature vector obtained after dimensionality reduction is input into the image reconstruction model, and a joint loss function is used to constrain and optimize the entire reconstruction process; S5. Spatiotemporal optical flow detection: Apply a hot air excitation to the circuit board under test for 2 seconds, with the temperature rising uniformly from 25°C to 40°C, and simultaneously acquire a fused image sequence at a rate of 30 frames per second during this process; S6. Defect Identification: Based on the optical flow field information and combined with the preset anomaly judgment rules, identify and locate the cold solder joint defects on the circuit board.

[0021] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S1, Stokes vectors S0, S1, and S2 are obtained, and the polarization degree and polarization phase data of the entire field of view are calculated accordingly. The specific steps are as follows: S1. Polarization characteristic measurement: The circuit board sample is measured using a polarization camera under the illumination of a ring-shaped LED polarization light source. The camera is rotated sequentially at 0°, 45°, 90°, and 135° polarizers, acquiring four images I0, I... 45 I90 I 135 According to the Stokes vector calculation formula, S0 = I0 + I 90 S1=I0-I 90 S2=I 45 -I 135 The Stokes vector for each pixel is calculated, and then the polarization degree of the entire field is calculated according to the formula DoLP=sqrt(S1²+S2²) / S0. The measurement data shows that the average polarization degree of the gold-plated pad area is 0.12, the FR-4 substrate area is 0.30, and the plastic connector area is 0.48.

[0022] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S2, based on the acquired polarization data, a mapping relationship between material type and polarization characteristics is established, and the optimal exposure time and adaptive stripe coding strategy for different material regions are determined accordingly. The specific steps are as follows: S21. Material Region Division and Characteristic Analysis: Using the K-means clustering algorithm, with the polarization degree value of each pixel as a unique feature, the entire circuit board surface was divided into k=3 clusters. The calculated cluster centers were 0.12, 0.30, and 0.48, respectively, precisely corresponding to the metal, substrate, and plastic materials. S22. Specular Region Identification and Processing: Based on Fresnel reflection theory, for each region, calculate its reflectivity ratio R for s-polarized light to p-polarized light. s / R p An empirical threshold of 3.5 was set, and areas with a reflectivity ratio greater than 3.5 were identified as specular highlight areas. After processing, 98.2% of the gold-plated pad areas were identified as highlight areas. S23. Adaptive Imaging Parameter Determination: In order to capture highlights and shadow details simultaneously in a single acquisition, differentiated optimal exposure parameters are determined. A short exposure of 3ms is used for the metal area to suppress overexposure, while an exposure of 8ms is used for the substrate and plastic areas to ensure the signal-to-noise ratio. For the highlight area, an adaptive fringe pattern is generated, and its maximum input grayscale value in the projector is limited from the standard 180 to 70.

[0023] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S3, based on adaptive imaging parameters, a two-dimensional brightness image, a three-dimensional topographic image based on the phase deflection principle, and a polarization angle image of the circuit board are acquired simultaneously. The specific steps are as follows: S31. Multimodal feature extraction: Construct a three-branch feature extraction network, with each branch consisting of a common convolutional layer and two local residual learning modules. In the polarization image branch, additional Gabor filter banks in four directions are used for convolution to enhance the response of anisotropic features. S32. Preliminary feature fusion: The 1024-dimensional feature vectors output from the three branches are fused with differentiated weights based on material partitions. For metal regions prone to defects, the polarization feature is given the highest weight of 0.5 to take advantage of its high sensitivity to surface conditions. For substrate and plastic regions, the two-dimensional texture feature is given a higher weight of 0.4. S33. Feature Dimensionality Reduction and Optimization: Principal component analysis was used to reduce the dimensionality of the spliced ​​3072-dimensional feature vector, retaining the first 128 principal components with a cumulative contribution rate of 99.2%, and removing redundant information.

[0024] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S4, the 128-dimensional feature vector obtained after dimensionality reduction is input into the image reconstruction model, and a joint loss function is used to constrain and optimize the entire reconstruction process. The specific steps are as follows: S41. Reconstruction Network Forward Propagation: The image reconstruction model consists of three convolutional layers with a kernel size of 3×3 and a 1×1 convolutional layer connected in series. The final output of the model is a fused image with 3 channels. S42. Joint Loss Calculation and Backpropagation: The joint loss function used is the total loss function, which specifically consists of mean square error reconstruction loss, texture consistency loss based on Gabor filter response, shape consistency loss based on Hausdorff distance, and polarization consistency loss based on polarization angle difference. The weight coefficients of each loss are set to 1.0, 0.5, 0.5, and 1.0, respectively. The polarization consistency loss is specifically calculated as the average absolute error between the reconstructed image and the input polarization angle image.

[0025] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S5, a hot air excitation lasting 2 seconds is applied to the circuit board under test, with the temperature uniformly rising from 25°C to 40°C. During this process, a fused image sequence is synchronously acquired at a rate of 30 frames per second. The specific steps are as follows: S51. Forward prediction of optical flow network: The sequence of 10 consecutively acquired images is used as input and fed into the feature pyramid extraction subnetwork NetC and the optical flow estimation subnetwork NetE of the optical flow neural network. S52. Optical flow estimation from coarse to fine: The optical flow estimation subnetwork NetE finally outputs optical flow fields at three different scales, with resolutions of 1 / 4, 1 / 2, and full resolution of the original image, respectively.

[0026] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S6, based on the optical flow field information and combined with preset anomaly judgment rules, the cold solder joint defects on the circuit board are identified and located. The specific steps are as follows: S61. Abnormal Rule Matching: The preset abnormal judgment rule is defined as follows: In the image sequence from the 3rd to the 6th frame after the start of thermal excitation, if the average optical flow vector amplitude of a certain solder ball region in the half-resolution optical flow field exceeds 2.5 standard deviations of the global average optical flow vector amplitude of the frame image, and the angle between the optical flow vector direction of the region and the mainstream direction of heat conduction is greater than 60°, then the solder ball region is determined to trigger an abnormal flag. S62. Defect Identification and Location: The system successfully detected an optical flow response that fully met the above-mentioned abnormal rules on a specific solder ball in the corner of the BGA package. Combined with the regional material information, the system finally determined that the solder ball had a cold solder joint defect and output that it was located in a 15-pixel × 15-pixel rectangular area centered at pixel coordinates x=245, y=178 in the full-resolution image.

[0027] Example 3: This embodiment focuses on detecting micron-level scratches on the surface of PCB copper foil. These scratches are shallow, typically between 1μm and 3μm, with extremely low contrast, and are almost invisible under normal lighting conditions. They are potential defects that can lead to open circuits or reduced current carrying capacity, posing a serious risk to product quality.

[0028] A multimodal imaging fusion technology for circuit board defect detection and reconstruction includes the following steps: S1. Polarization characteristics measurement: A monochromatic polarization camera is used, equipped with an electrically rotating polarizer with an accuracy better than 0.1°. The surface of the copper foil is measured under linearly polarized light illumination with an incident angle of 60°. Through a high-precision measurement system, the measurement uncertainty of its polarization phase is controlled within 0.1 degrees. S2. Modeling and Optimization: Based on the acquired high-precision polarization phase data, a quantitative correlation model between surface micro-geometric deformation and polarization phase gradient is established, and the optimal imaging parameters are determined accordingly. S3. Multimodal image acquisition and feature fusion: Simultaneously acquire two-dimensional brightness images, three-dimensional topography maps based on phase deflection, and high-precision polarization phase maps of the copper foil surface according to imaging parameters; S4. Image Reconstruction and Optimization: The dimensionality-reduced feature vectors are input into a dedicated image reconstruction model, and optimization is performed with the core objective of significantly enhancing scratch visibility; S5. Spatiotemporal optical flow detection: Under the simulated micro-vibration environment inherent in the conveyor belt of the production line, a fusion image sequence of 5 consecutive frames is acquired to introduce the necessary dynamic information; S6. Defect Identification: Based on the obtained full-resolution optical flow field gradient tensor analysis, micron-level scratch defects are identified and located.

[0029] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S2, based on the acquired high-precision polarization phase data, a quantitative correlation model between surface micro-geometric deformation and polarization phase gradient is established, and the optimal imaging parameters are determined accordingly. The specific steps are as follows: S21. Defect Feature Modeling: Using standard scratch samples with known depths of 1μm, 2μm and 3μm, basic measurements were performed. By performing linear regression analysis on the measurement data, a quantitative relationship model between scratch depth d and polarization phase gradient amplitude was established. S22. Adaptive Imaging Parameter Determination: Set the globally uniform optimal exposure time to 15ms, the projected stripes adopt the standard sinusoidal stripe pattern, and its maximum input grayscale value is set to 150.

[0030] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S3, a two-dimensional brightness image, a three-dimensional topography map based on phase deflection, and a high-precision polarization phase map of the copper foil surface are acquired simultaneously according to the imaging parameters. The specific steps are as follows: S31. Multimodal feature extraction: Construct a feature extraction network with three parallel branches. In the feature extraction branch responsible for processing polarization images, the Sobel operator is used to calculate the gradient magnitude of the polarization phase map. The calculated gradient map is then processed as an independent and important feature channel to directly enhance the response to scratch edges. S32. Preliminary feature fusion: The texture features extracted from the two-dimensional image, the height features extracted from the three-dimensional topography image, and the phase gradient features extracted from the polarization image are preliminarily fused using a direct channel stitching method to form a comprehensive feature set containing all modal information; S33. Feature Dimensionality Reduction and Optimization: The ultra-high-dimensional feature vector formed after initial splicing is reduced in dimensionality using principal component analysis, retaining the top 256 most important principal component features, with the cumulative contribution rate of principal components reaching 98.8%.

[0031] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S4, the dimensionality-reduced feature vector is input into a dedicated image reconstruction model, and optimization is performed with the core objective of significantly enhancing scratch visibility. The specific steps are as follows: S41. Reconstructing the forward propagation of the network: The gray value of each pixel in the image directly corresponds to the probability of a scratch defect existing at that location; S42. Joint Loss Calculation and Backpropagation: In the construction of the joint loss function, the weight of the texture consistency loss based on the multi-directional Gabor filter response is increased, and its coefficient is set to 2.0 to preserve and enhance the weak linear features caused by fine scratches.

[0032] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S5, under the simulated micro-vibration environment inherent in the conveyor belt of the production line, a sequence of 5 consecutive fused image frames is acquired to introduce necessary dynamic information. The specific steps are as follows: S51. Forward prediction of optical flow network: The acquired continuous image sequence is used as input and fed into the optical flow neural network for processing; S52. Coarse-to-fine optical flow estimation: The optical flow estimation subnetwork outputs full-resolution accurate optical flow field data for subsequent defect morphology analysis.

[0033] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S6, based on the obtained full-resolution optical flow field gradient tensor analysis, micron-level scratch defects are identified and located. The specific steps are as follows: S61. Anomaly rule matching: Calculate the maximum eigenvalue map of the optical flow field gradient tensor, use a linear structuring element with a length of 20 pixels and a width of 2 pixels to perform morphological path opening operation, and retain linear structure candidate regions that conform to the scratch morphology and have a response value greater than a preset threshold of 0.1. S62. Defect Identification and Location: The system successfully identified three linear structures on the copper foil surface that fully met the above morphological conditions. Combined with the quantitative relationship model, the system estimated the depths of these three scratches to be approximately 1.15μm, 2.08μm, and 1.75μm, respectively, and simultaneously achieved sub-pixel precision positioning of the scratch centerline.

[0034] Example 4: This embodiment detects and distinguishes typical contaminants and oxide spots on the surface of circuit boards. These defects are fundamentally different from the PCB substrate in terms of physicochemical composition, resulting in unique and distinguishable polarization characteristics after their interaction with light, which provides a physical basis for their classification and identification.

[0035] A multimodal imaging fusion technology for circuit board defect detection and reconstruction includes the following steps; S1. Polarization Characteristics Measurement: Using a polarization imaging system under a ring-shaped LED polarization light source, measurements were taken on known types of contaminants and oxide samples to obtain their complete Stokes vectors. The full-field polarization degree and polarization phase data were calculated. The distribution patterns of different contaminants in a two-dimensional feature space with polarization degree as the horizontal axis and polarization phase as the vertical axis were analyzed. Measurements revealed that fingerprint areas, due to the presence of grease, typically had a polarization degree below 0.10 and a dispersed polarization phase distribution; flux residue areas had a polarization degree between 0.15 and 0.25, with a relatively uniform phase; oxide spots exhibited a higher polarization degree, typically greater than 0.30, and a specific phase range. S2. Modeling Optimization: Based on the polarization degree and polarization phase data, establish a defect classification model based on polarization characteristics, rather than a simple material region division; S3. Multimodal image acquisition and feature fusion: Simultaneously acquire two-dimensional images, three-dimensional topographic images, and polarization images of the circuit board; S4. Image Reconstruction and Optimization: Input the dimensionality reduction features into the reconstruction model to generate a high-quality fused image; S5. Spatiotemporal optical flow detection: Acquire fused images of a single frame or a very short sequence under a stable observation environment; S6. Defect Identification: Combining polarization classification results with morphological rules, accurate identification and classification of pollutants are achieved.

[0036] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S2, a defect classification model based on polarization characteristics is established based on the polarization degree and polarization phase data, rather than a simple material region division. The specific steps are as follows: S21. Defect Feature Modeling: Based on sample data with known labels, a multi-classification model is trained using the support vector machine algorithm in the two-dimensional feature space of polarization degree and polarization phase. The input of the model is the mean polarization degree and standard deviation of polarization phase of each suspected region, and the output is the probability that the region belongs to "fingerprint", "flux", "oxidation" or "background". S22. Adaptive Imaging Parameter Determination: In order to achieve consistent and high-contrast imaging of different pollutants, the global optimal exposure time is set to 10ms, and the projection uses sinusoidal stripes with a maximum gray value of 130 to suppress highlights while retaining sufficient signal strength.

[0037] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S3, two-dimensional images, three-dimensional topographic images, and polarization images of the circuit board are acquired simultaneously. The specific steps are as follows: S31. Multimodal feature extraction: In the polarization feature branch, in addition to the original polarization image, the local variance map of the polarization degree is calculated as a feature channel. S32. Preliminary feature fusion: The features of the three modes are initially fused. For suspected pollutant areas initially identified by the classification model, the weight of their polarization features is increased from the basic 0.3 to 0.6 in the feature weighting fusion stage, forcing the fused features to pay more attention to the physicochemical properties of these areas. S33. Feature Dimensionality Reduction and Optimization: Principal component analysis is used to reduce the dimensionality of the fused high-dimensional features, and the first 200 principal component features are retained for subsequent reconstruction.

[0038] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S4, the dimensionality reduction features are input into the reconstruction model to generate a high-quality fused image. The specific steps are as follows: S41. Forward propagation of the reconstruction network: The image reconstruction model consists of an input layer, three hidden layers and an output layer connected in series. The input layer receives a 200-dimensional reduced feature vector. The three hidden layers are all convolutional layers with kernel sizes of 3x3, 3x3 and 3x3 respectively, and the number of channels are 128, 64 and 32 respectively. They all use parametric rectified linear units as activation functions to introduce controllable nonlinearity and alleviate the gradient vanishing problem. The output layer is a 1x1 convolutional layer with a hyperbolic tangent function as activation function. The final output is mapped to the interval [-1,1] and rescaled into a three-channel fused image. S42. Joint Loss Calculation and Backpropagation: In the joint loss function, the weight of polarization consistency loss is increased to 1.5 to ensure that the reconstructed image is highly consistent with the input data in terms of polarization properties, thereby highlighting the difference between contaminants and substrate in the results.

[0039] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S5, under a stable observation environment, a single frame or a very short sequence of fused images is acquired. The specific steps are as follows: S51. Optical Flow Network Forward Prediction: The current frame and a clean reference background frame are input into the optical flow neural network. The network adopts an architecture consisting of a feature pyramid extraction subnetwork NetC and an optical flow estimation subnetwork NetE. The NetC subnetwork is built based on the VGG-16 backbone network and extracts multi-scale feature pyramids of image pairs. The NetE subnetwork is based on the cascade optimization idea of ​​PWC-Net, uses the Cost Volume layer to match in the feature space, and achieves coarse-to-fine optical flow estimation through a series of convolutional layers and upsampling layers. S52. Coarse-to-fine optical flow estimation: The optical flow estimation subnetwork NetE outputs a full-resolution optical flow field, whose main components come from the random noise of the system itself, the unavoidable small deformations of the test object, or the subtle fluctuations in lighting conditions. Under these micro-movements, the contaminants exhibit small displacement or deformation patterns that are different from those of the surrounding stable substrate, and these patterns are encoded in the optical flow field.

[0040] Furthermore, a multimodal imaging fusion technology for circuit board defect detection and reconstruction is proposed. In step S6, the polarization classification results are combined with morphological rules to achieve accurate identification and classification of pollutants. The specific steps are as follows: S61. Anomaly Rule Matching and Feature Fusion Decision: The pre-defined anomaly rule base is a multi-level, multi-criteria decision system that integrates polarization physical properties, morphological geometric features, and dynamic responses. Polarization attribute rules: Based on the trained support vector machine classification model, a probability threshold is set. When the probability of a region being identified as a "fingerprint" is higher than 0.90, or the probability of being identified as "flux" is higher than 0.85, or the probability of being identified as "oxidation" is higher than 0.88, the region will obtain a preliminary classification label with high confidence. There are auxiliary rules based on the original polarization parameters, such as "if the mean polarization degree of a region is lower than 0.12 and the standard deviation of polarization phase is higher than 0.05, then the evidence that it is identified as a 'fingerprint' or 'oil contaminant' is strengthened". Morphological rules: For candidate defect regions obtained through image segmentation, a series of morphological features are calculated, and thresholds are set for filtering and verification; Area rules: Noise points with an area less than 25 square pixels are excluded, and "fingerprint" contamination is usually defined as having an area greater than 300 square pixels; Shape rules: The roundness and elongation of the region are calculated. "Fluoride residue" usually manifests as an irregular shape with rough boundaries, while "oxidation spots" may be closer to a circle or ellipse; Distribution rules: Multiple small regions distributed in a splatter or linear pattern are identified and merged to determine "fluoride sputtering"; Dynamic response rule: Analyze the optical flow field and calculate the average optical flow vector amplitude within each candidate defect region. Compared to a stable substrate or oxide layer, incompletely cured "flux" or attached "dust" exhibits a slightly higher optical flow response under micro-vibration conditions due to the instability of its physical state, with the average amplitude exceeding the background mean by more than 1.5 standard deviations. S62. Defect Identification and Location: The system executes a weighted decision-making process to make a final judgment on each candidate region. Feature synthesis: For each candidate defect region, the system synthesizes its support vector machine classification probability vector, morphological feature vector, and regional average optical flow amplitude; Decision-making: If a region is classified as "flux" by the support vector machine with a 92% probability, and its shape is irregular with a significant optical flow response, the system will ultimately classify it as "flux residue" and assign it a high confidence level. If a region has an ambiguous classification probability by the support vector machine, but its polarization degree is consistently higher than 0.35 and its shape is regular, the system may classify it as "oxidation spot" based on polarization attribute rules and morphological rules, but the labeling confidence level will be medium, and manual review is recommended. Precise location and output: For the finally confirmed defects, the system uses its minimum bounding rectangle or precise pixel-level contour for location. The output results include: defect type, location information, confidence level and estimated defect area. This structured data is sent to the manufacturing execution system in real time for subsequent alarm, statistical analysis and maintenance guidance.

Claims

1. A method for detecting and reconstructing circuit board defects using multimodal imaging fusion, characterized in that, Includes the following steps; S1. Polarization characteristic measurement: Using polarization measurement equipment, circuit board samples of different materials, processes and structures are measured to obtain their polarization degree and polarization phase data; S2. Modeling and Optimization: Based on polarization degree and polarization phase data, establish the correspondence between material type and polarization characteristics, and determine the optimal exposure time and adaptive stripe coding strategy for different material regions; based on the polarization phase information, calculate the Fresnel reflection coefficient of the object surface to s-rays and p-rays; according to the region characteristics where the vertical polarization component is greater than the horizontal polarization component, identify and segment the highlight region on the circuit board surface; mark the highlight region at the pixel level, and reduce the maximum input gray value of its corresponding projected stripes to suppress overexposure; The projection is performed using sinusoidal stripes with a maximum input gray value of 120, and the intensity mapping coefficient of each region is optimized by weighted nonlinear least squares regression to generate an adaptive projection stripe pattern. S3. Multimodal Image Acquisition and Feature Fusion: Two-dimensional (2D), three-dimensional (3D) topography, and polarization images of the circuit board are acquired simultaneously. Texture features from the 2D images, shape features from the 3D topography, and polarization features from the polarization images are extracted, and weighted fusion and dimensionality reduction optimization are performed. A network containing three parallel feature extraction streams is used for feature extraction. Each feature extraction stream consists of a common convolutional layer and two local residual learning modules. The common convolutional layer uses parametric rectified linear units (RCUs) as activation functions. Feature fusion employs a weighted average algorithm, and principal component analysis (PCA) is used to reduce the dimensionality of the initially fused features to extract principal components and remove redundant information. S4. Image Reconstruction and Optimization: The fused features are input into an image reconstruction model based on a deep convolutional neural network, and a joint loss function including reconstruction error, texture consistency, shape consistency and polarization consistency is used to constrain the reconstruction process to generate a fused image; S5. Spatiotemporal optical flow detection: Input the serialized fused image into the optical flow neural network to predict the spatiotemporal motion information on the surface of the circuit board; S6. Defect Identification: Based on spatiotemporal motion information and preset anomaly rules, identify and locate defects on the circuit board.

2. The method for circuit board defect detection and reconstruction based on multimodal imaging fusion as described in claim 1, characterized in that, The optical flow neural network consists of a first sub-network NetC and a second sub-network NetE. The NetC sub-network is used to convert the input image pairs into multi-scale high-dimensional feature pyramids, and the NetE sub-network contains a cascaded flow inference module and a regularization module to achieve coarse-to-fine flow field estimation.

3. The method for circuit board defect detection and reconstruction based on multimodal imaging fusion as described in claim 1, characterized in that, In the image reconstruction and optimization step, the image reconstruction model consists of three cascaded convolutional layers and one 1×1 convolutional layer. The cascaded convolutional layers use parametric rectified linear units as activation functions, and the 1×1 convolutional layer uses the Tanh function as activation function. The joint loss function includes mean square error reconstruction loss, texture consistency loss based on Gabor filter, shape consistency loss based on Hausdorff distance, and polarization consistency loss based on polarization parameter difference.

4. The method for circuit board defect detection and reconstruction based on multimodal imaging fusion as described in claim 1, characterized in that, The joint loss function is optimized using the Adam optimization algorithm for iterative training, and the weights of each loss term are dynamically adjusted based on the quantitative and qualitative evaluation of the reconstruction results to balance errors from different aspects.

5. A system for circuit board defect detection and reconstruction using multimodal imaging fusion, characterized in that, The system of the circuit board defect detection and reconstruction method based on multimodal imaging fusion is used to implement the circuit board defect detection and reconstruction method based on multimodal imaging fusion as described in any one of claims 1-4; the system of the circuit board defect detection and reconstruction method based on multimodal imaging fusion includes: a polarization characteristic measurement module, a modeling optimization module, a multimodal image acquisition and feature fusion module, an image reconstruction and optimization module, a spatiotemporal optical flow prediction module, and a defect identification module; The polarization characteristic measurement module uses polarization light measurement equipment to measure circuit board samples of different materials, processes and structures to obtain their polarization degree and polarization phase data. Modeling and optimization module: Based on the polarization degree and polarization phase data, establish the correspondence between material type and polarization characteristics, and determine the optimal exposure time and adaptive stripe coding strategy for different material regions; Multimodal image acquisition and feature fusion module: Based on the optimal exposure time and adaptive stripe coding strategy, it simultaneously acquires two-dimensional images, three-dimensional topographic images and polarization images of the circuit board; extracts texture features from the two-dimensional images, shape features from the three-dimensional topographic images and polarization features from the polarization images, and performs feature optimization using weighted average fusion and principal component analysis dimensionality reduction methods; Image Reconstruction and Optimization Module: Inputs the fused features into the image reconstruction model based on a deep convolutional neural network, integrates relevant joint loss optimization units, and uses a joint loss function that includes reconstruction error, texture consistency, shape consistency and polarization consistency to constrain the reconstruction process in order to generate a high-quality fused image; Spatiotemporal optical flow prediction module: The serialized fused image is input into the optical flow neural network, which includes a feature pyramid extraction subnetwork NetC and an optical flow estimation subnetwork NetE, for predicting the spatiotemporal motion information of the circuit board surface; Defect identification module: Based on the predicted spatiotemporal motion information and preset anomaly rules, identify and locate defects on the circuit board.

Images