Circuit board defect recognition method and system based on multi-dimensional image data

Abstract

The present application relates to a circuit board defect recognition method and system based on multi-dimensional image data, belonging to the technical field of data recognition processing, comprising the following steps: obtaining the synchronous image data of the circuit board to be detected under multiple imaging modalities; performing spatial-spectral joint registration on each modality image to generate a multi-dimensional image cube with a unified coordinate system and pixel alignment; inputting the multi-dimensional image cube into a pre-trained multi-branch heterogeneous fusion neural network; generating a pixel-level defect probability map using a defect perception context decoder, and performing geometric constraint optimization on the probability map in combination with prior information of the circuit board design layout; outputting the defect type, location and confidence, and establishing an interpretable defect fingerprint library according to the multi-dimensional response characteristics of the defects; the beneficial effects of the present application are: it can effectively suppress false defect signals caused by image noise and circuit board surface texture interference, greatly reduce the missed detection rate and false detection rate, and realize accurate pixel-level defect positioning.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data recognition processing, and particularly relates to a circuit board defect recognition method and system based on multi-dimensional image data. BACKGROUND

[0002] As a core component of electronic equipment, the quality of a circuit board directly affects the performance and reliability of the electronic equipment. With the continuous development of electronic technology, circuit boards are developing towards high density, miniaturization and multi-layerization, and the traditional manual detection method has been difficult to meet the requirements of detection accuracy and efficiency. Therefore, using automated and high-precision circuit board defect recognition technology has become a research hotspot.

[0003] The existing circuit board defect recognition method is based on single image data for defect recognition, such as visible light image detection. Visible light image detection mainly identifies the color and texture changes on the surface of the circuit board to determine defects, but it is difficult to effectively identify the internal virtual welding, short circuit and hidden defects of the circuit board. In addition, single image data is easily affected by environmental factors such as light and noise, resulting in low recognition accuracy.

[0004] In order to improve the defect recognition effect, some studies use two kinds of image data for fusion detection, such as visible light and infrared image fusion. However, this two-dimensional image fusion method still has the problem of insufficient information complementation, and cannot fully reflect the physical characteristics and defect state of the circuit board. At the same time, the existing fusion method mainly uses simple pixel-level fusion or feature-level fusion, which is difficult to fully excavate the deep features of image data in each dimension, resulting in poor fusion effect and the need to improve the accuracy of defect recognition. SUMMARY

[0005] The application provides a circuit board defect recognition method and system based on multi-dimensional image data, which solves the technical problem of low defect recognition accuracy of image data on circuit boards in the prior art, combines circuit board design layout prior information for geometric constraint optimization, can effectively suppress false defect signals caused by image noise and circuit board surface texture interference, greatly reduces the miss detection rate and false detection rate, and finally realizes pixel-level accurate defect positioning.

[0006] In order to achieve the above purpose, the application realizes the following technical scheme:

[0007] The circuit board defect recognition method based on multi-dimensional image data comprises the following steps:

[0008] Synchronous image data of a circuit board to be detected under multiple imaging modalities is obtained;

[0009] The spatial-spectral joint registration is performed on the images of each modality to generate a multi-dimensional image cube with a unified coordinate system and pixel alignment;

[0010] A multidimensional image cube is input into a pre-trained multi-branch heterogeneous fusion neural network; each branch processes the features of its corresponding modality, and the image information of different modalities in the local region is dynamically weighted and fused through a cross-modal attention mechanism.

[0011] Based on the fused feature map, a pixel-level defect probability map is generated using a defect-aware context decoder, and geometric constraint optimization is performed on the probability map by combining prior information of the circuit board design layout.

[0012] Based on the optimized defect probability map, the defect type, location, and confidence level are output. An interpretable defect fingerprint database is established based on the multidimensional response characteristics of the defects for subsequent defect tracing and process feedback.

[0013] Optionally, for simultaneous image data acquisition across multiple imaging modalities, the following steps can be taken:

[0014] Synchronous triggering is initiated by sending a unified trigger signal through the synchronization controller, causing the three imaging devices to start exposure / acquisition at the same time. For pushbroom hyperspectral cameras, a precision displacement platform needs to move at a constant speed while simultaneously triggering line scanning.

[0015] Image acquisition records RGB or grayscale images using visible light to reflect solder joints, traces, and component appearance; infrared thermal imaging records temperature distribution maps to identify overheated areas or cold solder joints.

[0016] Post-processing and data alignment employ temporal alignment and spatial registration, which maps infrared and hyperspectral images to the visible light image coordinate system through affine transformation, thin plate spline, or deep learning registration methods, outputting a three-channel or multi-channel fused image.

[0017] Optionally, multiple images of different modalities can be acquired. For spatial-spectral joint registration, a reference image is used as a reference, and geometric transformations are applied to other images to achieve spatial registration. Since different sensors have different spectral responses, spectral matching or reconstruction is required to achieve spectrally consistent registration.

[0018] Furthermore, spatial registration uses the spatial registration loss function in image registration to measure the degree of alignment between the transformed registered image and the reference image in terms of spatial structure.

[0019] Spectral consistency registration uses a spectral registration loss function to measure the degree of matching between the spectral information of an image and a reference image after spatial transformation and spectral mapping.

[0020] Furthermore, to generate a multidimensional image cube with a unified coordinate system and pixel alignment, the following steps are taken:

[0021] Standardize and preprocess the original multi-source, multi-temporal, or multispectral image datasets;

[0022] Then, a high-precision spatial registration algorithm is used to accurately transform all images into a pre-defined unified coordinate system, achieving pixel-level precise alignment.

[0023] In the constructed multidimensional image cube, each layer corresponds to a two-dimensional slice of the original dataset, clearly preserving the spatial two-dimensional information of the slice itself.

[0024] Optionally, for each branch, the corresponding modal features are processed separately. A multi-branch feature extraction method is adopted, which extracts high-level features for each modality through the encoder branch corresponding to each modality.

[0025] The method for constructing cross-modal attention weights to dynamically weight and fuse image information from different modalities in local regions is based on query-key cross-modal attention. It uses each spatial location to calculate the importance weight of each modality to the spatial location, with a reference modality as the query and the other modalities as the key.

[0026] Optionally, defect-aware context decoding utilizes the fused feature map and employs a context decoder to capture contextual information. The decoder then generates a probability value for each pixel belonging to a defect region, forming a defect probability map.

[0027] Furthermore, the defect probability map is used to calculate the probability that a specified location is a defect. The softmax function or other normalization methods are applied to the last layer of the decoder to obtain a probability value between 0 and 1. This process transforms defective or qualified features into digital signal strength, weighted signals, and probabilities, quantifying the possibility that physical features deviate from the standard.

[0028] Optionally, the construction of the interpretable defect fingerprint database involves the structured extraction and storage of multidimensional response features of defects to create a database that ensures each defect possesses a unique feature fingerprint. The specific construction process is as follows:

[0029] Structured extraction of multidimensional response features of defects; for each identified defect, the core feature parameters under the three imaging modalities are extracted to form a structured defect fingerprint;

[0030] The defect fingerprint is classified, stored, and indexed. The extracted structured defect fingerprints are classified and stored according to defect type, circuit board model, and production batch. At the same time, a multi-dimensional index is built for each defect fingerprint. Each defect fingerprint is also associated with the original multi-dimensional image data, the optimized probability map, and the core defect information, so as to ensure that complete detection data can be quickly retrieved during subsequent traceability, and realize the whole process of traceability from defect fingerprint to original data to recognition result.

[0031] A circuit board defect identification system based on multidimensional image data includes:

[0032] The multi-dimensional image data acquisition module is used to acquire synchronous image data of the circuit board under inspection in multiple imaging modalities;

[0033] The image registration module is used to perform spatial-spectral joint registration on images of various modalities to generate a multidimensional image cube with a unified coordinate system and pixel alignment.

[0034] The multimodal feature fusion module has a built-in pre-trained multi-branch heterogeneous fusion neural network. Each branch of the neural network processes the features of the corresponding modality and dynamically weights and fuses the image information of different modalities in the local area through a cross-modal attention mechanism.

[0035] The defect probability map generation and optimization module is used to generate a pixel-level defect probability map based on the fused feature map output by the multimodal feature fusion module, and to perform geometric constraint optimization on the defect probability map by combining the prior information of the circuit board design layout.

[0036] The defect identification and tracing module is used to output the defect type, location and confidence level based on the optimized defect probability map, and to establish an interpretable defect fingerprint database based on the multidimensional response characteristics of the defect. The defect fingerprint database is used for subsequent defect tracing and process feedback.

[0037] The beneficial effects of this invention are:

[0038] The defect probability map generation and optimization module of the present invention combines prior information of circuit board design layout for geometric constraint optimization, which can effectively suppress false defect signals caused by image noise and interference from circuit board surface texture, significantly reduce the false detection rate and false detection rate, and ultimately achieve pixel-level accurate defect localization.

[0039] Specifically, to enhance the accuracy of defect localization, a pixel-level defect probability map is generated through a defect-aware context decoder, and prior information from the circuit board design layout is incorporated for geometric constraint optimization, achieving precise defect localization and contour refinement. The integration of prior design layout information effectively eliminates interference from the normal structure of the circuit board on defect identification, corrects blurred defect boundaries in the probability map, and makes defect location more accurate and contours clearer, providing precise locational basis for subsequent defect assessment and repair. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0042] Figure 2 This is a schematic diagram of the workflow of the present invention. Detailed Implementation

[0043] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0044] Example 1;

[0045] like Figure 1 As shown, this embodiment provides a circuit board defect identification system based on multi-dimensional image data, including:

[0046] The multi-dimensional image data acquisition module is used to acquire synchronous image data of the circuit board under inspection in multiple imaging modalities;

[0047] The image registration module is used to perform spatial-spectral joint registration on images of various modalities to generate a multidimensional image cube with a unified coordinate system and pixel alignment.

[0048] The multimodal feature fusion module has a built-in pre-trained multi-branch heterogeneous fusion neural network. Each branch of the neural network processes the features of the corresponding modality and dynamically weights and fuses the image information of different modalities in the local area through a cross-modal attention mechanism.

[0049] The defect probability map generation and optimization module is used to generate a pixel-level defect probability map based on the fused feature map output by the multimodal feature fusion module, using a defect-aware context decoder, and to perform geometric constraint optimization on the defect probability map by combining prior information of the circuit board design layout.

[0050] The defect identification and tracing module is used to output the defect type, location and confidence level based on the optimized defect probability map, and to establish an interpretable defect fingerprint database based on the multidimensional response characteristics of the defect. The defect fingerprint database is used for subsequent defect tracing and process feedback.

[0051] Among them, the defect probability map generation and optimization module combines prior information of circuit board design layout for geometric constraint optimization, which can effectively suppress false defect signals caused by image noise and interference from circuit board surface texture, significantly reduce the false detection rate and false detection rate, and ultimately achieve pixel-level accurate defect location.

[0052] Example 2;

[0053] Based on Example 1, such as Figure 2 As shown, this embodiment provides a circuit board defect identification method based on multi-dimensional image data, including the following steps:

[0054] Step 1: Acquire synchronous image data of the circuit board under test in multiple imaging modalities; wherein, the multiple imaging modalities include visible light images, infrared thermal imaging images, and hyperspectral images;

[0055] Step 2: Perform spatial-spectral joint registration on each modality image to generate a multidimensional image cube with a unified coordinate system and pixel alignment;

[0056] Step 3: Input the multidimensional image cube into a pre-trained multi-branch heterogeneous fusion neural network; where each branch processes the features of its corresponding modality and dynamically weights and fuses the image information of different modalities in the local region through a cross-modal attention mechanism;

[0057] Step 4: Based on the fused feature map, a pixel-level defect probability map is generated using a defect-aware context decoder, and geometric constraint optimization is performed on the probability map by combining prior information of the circuit board design layout.

[0058] Step 5: Based on the optimized defect probability map, output the defect type, location, and confidence level, and establish an interpretable defect fingerprint database based on the multidimensional response characteristics of the defects for subsequent defect tracing and process feedback.

[0059] This method, through the synergistic design of multimodal data fusion and advanced neural network architecture, effectively overcomes the limitations of traditional single-modal defect identification methods, significantly improving recognition performance and practicality. Specific technical effects are as follows:

[0060] Multi-dimensional data complementarity enhances defect coverage and detection capabilities. By simultaneously acquiring image data from three different imaging modalities—visible light, infrared thermal imaging, and hyperspectral imaging—comprehensive capture of circuit board defect information is achieved. Visible light images clearly reveal surface physical defects (e.g., scratches, uneven soldering); infrared thermal imaging accurately reflects abnormal circuit heating defects (e.g., short circuits, uneven heat distribution caused by poor soldering); and hyperspectral images uncover latent defects related to material composition changes (e.g., insulation aging, core material damage). The synergistic application of multi-modal data effectively compensates for the limitations of single-modal methods in detecting complex and latent defects, significantly improving the defect detection rate and reducing the risk of missed detections.

[0061] Precise registration ensures a solid foundation for multimodal fusion. A spatial-spectral joint registration process generates a multidimensional image cube with a unified coordinate system and pixel alignment, completely resolving spatial misalignment and pixel mismatch issues caused by differences in imaging principles between different modalities. The registered multidimensional image data enables pixel-level information association, ensuring accurate correspondence of modal information during subsequent feature fusion and avoiding fusion deviations caused by data misalignment. This provides a solid data foundation for high-quality feature extraction and fusion.

[0062] Intelligent fusion enhances the accuracy and generalization of defect identification. A pre-trained multi-branch heterogeneous fusion neural network is employed, where each branch specifically processes corresponding modal features. Combined with a cross-modal attention mechanism, local region image information is dynamically weighted and fused, achieving intelligent filtering and efficient fusion of multimodal features. This architecture adaptively identifies the information contribution of different modalities in a specified defect region, strengthening effective features and suppressing redundant noise, significantly improving the completeness and discriminative power of feature representation. Compared to traditional fixed-weight fusion methods, it is more adaptable to complex scenarios and diverse defect types, effectively improving defect identification accuracy, reducing false detection rates, and exhibiting superior generalization capabilities.

[0063] Prior constraint optimization enhances the accuracy of defect localization. A pixel-level defect probability map is generated through a defect-aware context decoder, and prior information from the circuit board design layout is incorporated for geometric constraint optimization, achieving precise defect localization and contour refinement. The integration of prior design layout information effectively eliminates interference from the normal circuit board structure on defect identification, corrects blurred defect boundaries in the probability map, and makes defect location more accurate and contours clearer, providing precise locational basis for subsequent defect assessment and repair.

[0064] Interpretability and traceability enable a closed-loop process optimization mechanism. By outputting complete information on defect type, location, and confidence level, and establishing an interpretable defect fingerprint database based on the multidimensional response characteristics of defects, this not only overcomes the drawbacks of traditional deep learning models' "black box" identification, making defect identification results traceable, but also provides data support for subsequent defect tracing and process feedback. Relying on the defect fingerprint database, it is possible to quickly associate defects with corresponding process steps (such as welding processes, material selection, and assembly procedures), accurately locate process weaknesses, and help enterprises optimize production processes and improve product quality, forming a closed-loop management system of "identification-tracing-optimization," demonstrating significant engineering application value and economic benefits.

[0065] Example 3;

[0066] Based on Example 2, in step 1, the following steps are used for acquiring synchronous image data under multiple imaging modes:

[0067] Step a: Place the circuit board to be tested; fix the circuit board on the stage, ensuring the surface is flat and unobstructed; if it is necessary to detect abnormalities in the working state (such as overheating or short circuit), the circuit board can be powered on.

[0068] Step b: Initiate synchronous triggering; send a unified trigger signal through the synchronization controller so that the three imaging devices start exposure / acquisition at the same time; for pushbroom hyperspectral cameras, a precision displacement platform needs to move at a constant speed while simultaneously triggering line scanning; or a snapshot hyperspectral camera can be used to achieve single-frame full-frame acquisition, which is easier to synchronize.

[0069] Step c: Image acquisition; visible light image recording RGB or grayscale images to reflect solder joints, traces and component appearance; infrared thermal imaging image recording temperature distribution map to identify overheated areas or cold solder joints; hyperspectral image acquisition of the spectral curve of each pixel for material identification and oxidation / contamination detection.

[0070] Step d: Post-processing and data alignment;

[0071] Time alignment uses hardware timestamps or software synchronized logs to ensure that the three-modal images correspond to the same physical moment.

[0072] Spatial registration (image fusion) is the process of mapping infrared and hyperspectral images to the visible light image coordinate system through affine transformation, thin plate spline (TPS) or deep learning registration methods, and outputting a three-channel or multi-channel fused image (e.g., RGB + temperature + spectral features).

[0073] Data storage involves saving the original image and the registered data. It is recommended to use a common format (such as HDF5 and TIFF sequences) and include metadata (time, device parameters, and coordinate transformation matrix).

[0074] Example 4;

[0075] Based on Example 2, in step 2, multiple images of different modalities are acquired. For spatial-spectral joint registration, a reference image is used as a reference, and geometric transformations are applied to other images to achieve spatial registration. Since the spectral responses of different sensors are different, spectral matching or reconstruction is required to achieve spectral consistency registration.

[0076] Spatial registration, using the spatial registration loss function in image registration, measures the degree of alignment between the transformed registered image and the reference image in terms of spatial structure.

[0077] ;

[0078] in, For the first A single image depends on spatial transformation The spatial registration loss function outputs a scalar error value; It is the symbol for loss, representing the function used to measure prediction error during model training; superscript Indicates the first The loss corresponding to each image; the subscript spatial indicates the spatial dimension loss (used for images and 3D point clouds with spatial structure, focusing on the prediction error of spatial location / structure). The spatial dimension loss is for the first... Image To start from the image To reference image Spatial transformation functions (e.g., affine, non-rigid deformation fields);

[0079] For reference image In position The pixel value at that location (which could be a single-channel grayscale, RGB, or spectral vector); feature extraction operator, It is a feature extraction operator used to enhance image structural information and avoid the problem of poor robustness caused by directly comparing pixel values;

[0080] Position in coordinate system The inverse operation of the position transformation changes the position in the coordinate system. Pixels at that location are mapped back to the starting image. The position in the middle, To position in coordinates The pixels at that location are transformed by the inverse operation. Mapping to the starting image The pixel values ​​are then read and resampled.

[0081] The squared Euclidean distance (i.e., the L2 norm) measures the difference between two feature vectors. For the entire reference image region Integrate to obtain the average error of all pixels.

[0082] Spatial registration in this formula is used to measure the first... Image after After transformation, is it similar to the reference image? Align structurally; For feature extraction, i.e., extracting structural fingerprints, directly comparing pixel values ​​is easily affected by differences in lighting and sensors. Therefore, a method is introduced... To extract more robust local structural information; It calculates the squared Euclidean distance between two feature vectors; the smaller the value, the more similar the structures, measuring the structural consistency between two images at the same geographical location. (Integral) The global alignment quality is obtained by summing the entire reference image region (i.e., integrating in the case of continuous regions) and combining the structural matching degree at all locations.

[0083] Spectral registration consistency is measured using the spectral registration loss function to evaluate the performance of the first-order spectral registration. After spatial transformation and spectral mapping, the degree of matching between the spectral information of the image and the reference image is measured.

[0084] ;

[0085] in, For the first Amplitude images depend on spectral mapping Spectral registration loss function; It is the symbol for loss, representing the function used to measure prediction error during model training; superscript Indicates the first The loss corresponding to each image; the subscript spectral indicates the spectral loss (loss in image processing and signal processing). The spatial dimension loss depends on the spectral mapping. ; The spectral mapping function maps the starting image. The spectral response is converted to the reference spectral domain. The mapping below;

[0086] Position in coordinate system The inverse operation of the transformation; In spatial location The complete spectral vector extracted at the location (i.e., values ​​for all bands); It is the mapping of the spectrum at wavelength. Output value at; For reference image Position in the coordinate system Location and wavelength Pixel value at; It integrates the entire reference image region (spatial dimension) and summarizes the errors at all spatial locations; It integrates over the reference spectral band range (spectral dimension) and calculates the total error for each spatial point over the entire spectral range.

[0087] It is a scalar, representing the value calculated using the difference of squares at location. Location and wavelength The spectral differences at the location facilitate gradient optimization;

[0088] This formula utilizes a double integral of spatial and spectral data. When an image is spatially aligned, its spectrum is mapped through spectral mapping. When converting to the reference spectral domain, the degree of color / reflectance difference with the reference image at each spatial location and wavelength is calculated. After spatial alignment, the spectral information of each image is obtained through spectral mapping. Then, the spectrum can be close to that of the reference image, enabling the fusion of sensors in different bands and different spectral responses to obtain global spectral consistency.

[0089] To generate a multidimensional image cube with a unified coordinate system and pixel alignment, the following method is used:

[0090] Place all input 2D images or data layers in the same coordinate system to ensure that all images are in the same coordinate system, which facilitates accurate data overlay and analysis.

[0091] To achieve pixel alignment, the pixels in each image are aligned with the pixel positions in other images. The resolution and aspect ratio of different images are kept consistent, and there is no rotation or distortion between them. If the original image does not meet these requirements, preprocessing is required, such as resampling, scaling, and rotation, to achieve pixel-level alignment.

[0092] The generation of a multidimensional image cube involves transforming all images to the same coordinate system and aligning them pixel-wise, then stacking the multidimensional images to form a multidimensional image cube. In this cube, each layer represents a slice of the original dataset, while the entire cube contains spatial information as well as information in other dimensions, such as time and wavelength.

[0093] The specific working method of generating multidimensional image cubes is as follows:

[0094] Standardization preprocessing is performed on the original multi-source, multi-temporal, or multispectral image datasets, including basic operations such as image denoising, resolution unification, and radiometric correction, laying the foundation for subsequent coordinate unification.

[0095] Subsequently, a high-precision spatial registration algorithm is used to accurately transform all images into a pre-defined unified coordinate system, achieving pixel-level precise alignment. This ensures that pixels at corresponding spatial locations in different images can be accurately matched, avoiding deviations in subsequent multidimensional information fusion due to spatial offset. After standardization and pixel alignment of all images, the multidimensional images are stacked sequentially according to a preset dimensional order (e.g., time series sequence, wavelength from short to long, etc.), ultimately forming a structured multidimensional image cube.

[0096] In the constructed multidimensional image cube, each layer corresponds to a two-dimensional slice of the original dataset, clearly preserving the spatial two-dimensional information of the slice itself (such as the shape and position of the target).

[0097] The entire cube structure breaks through the dimensional limitations of traditional two-dimensional images, integrating multi-dimensional information: in addition to the horizontal distribution information of the spatial dimension, it also covers additional key information carried by the vertical stacking dimension. For example, the time dimension can record the dynamic changes of the same spatial area over time (such as vegetation growth cycle, disaster evolution trajectory), and the wavelength dimension can present the radiation characteristics of the target in different spectral bands (such as the spectral reflectance curve of ground objects, spectral response of biological tissues).

[0098] The multidimensional image cube enables the integrated presentation of multidimensional information, making subsequent cross-dimensional analysis, feature extraction, and pattern recognition possible. It provides an efficient data carrier for in-depth research in multiple fields, including remote sensing monitoring, medical image analysis, and materials characterization. Heterogeneous image data is represented by a structured, computable, and interpretable unified high-dimensional tensor, providing a reliable foundation for subsequent quantitative analysis, intelligent algorithms, and scientific decision-making. Through aligned or empowered paradigms, it achieves image information fusion and intelligent perception, enabling effective data analysis and visualization.

[0099] Example 5;

[0100] Based on Example 2, in step 3, the pre-trained multi-branch heterogeneous fusion neural network is a composite deep learning model that combines pre-training paradigm, multi-branch structure, heterogeneous network units and feature fusion mechanism. It uses modular heterogeneous branches to extract diverse features and utilizes the transfer learning capability of pre-training to improve the performance and generalization of the model in downstream tasks.

[0101] For each branch, its corresponding modality features are processed separately. Multi-branch feature extraction is used to process the features of each modality. High-level features are extracted through the encoder branches (CNN, Transformer, or hybrid architecture) corresponding to each modality:

[0102] ;

[0103] in, For the first The feature maps output by each branch (the final features obtained); For the first Feature extraction functions for each branch (e.g., a convolutional layer, a sub-network); For the first The modality, or the input to the first modality. The raw or intermediate data of each branch (e.g., input image, features of the previous layer);

[0104] For feature map dimensional shape; The height of the feature map; The width of the feature map; The depth of the feature map (i.e., the feature map depth when processing 3D data, such as video / point cloud features). The number of channels in the feature map (representing the dimension of the feature); The feature dataset in the feature map;

[0105] The purpose of multi-branch structures is to allow multiple branches (e.g., the residual branches of ResNet, the multi-scale branches of Inception) to be used simultaneously. Process the data, and finally convert all branches Fusion (e.g., splicing and addition) is used to extract richer features. Multiple different feature detectors simultaneously observe the input data, each branch capturing a local feature dimension of the data (e.g., texture, shape, and color), and these dimensions are then fused together.

[0106] For constructing cross-modal attention weights and dynamically weighting and fusing image information from different modalities in local regions, a query-key-based cross-modal attention approach is used, utilizing each spatial location. The importance weight of each modality to spatial location is calculated using a reference modality as the query and the other modalities as the key.

[0107] ;

[0108] in, For each spatial location ; In spatial location Above, the first Attention weights for each modality (i.e., the importance of that modality to the current position); The query vector, derived from the reference modality or shared encoder, represents the current spatial location. Feature representation; For the first The spatial location of each modality The key vector reflects the feature information of the mode at that position; The dimensions (number of channels) of the key vector and query vector are used to scale the dot product to prevent gradient vanishing. The total number of modes (e.g., T1, T2, and FLAIR). kind), For the first The spatial location of each modality The key vector (Key); This is an exponential function used to map similarity to a positive number interval.

[0109] The dot product similarity represents the query vector. With the Key vectors of each modality The inner product of the two measures the spatial similarity between them. A larger value indicates that the mode is more similar in spatial location. Related; For scaling, when When the value is very large, the dot product is prone to exploding (due to excessively large values), causing the softmax output to approach 0 or 1, resulting in gradient vanishing. It's for the purpose of stabilizing training.

[0110] For exponentialization, the similarity is converted into a positive real number, with larger values ​​for more similar modes; This involves summing the exponential values ​​of all modes to achieve Softmax normalization, ensuring the final result... This forms a probability distribution.

[0111] The molecule in the formula It is a query and the first Similarity calculation of keys for each modality. It is a scaling factor designed to mitigate the curse of dimensionality, through... By amplifying the differences, more relevant modal features can be highlighted.

[0112] The denominator in the formula is all modes (total) The summation of the exp values ​​of the similarity of each pair is finally obtained. It will be normalized to a probability value between 0 and 1, ensuring that the sum of the attention weights of all modalities is 1, and evenly distributing the contribution ratio of different modalities to the current query.

[0113] Cross-modal tasks (such as image and text retrieval and multimodal classification) require the fusion of information from different modalities (such as text and images). Weighting formulas enable the model to automatically focus on modal features that better match the current query, thereby efficiently integrating multimodal information into a unified representation. Enabling the model to intelligently select which modal information to prioritize is one of the core components for achieving modality alignment and feature fusion in cross-modal tasks.

[0114] Example 6;

[0115] Based on Example 2, in step 4, a pixel-level defect probability map is generated using a defect-aware context decoder based on the fused feature map, and geometric constraint optimization is performed on the probability map in combination with prior information of the circuit board design layout.

[0116] The fused feature map needs to extract multi-scale features from the input image and then fuse these features. This is achieved using a convolutional neural network (CNN), where feature maps at different levels represent information of different scales and levels of abstraction in the image.

[0117] Defect-aware contextual decoding utilizes the fused feature map and employs a contextual decoder (such as RNN, LSTM, or Transformer architecture) to capture broader contextual information. The decoder generates a probability value for each pixel belonging to a defect region, forming a defect probability map.

[0118] Specifically, the defect probability map is used to calculate the probability that a specified location is a defect. A softmax function or other normalization method is applied to the last layer of the decoder to obtain a probability value between 0 and 1. Considering a binary classification problem (i.e., whether it is a defect), the calculation method is as follows:

[0119] ;

[0120] in, Indicates input and The probability of identifying a defect when there are two samples or two feature vectors.

[0121] The score or logarithmic advantage of the defect category represents the degree of support for the defect; the higher the score, the stronger the support. The score for the non-defect category represents the degree of support for non-defects;

[0122] The index weights for defect classes, Yes and Evaluation scores that fall under the category of defects, such as parameter deviation values ​​or defect feature matching scores. It is an exponential function. It converts the defect assessment score into weights; the higher the score, the greater the weight, and vice versa. This strengthens the signal supporting the defect.

[0123] The index weights are for non-defect classes. Yes and The assessment score is for non-defective items; It is an exponential function. It transforms the non-defect assessment scores into weights; the higher the score, the greater the weight, and vice versa. This strengthens the signal strength supporting the non-defect assessment.

[0124] Probability of defects In this formula, the numerator is the defect weight, which is equivalent to using the weight of the defect to represent the probability of being a defect in the total weight. The greater the deviation of the defect from the standard, the easier it is to be judged as a defect. This non-linear strengthening can be reflected by an exponential function. The weighted intensity of the defect signal is reflected in the proportion of the total intensity of all signals (defect + qualified). The higher the proportion, the stronger the signal of the product to be tested that deviates from the standard, and the higher the physical probability of it being a defective product. The denominator is the defect weight + non-defect weight, which is normalized and is the sum of the two types of weights, defect weight and non-defect weight.

[0125] Probability of defects It involves adding the index weights of the two categories together, then dividing the numerator (the weight of the defect) by the sum, so that the result falls between 0 and 1, becoming the probability. This process transforms the difference in physical parameters (defective or qualified features) into digital signal strength, weighted signal (amplifying the difference), and probability (signal proportion). Essentially, it quantifies the possibility of physical features deviating from the standard.

[0126] Prior information is used to perform geometric constraint optimization on the probabilistic graph. This involves applying geometric constraints to the graph after obtaining the initial defect probability graph, combined with prior information provided by the circuit board design layout (e.g., known wire positions, component placement, etc.). During geometric constraint optimization, gradient descent or other optimization algorithms are used to minimize the energy function, thereby adjusting the defect probability graph to better conform to the actual circuit board design layout. This process iterates multiple times until a predetermined stopping condition is met, such as the energy change being less than a threshold or reaching the maximum number of iterations.

[0127] Geometric constraint optimization is achieved by defining an energy function, which is calculated by minimizing the penalty term (data fitting term) for inconsistencies with the known design layout and maximizing the reward term (regularization term) for conforming to the expected geometry. Specifically:

[0128] ;

[0129] in, Let be the objective function, representing the total energy that needs to be minimized; This is a data fitting term, and also a defect penalty term; These represent different geometric elements to be optimized, such as vertices of a 3D mesh and pixels of an image; To measure geometric elements The degree of deviation from the ideal state, for example: measuring the positional error of mesh vertices, the segmentation error of pixels; the larger the value, the more defects there are; data fitting terms. The summation is an accumulation of defects in all geometric elements to ensure overall data fit.

[0130] This is a control coefficient, ranging from 0 to 1, used to control the balance between data fitting and geometric constraints. The larger the size, the more emphasis is placed on geometric priors;

[0131] For the currently predicted geometry (such as the reconstructed 3D model); Geometric priors (e.g., known shape templates, smoothness of physical constraints, and statistical regularities); To measure the difference between predicted geometry and prior geometry (such as shape similarity and smoothness loss), and to ensure that the results conform to geometric common sense.

[0132] Potential energy representing data error is unstable energy accumulated due to deviation from observed data. For example, in 3D reconstruction, if the difference between the grid vertex position and the actual observation point is large, it is like a spring being pulled away from its equilibrium position, which will accumulate elastic potential energy. The larger the error, the higher this part of energy, and the more unstable the system is.

[0133] This represents the constraint energy accumulated due to violations of geometric priors (such as shape and smoothness). For example, the deformation potential energy of an elastic body in physics. If the reconstructed mesh is too distorted (deviating from the prior of smoothness), like a rubber being forcibly twisted, deformation potential energy will accumulate; the more severe the deviation, the higher this part of energy.

[0134] This approach balances the accuracy and rationality of a model by using data fitting terms and regularization terms, and is applied to computer vision processing tasks such as 3D reconstruction and image segmentation. Objective function The total energy balance allows the geometric optimization process to be viewed as a physical system. The total energy consists of the potential energy of data error and the constraint potential energy that deviates from the prior. The goal of optimization is to make the system move from a high-energy unstable state (large error and unreasonable shape) to a low-energy stable state (small error and conformity to geometric laws).

[0135] Example 7;

[0136] Based on Example 2, in step 5, after optimizing the geometric constraints of the prior information of the circuit board design layout, the defect probability map has extremely high accuracy. It not only eliminates false defect signals caused by image noise and registration deviation, but also clarifies the spatial relationship between defects and the key structures of circuit board pads, lines and components through geometric position calibration of the design layout.

[0137] The core of this embodiment is to extract and output three key defect information from the optimized probability map:

[0138] Defect type determination: Based on the multi-dimensional feature response of the defect region in the fused feature map (such as morphological features in the visible light mode, abnormal thermal radiation features in the infrared mode, and material composition features in the hyperspectral mode), combined with preset defect type determination rules (such as short circuit defects manifested as high grayscale response of abnormal connection between lines, open circuit defects manifested as low grayscale response of line interruption area, and cold solder joint defects manifested as uneven infrared thermal radiation and abnormal hyperspectral material features in the pad area), accurate classification of circuit board defects such as short circuit, open circuit, cold solder joint, missing solder joint, and component misalignment is achieved.

[0139] Defect location: By leveraging the precise alignment between the optimized probability map and the circuit board design layout, the pixel coordinates of the defect area are converted into the physical coordinates of the circuit board (e.g., a millimeter-level coordinate system with the upper left corner of the circuit board as the origin). At the same time, the relative position description of the defect with respect to surrounding key components (e.g., resistors, capacitors, and chip pins) or circuit networks is output, providing a precise location basis for subsequent repair and re-inspection.

[0140] Defect confidence assessment: Confidence reflects the reliability of defect identification results. It is calculated based on the probability distribution of the defect area—the mean and maximum probability values ​​of pixels within the defect area, as well as the probability difference between the defect area and the surrounding normal area. Combined with the consistency of multi-dimensional modal features (e.g., defects that detect anomalies in all three modalities have higher confidence than defects that detect anomalies in a single modality), the confidence value is output in the range of 0-1 (the closer the value is to 1, the more reliable the identification result). This facilitates prioritizing high-confidence defects and improving detection efficiency.

[0141] Regarding the construction of an interpretable defect fingerprint database, traditional defect identification methods can only output defect results without explaining why a defect was identified as that type, and are difficult to support subsequent defect tracing. This embodiment constructs an interpretable defect fingerprint database through the structured extraction and storage of multidimensional defect response features, ensuring that each defect possesses a unique feature fingerprint, thus achieving interpretability and traceability of defect identification results. The specific construction process is as follows:

[0142] Structured extraction of multidimensional response features of defects: For each identified defect, the core feature parameters under three imaging modalities are extracted to form a structured defect fingerprint. The extracted features include: ① Visible light modality: area, perimeter, morphological complexity, and edge gradient value of the defect region; ② Infrared modality: average temperature, temperature gradient, and duration of thermal radiation anomalies in the defect region (if it is a dynamic detection scenario); ③ Hyperspectral modality: characteristic spectral peaks and spectral curve similarity of the defect region (deviation from the spectrum of normal materials); ④ Geometric correlation features: offset of the defect from the standard structure in the design layout, and the network number of the line where the defect is located.

[0143] Defect fingerprint classification, storage, and index construction: Extracted structured defect fingerprints are classified and stored according to defect type (e.g., short circuit and open circuit), circuit board model, and production batch. A multi-dimensional index is built for each defect fingerprint (e.g., by defect type + confidence level, by production batch + defect location). Furthermore, each defect fingerprint is associated with corresponding original multi-dimensional image data, an optimized probability map, and core defect information (type, location, and confidence level), ensuring rapid retrieval of complete detection data during subsequent traceability, achieving end-to-end traceability from defect fingerprint to raw data to identification results.

[0144] Applications of the defect fingerprint database: defect tracing and process feedback;

[0145] The core value of building an interpretable defect fingerprint database lies in supporting subsequent defect tracing and production process optimization, forming a closed-loop management system of "detection-identification-tracing-optimization".

[0146] Defect Root Cause Analysis: When a batch of circuit boards exhibits concentrated defects, a defect fingerprint database can be consulted to compare and analyze the commonalities in multidimensional feature fingerprints of defects of the same batch and type (e.g., a batch of cold solder joint defects all exhibit uneven infrared thermal radiation in the solder pad area and abnormal hyperspectral solder paste composition characteristics). Combined with production process parameters (e.g., soldering temperature, soldering time, solder paste application amount), the root cause of the defect can be identified (e.g., low soldering temperature leading to incomplete solder paste melting). Furthermore, by comparing defect fingerprints with historical inspection data, it is also possible to trace whether persistent defect problems stem from the same process step.

[0147] Process Feedback: The root cause analysis results obtained from defect tracing are fed back to the circuit board manufacturing process to optimize process parameters in a targeted manner (e.g., adjusting soldering temperature, optimizing solder paste application process, and improving component placement accuracy). Furthermore, as the defect fingerprint database accumulates, big data analysis can be used to uncover the correlation between defect types and process parameters (e.g., the changing trend of the incidence of cold solder joints within a certain soldering temperature range), providing data support for intelligent optimization of the production process and reducing the defect rate at its root.

[0148] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope described in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A circuit board defect identification method based on multidimensional image data, characterized in that, Includes the following steps: Acquire synchronous image data of the circuit board under test in multiple imaging modalities; Spatial-spectral joint registration is performed on images of each modality to generate a multidimensional image cube with a unified coordinate system and pixel alignment; The multidimensional image cube is input into a pre-trained multi-branch heterogeneous fusion neural network; wherein each branch processes the features of its corresponding modality and dynamically weights and fuses the image information of different modalities in the local region through a cross-modal attention mechanism; Based on the fused feature map, a pixel-level defect probability map is generated using a defect-aware context decoder, and geometric constraint optimization is performed on the probability map by combining prior information of the circuit board design layout. Regarding defect-aware contextual decoding, it utilizes the fused feature map and employs a contextual decoder to capture broader contextual information. The decoder generates a probability value for each pixel belonging to a defect region, forming a defect probability map. The calculation method is as follows: ； in, Indicates input and Given two samples or two feature vectors, the probability of classifying it as a defect; The score or logarithmic advantage of the defect category represents the degree of support for the defect; The score for the non-defect category represents the degree of support for non-defects; The index weights for defect classes, Yes and The assessment score for defects is either the parameter deviation value or the defect feature matching score. It is an exponential function. It converts the defect assessment score into a weight. The higher the score, the greater the weight will be, and the lower the score, the greater the weight will be, thus strengthening the signal strength supporting the defect. The index weights are for non-defect classes. Yes and The assessment score is for non-defective items; It is an exponential function. It transforms the non-defect assessment score into a weight. The higher the score, the greater the weight will be, and the lower the score, the greater the weight will be, thus strengthening the signal strength supporting the non-defect. Geometric constraint optimization is achieved by defining an energy function, which combines minimizing the penalty or data fitting term for inconsistencies with the known design layout with maximizing the reward term for conforming to the expected geometry. Specifically: ； in, Let be the objective function, representing the total energy that needs to be minimized; This is a data fitting term, also known as a defect penalty term, representing the potential energy of the data error, the unstable energy accumulated due to deviation from the observed data; These represent different geometric elements that need to be optimized; To measure geometric elements The degree of deviation from the ideal state; data fitting term The summation is an accumulation of defects in all geometric elements to ensure overall data fit; This is a control coefficient, with a value ranging from 0 to 1, used to control the balance between data fitting and geometric constraints; The current predicted geometry; The geometric priors are known shape templates, the smoothness of physical constraints, and statistical regularities; To measure the difference between predicted geometry and prior geometry, and to ensure that the results conform to geometric common sense; This represents the constraint energy accumulated due to violations of geometric priors; Based on the optimized defect probability map, the defect type, location, and confidence level are output. An interpretable defect fingerprint database is established based on the multidimensional response characteristics of the defects for subsequent defect tracing and process feedback.

2. The circuit board defect identification method based on multidimensional image data according to claim 1, characterized in that, For simultaneous image data acquisition across multiple imaging modalities, the following steps are employed: Synchronous triggering is initiated by sending a unified trigger signal through the synchronization controller, causing the three imaging devices to start exposure / acquisition at the same time. For pushbroom hyperspectral cameras, a precision displacement platform needs to move at a constant speed while simultaneously triggering line scanning. Image acquisition records RGB or grayscale images using visible light to reflect solder joints, traces, and component appearance; infrared thermal imaging records temperature distribution maps to identify overheated areas or cold solder joints. Post-processing and data alignment employ temporal alignment and spatial registration, which maps infrared and hyperspectral images to the visible light image coordinate system through affine transformation, thin plate spline, or deep learning registration methods, outputting a three-channel or multi-channel fused image.

3. The circuit board defect identification method based on multidimensional image data according to claim 1, characterized in that, To acquire multiple images with different modalities, spatial-spectral joint registration is achieved by applying geometric transformations to other images based on a reference image. Since different sensors have different spectral responses, spectral matching or reconstruction is required to achieve spectrally consistent registration. Spatial registration, using the spatial registration loss function in image registration, measures the degree of alignment between the transformed registered image and the reference image in terms of spatial structure. ； in, For the first A single image depends on spatial transformation The spatial registration loss function outputs a scalar error value; The superscript is the symbol for loss, representing the function used to measure prediction error during model training. Indicates the first The loss corresponding to each image; the subscript spatial indicates the loss in the spatial dimension; The spatial dimension loss is for the first... Image To start from the image To reference image Spatial transformation function; For reference image In position Pixel value at; It is a feature extraction operator used to enhance image structural information and avoid the problem of poor robustness caused by directly comparing pixel values; Position in coordinate system The inverse operation of the position transformation changes the position in the coordinate system. The pixels at that location are mapped back to the starting image. The position in the middle, To position in coordinates The pixels at that location are transformed by the inverse operation. Mapping to the starting image The pixel values ​​are then read and resampled. The squared Euclidean distance measures the difference between two feature vectors. For the entire reference image region Integrate to obtain the average error of all pixels.

4. The circuit board defect identification method based on multidimensional image data according to claim 3, characterized in that, The spectral consistency registration uses a spectral registration loss function to measure the degree of matching between the spectral information of an image and a reference image after spatial transformation and spectral mapping. ； in, For the first Amplitude images depend on spectral mapping Spectral registration loss function; It is the symbol for loss, representing the function used to measure prediction error during model training; superscript Indicates the first The loss corresponding to each image; the subscript spectral indicates the spectral loss, which is the loss in image processing and signal processing. The spatial dimension loss depends on the spectral mapping. ; The spectral mapping function maps the starting image. The spectral response is converted to the reference spectral domain. The mapping below; Position in coordinate system The inverse operation of the transformation; In spatial location The complete spectral vector extracted at the location; It is the mapping of the spectrum at wavelength. Output value at; For reference image Position in the coordinate system Location and wavelength Pixel value at; It is the integral or spatial dimension of the entire reference image region, summarizing the error at all spatial locations; It is an integral over the reference spectral band range or spectral dimension, and for each spatial point, it calculates the total error over the entire spectral range. It is a scalar, representing the value calculated using the difference of squares at location. Location and wavelength The spectral differences at the location facilitate gradient optimization.

5. The circuit board defect identification method based on multidimensional image data according to claim 3, characterized in that, To generate the multidimensional image cube with a unified coordinate system and pixel alignment, the following steps are taken: Standardize and preprocess the original multi-source, multi-temporal, or multispectral image datasets; Then, a high-precision spatial registration algorithm is used to accurately transform all images into a pre-defined unified coordinate system, achieving pixel-level precise alignment. In the constructed multidimensional image cube, each layer corresponds to a two-dimensional slice of the original dataset, clearly preserving the spatial two-dimensional information of the slice itself.

6. The circuit board defect identification method based on multidimensional image data according to claim 1, characterized in that, For each branch, its corresponding modality features are processed separately. Multi-branch feature extraction is used to process the features of each modality. High-level features are extracted through the encoder branches corresponding to each modality: ； in, For the first Feature maps output by each branch; For the first Feature extraction function for each branch; For the first The modality, or the input to the first modality. The original or intermediate data of each branch; For feature map dimensional shape; The height of the feature map; The width of the feature map; The depth of the feature map; The number of channels in the feature map; The feature dataset in the feature map; The purpose of multi-branching is to allow multiple branches to be used simultaneously in a multi-branch structure. Process the data, and finally convert all branches By fusing these features to extract richer characteristics, multiple different feature detectors are used to observe the input data simultaneously. Each branch captures a local feature dimension of the data, and finally these dimensions are fused together. For constructing cross-modal attention weights and dynamically weighting and fusing image information from different modalities in local regions, a query-key-based cross-modal attention approach is used, utilizing each spatial location. The importance weight of each modality to spatial location is calculated using a reference modality as the query and the other modalities as the key. ； in, For each spatial location ; In spatial location Above, the first Attention weights for each modality; The query vector, or Query, comes from the reference modality or shared encoder and represents the current spatial location. Feature representation; For the first The spatial location of each modality The key vector or key reflects the feature information of the modality at that position; The dimension or number of channels of the key vector and query vector, used to scale the dot product to prevent gradient vanishing; The total number of modes, For the first The spatial location of each modality The key vector; This is an exponential function used to map similarity to a positive number interval; The dot product similarity represents the query vector. With the Key vectors of each mode The inner product of the two measures their spatial similarity. For exponentialization, the similarity is converted into a positive real number, with larger values ​​for more similar modes; This involves summing the exponential values ​​of all modes to achieve Softmax normalization, ensuring the final result... This constitutes a probability distribution; The molecule in the formula Is query and the first Similarity calculation of keys in each modality It is a scaling factor designed to mitigate the curse of dimensionality, through... By amplifying the differences, more relevant modal features can be highlighted.

7. The circuit board defect identification method based on multidimensional image data according to claim 1, characterized in that, The defect probability map is used to calculate the probability that a specified location is a defect. A softmax function is applied to the last layer of the decoder to obtain a probability value between 0 and 1. This process transforms defective or qualified features into digital signal strength, weighted signals, and probabilities, quantifying the possibility that physical features deviate from the standard.

8. The circuit board defect identification method based on multidimensional image data according to claim 1, characterized in that, The construction of the interpretable defect fingerprint database involves the structured extraction and storage of multidimensional response features of defects to create a database where each defect possesses a unique feature fingerprint. The specific construction process is as follows: Structured extraction of multidimensional response features of defects; for each identified defect, the core feature parameters under the three imaging modalities are extracted to form a structured defect fingerprint; Defect fingerprint classification, storage, and index construction: The extracted structured defect fingerprints are classified and stored according to defect type, circuit board model, and production batch, and a multidimensional index is built for each defect fingerprint. Each defect fingerprint is also associated with the original multidimensional image data, the optimized probability map, and the core information of the defect, ensuring that complete detection data can be quickly retrieved during subsequent tracing, realizing full traceability from defect fingerprint to original data to recognition results.

9. A circuit board defect identification system based on multi-dimensional image data, used to execute the circuit board defect identification method based on multi-dimensional image data according to any one of claims 1-8, characterized in that, include: The multi-dimensional image data acquisition module is used to acquire synchronous image data of the circuit board under inspection in multiple imaging modalities; The image registration module is used to perform spatial-spectral joint registration on images of various modalities to generate a multidimensional image cube with a unified coordinate system and pixel alignment. The multimodal feature fusion module has a built-in pre-trained multi-branch heterogeneous fusion neural network. Each branch of the neural network processes the features of the corresponding modality and dynamically weights and fuses the image information of different modalities in the local area through a cross-modal attention mechanism. The defect probability map generation and optimization module is used to generate a pixel-level defect probability map based on the fused feature map output by the multimodal feature fusion module, using a defect-aware context decoder, and to perform geometric constraint optimization on the defect probability map by combining prior information of the circuit board design layout. The defect identification and tracing module is used to output the defect type, location and confidence level based on the optimized defect probability map, and to establish an interpretable defect fingerprint database based on the multidimensional response characteristics of the defect. The defect fingerprint database is used for subsequent defect tracing and process feedback.

Images