Printed circuit board microdefect detection method and system based on image recognition

Abstract

The invention discloses a printed circuit board microdefect detection method and system based on image recognition, and relates to the technical field of electronic component detection. The method specifically comprises the following steps: acquiring a multispectral image sequence on the surface of a to-be-detected component, and generating a preliminary defect thermodynamic diagram containing a spatial position and an abnormal degree; establishing a form rule base according to a component physical structure and a material attribute, and defining a defect typical geometric form and a spectral response mode; performing class-by-class matching on the thermodynamic diagram and the rule base, performing geometric constraint verification, and filtering abnormal regions to generate a refined list; extracting a multi-spectral reflection curve of the refining area, and calculating the goodness of fit by combining a rule base spectral mode; and confirming classification according to the goodness of fit and the abnormal degree of the thermodynamic diagram, and outputting a report containing defect types, coordinates and severity levels. Abnormality is quantified and positioned through the multispectral thermodynamic diagram, dual verification is combined with the form rule base, and the microdefect detection precision and reliability are improved.

Description

Technical Field

[0001] This application relates to the field of electronic component inspection technology, and in particular to a method and system for detecting micro-defects in printed circuit boards based on image recognition. Background Technology

[0002] In the field of printed circuit board (PCB) manufacturing and inspection, defect detection is a core aspect of ensuring product quality. Current technologies primarily acquire images of the PCB surface through optical imaging and utilize methods based on grayscale thresholding, edge detection, or general machine learning models to identify defects such as scratches, stains, and poor solder joints. However, these methods have significant limitations: First, at the defect perception and characterization level, they generally rely on single-spectrum or limited multispectral images, resulting in insufficient information dimensions and difficulty in generating comprehensive characterization maps that simultaneously and accurately reflect the spatial location of defects and quantify the degree of anomaly, leading to a coarse assessment of defect severity. Second, at the defect discrimination and verification level, they often employ general image features or data-driven models that are disconnected from the physical characteristics of the PCB, failing to incorporate prior knowledge rules regarding PCB materials, structure, and typical defect patterns. Consequently, in complex backgrounds, uneven lighting, or the presence of texture interference, it is difficult to accurately distinguish between genuine defects and pseudo-anomalies with similar appearances, ultimately leading to a high rate of missed detections and false detections in the inspection system. Summary of the Invention

[0003] The purpose of this application is to address the shortcomings of existing technologies by proposing an image recognition-based method and system for detecting micro-defects in printed circuit boards.

[0004] To achieve the above objectives, this application adopts the following technical solution: a printed circuit board micro-defect detection method based on image recognition, comprising: A surface multispectral image sequence of the electronic component to be inspected is acquired, and a preliminary defect thermal map containing spatial location and degree of anomaly is generated based on the surface multispectral image sequence. Based on the known physical structure and material properties of the electronic component to be tested, a morphological rule library corresponding to the defect type is established; The preliminary defect heatmap is matched with the morphology rule base by class and geometric constraint verification to filter out abnormal areas that do not conform to any type of defect morphology rule, and a refined defect area list is generated. For each defect region in the refined defect region list, the corresponding multispectral reflectance curve is extracted from the multichannel fused image, and the spectral fit is calculated by combining the spectral response mode of the corresponding type of defect in the morphological rule library. Based on the calculation results of spectral fit and the degree of anomaly of the corresponding area in the preliminary defect heat map, each defect in the list of refining defect areas is finally confirmed and classified, and a test report containing defect type, coordinates and quantified severity level is output.

[0005] As a further aspect of this application, generating a preliminary defect thermal map containing spatial location and anomaly degree based on the surface multispectral image sequence includes: Grayscale images of the surface of the electronic component under test are acquired under multiple monochromatic light sources of different wavelengths to form a multispectral image sequence of the surface. The surface multispectral image sequence is registered in the same region and aligned with pixels to synthesize a multi-channel fused image of the electronic component to be detected. Each pixel in the multi-channel fused image contains reflection intensity information from multiple wavelength light sources. A template image matching the model of the electronic component to be detected is retrieved from the standard component database, and the multi-channel fused image and the template image are normalized in spatial scale and matched with reference feature points. Based on the matching correspondence, calculate the element values ​​of the pixel intensity difference distribution map and the corresponding element values ​​of the texture structure similarity matrix of each local image block in the multi-channel fused image and the corresponding region in the template image; A hierarchical defect feature extraction network is constructed, which includes a shallow edge response layer and a deep semantic abstraction layer. The element values ​​of the pixel intensity difference distribution map and the corresponding element values ​​of the texture structure similarity matrix are input into the hierarchical defect feature extraction network in parallel for multi-scale feature extraction. By fusing the edge discontinuity feature map output by the shallow edge response layer with the abnormal pattern feature map output by the deep semantic abstraction layer, a preliminary defect heatmap containing spatial location and degree of abnormality is generated.

[0006] As a further aspect of this application, the step of performing same-region registration and pixel alignment operations on the surface multispectral image sequence to synthesize a multi-channel fused image of the electronic component to be detected includes: The image acquired under the center wavelength light source in the surface multispectral image sequence is selected as the spatial reference image; For each grayscale image acquired under a non-central wavelength light source in the sequence, an image registration algorithm based on feature point detection and affine transformation model is used to calculate the coordinate transformation parameters between each grayscale image acquired under a non-central wavelength light source in the surface multispectral image sequence and the spatial reference image. The coordinate transformation parameters are used to perform spatial transformation on each grayscale image under a non-central wavelength light source, so that the pixels in each grayscale image under a non-central wavelength light source are aligned one by one with the pixels in the spatial reference image in spatial position. In order of wavelength, grayscale images of all wavelength light sources that have completed pixel alignment are stacked to construct a multi-channel fused image.

[0007] As a further aspect of this application, the calculation of the element values ​​of the pixel intensity difference distribution map between each local image patch in the multi-channel fused image and the corresponding region in the template image, and the corresponding element values ​​of the texture structure similarity matrix, includes: The normalized multi-channel fused image and the template image are evenly divided into multiple local image blocks of the same size that do not overlap. For each pair of local image blocks corresponding to spatial locations, the absolute intensity difference in different spectral channels is calculated pixel by pixel as the spectral channel difference, and the average value of multiple spectral channel differences is taken as the comprehensive intensity difference value of the pixel. The combined intensity difference value of all pixels within each local image block is statistically analyzed, and the mean of all combined intensity difference values ​​is calculated as the element value of the pixel intensity difference distribution map of the local image block region. Meanwhile, for each pair of local image patches, the correlation coefficient and contrast difference on the gray-level co-occurrence matrix features are calculated, and the combined measure of the correlation coefficient and contrast difference is used as the corresponding element value of the texture structure similarity matrix.

[0008] As a further aspect of this application, the hierarchical defect feature extraction network is constructed, comprising a shallow edge response layer and a deep semantic abstraction layer. The element values ​​of the pixel intensity difference distribution map and the corresponding element values ​​of the texture structure similarity matrix are input in parallel into the hierarchical defect feature extraction network for multi-scale feature extraction, including: The shallow edge response layer is composed of multiple edge detection operators of different scales connected in parallel, including the Sobel operator and the Laplacian operator; The element values ​​of the pixel intensity difference distribution map are input to the shallow edge response layer, and each edge detection operator performs convolution operation independently to extract edge gradient and direction information at different scales; The edge gradient and direction information output by each edge detection operator are weighted and fused to generate the edge discontinuity feature map that emphasizes the discontinuity of edges in the difference region. The corresponding element values ​​of the texture structure similarity matrix undergo multiple nonlinear transformations and feature dimensionality reduction in the deep semantic abstraction layer, gradually abstracting high-dimensional feature expressions associated with typical defect patterns. The last layer of the deep semantic abstraction layer outputs a feature map, namely the abnormal pattern feature map, from the high-dimensional feature representation.

[0009] As a further aspect of this application, the step of performing class-by-class pattern matching and geometric constraint verification between the preliminary defect heatmap and the morphological rule library, filtering out abnormal regions that do not conform to any type of defect morphological rule, and generating a refined defect region list includes: The typical geometric shape description of each type of defect is read one by one from the morphological rule library. The typical geometric shape description includes aspect ratio range, area threshold, boundary curvature features and principal axis direction. On the preliminary defect heatmap, the contours of all potential defect regions with abnormal intensity exceeding the threshold are extracted using a connected component analysis algorithm; For each extracted potential defect region contour, its actual geometric morphological parameters are calculated, and the actual geometric morphological parameters are compared one by one with the typical geometric morphological description of each type of defect in the morphological rule library. Only potential defect regions whose actual geometric morphology parameters match the typical geometric morphology description of the defect more than a set threshold are retained, and their contour information and the matching defect type assumptions are recorded in the refined defect region list.

[0010] As a further aspect of this application, the step of extracting the corresponding multispectral reflectance curve from the multichannel fused image for each defect region in the refined defect region list includes: For each entry in the list of refining defect areas, the pixel position of the refining defect area in the multi-channel fused image is located based on the contour information recorded for each entry. Acquire all pixel data of the multi-channel fused image within the contour region, where each pixel contains the reflection intensity value under all acquired wavelength light sources; For the refined defect area, calculate the regional average value of the reflection intensity of all pixels under each specific wavelength light source; The calculated regional average values ​​under multiple wavelength light sources are connected sequentially in order of wavelength from shortest to longest, and plotted as a curve with wavelength as the horizontal axis and average reflection intensity as the vertical axis. This curve is the multispectral reflection curve of the refining defect area.

[0011] As a further aspect of this application, the calculation of spectral fit based on the spectral response patterns of corresponding types of defects in the morphological rule library includes: Obtain the standard spectral response curve corresponding to the assumed defect type of the current refining defect region from the morphological rule library; Align the sampling points of the multispectral reflectance curve of the current defect region with the standard spectral response curve of the same type. Calculate the difference in reflection intensity between all corresponding sampling points on the two curves, and obtain the root mean square error of the difference in reflection intensity. The root mean square error is input into a preset conversion function to obtain the spectral fit.

[0012] As a further aspect of this application, the final confirmation and classification labeling of each defect in the refining defect region list based on the calculation results of the spectral fit and the degree of anomaly of the corresponding region in the preliminary defect thermal map includes: An anomaly level confirmation threshold is preset, and a spectral fit confirmation threshold is set for each defect type; For each entry in the list of refining defect areas, determine whether the calculated spectral fit reaches the confirmation threshold of the spectral fit. At the same time, the average value of the abnormality of the refining defect area is read from the preliminary defect heat map to determine whether it reaches the abnormality confirmation threshold of the assumed defect type. Only when the average value of the spectral fit and the degree of anomaly simultaneously reaches or exceeds their respective confirmation thresholds is the refining defect region finally confirmed as the hypothetical defect type, and it is classified and labeled as the hypothetical defect type. If the average value of the spectral consistency and the degree of anomaly fails to simultaneously reach or exceed their respective confirmation thresholds, the entry will be removed from the list of refining defect areas or marked as a suspected defect to be reviewed.

[0013] As a further aspect of this application, this application also includes an image recognition-based printed circuit board micro-defect detection system, the system including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements the steps of the image recognition-based printed circuit board micro-defect detection method described above.

[0014] Compared with the prior art, the advantages and positive effects of this application are as follows: A preliminary defect thermal map, incorporating spatial location and anomaly severity, is generated based on multispectral image sequences. By acquiring multispectral image sequences of the component surface, the response intensities of anomalous pixels in each band are normalized and weighted, and a comprehensive anomaly score is calculated for each pixel, forming a thermal map indexed by spatial coordinates and characterized by anomaly severity. Compared to conventional single-spectral imaging, which merely marks the presence or absence of defects, multispectral fusion captures the differences in reflectance of different materials across different bands, quantifies anomaly severity to avoid subjective threshold settings, and allows spatial coordinate localization to directly correlate with the component's physical location in subsequent analyses. This improves the comprehensiveness and accuracy of the initial screening, reducing missed detections due to limited spectral information.

[0015] A morphological rule library corresponding to defect types is established, and defects are verified through a dual approach of geometric constraint validation and spectral fit calculation. Based on the known physical structure and material properties of electronic components, typical geometric morphological parameters and spectral response patterns for defects such as scratches and stains are predefined. First, the abnormal areas in the preliminary heatmap are matched with the geometric morphology in the rule library to filter out mismatched areas. Then, the multispectral reflectance curves of the retained areas are extracted and their fit is calculated with the corresponding standard spectral patterns of the defects. Compared to conventional verification relying on general features or black-box models, explicit rules utilize prior knowledge to clarify the essential characteristics of defects, geometric constraints eliminate false anomalies, and spectral fit verification combines material spectral characteristics. This dual mechanism significantly reduces the false positive rate and improves the accuracy and interpretability of classification and labeling. Attached Figure Description

[0016] Figure 1 This is a flowchart of the image recognition-based micro-defect detection method for printed circuit boards described in this application; Figure 2 Flowchart for generating multi-channel fused images; Figure 3 Multispectral reflectance intensity curves of different defect types and normal regions at different wavelengths; Figure 4 Geometric parameter constraint diagrams defined in the morphological rule library for typical surface defects of electronic components; Figure 5 This is a comparison chart of the multispectral reflectance curves of scratch defects. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] In the description of this application, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, in the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0019] See Figure 1The process involves acquiring a sequence of multispectral images of the surface of the electronic component under test, and generating a preliminary defect thermal map containing spatial location and anomaly severity based on this sequence. According to the known physical structure and material properties of the electronic component, a morphological rule library corresponding to the defect type is established. This rule library defines the typical geometric shapes and spectral response patterns of defects such as scratches, stains, poor solder joints, and coating defects. Next, the preliminary defect thermal map and the morphological rule library are matched and geometrically constrained, filtering out abnormal regions that do not conform to any type of defect morphological rule, generating a refined defect region list. For each defect region in the refined defect region list, the corresponding multispectral reflectance curve is extracted from the multichannel fused image, and spectral consistency is calculated by combining it with the spectral response pattern of the corresponding type of defect in the morphological rule library. Based on the spectral consistency calculation results and the anomaly severity of the corresponding region in the preliminary defect thermal map, each defect in the refined defect region list is finally confirmed and classified, and a test report containing the defect type, coordinates, and quantified severity level is output.

[0020] In one embodiment of this application, the surface multispectral image sequence includes grayscale images of the same printed circuit board surface acquired under multiple monochromatic light sources of different wavelengths. For example, three grayscale images are acquired under light sources with wavelengths of 450 nm, 550 nm, and 650 nm, respectively. These images capture the reflection response of the printed circuit board surface to different wavelengths of light at the same spatial location. Data comparison shows that the 450 nm image highlights the reflection characteristics of metal solder joints, while the 650 nm image is more likely to penetrate the coating and reveal the characteristics of the underlying material. The surface multispectral image sequence is subjected to same-region registration and pixel alignment operations to synthesize a multi-channel fused image of the printed circuit board. Each pixel in the multi-channel fused image contains reflection intensity information from multiple wavelength light sources, so that each spatial location has multi-dimensional spectral data to enhance the distinction between defects and background.

[0021] In some embodiments, a template image matching the printed circuit board model is retrieved from a standard component database. The template image represents the imaging result of a defect-free standard under the same multispectral conditions. The multichannel fused image and the template image are then normalized in spatial scale and matched with reference feature points, such as corner marks on the printed circuit board or the center point of a specific component. Based on the matched correspondence, the element values ​​of the pixel intensity difference distribution map and the corresponding region in the template image for each local image patch in the multichannel fused image are calculated, along with the corresponding element values ​​of the texture structure similarity matrix. The size of the local image patch is set to 8 pixels × 8 pixels. The element values ​​of the pixel intensity difference distribution map reflect the average intensity deviation of the local region, while the element values ​​of the texture structure similarity matrix characterize the degree of local texture variation.

[0022] In the specific implementation, a hierarchical defect feature extraction network is constructed, which includes a shallow edge response layer and a deep semantic abstraction layer. The element values ​​of the pixel intensity difference distribution map and the corresponding element values ​​of the texture structure similarity matrix are input in parallel into the hierarchical defect feature extraction network for multi-scale feature extraction. The shallow edge response layer consists of multiple edge detection operators of different scales connected in parallel, including the Sobel operator and the Laplacian operator. The Sobel operator uses a 3×3 convolution kernel to extract horizontal and vertical gradients, while the Laplacian operator uses a 5×5 convolution kernel to detect changes in the second derivative. The element values ​​of the pixel intensity difference distribution map are input into the shallow edge response layer, where each edge detection operator independently performs convolution operations to extract edge gradient and direction information at different scales. The outputs of each edge detection operator are weighted and fused to generate an edge discontinuity feature map. The weighted fusion formula is expressed as:

[0023] in: This represents the edge discontinuity feature map after fusion. This represents the total number of edge detection operators. Indicates the first The weight coefficients output by each edge detection operator. Indicates the first The feature map output by each edge detection operator, weight coefficients Edge detection sensitivity is optimized by pre-determining network training.

[0024] It can be understood that the deep semantic abstraction layer consists of multiple cascaded convolutional and pooling layers. The corresponding element values ​​of the texture structure similarity matrix are input into the deep semantic abstraction layer. These element values ​​undergo multiple nonlinear transformations and feature dimensionality reduction within the deep semantic abstraction layer to gradually abstract high-dimensional feature representations associated with typical defect patterns. The final layer of the deep semantic abstraction layer outputs the high-dimensional feature representation as a feature map, namely an anomaly pattern feature map. This anomaly pattern feature map encodes the semantic-level differences between texture anomaly regions and normal regions. The edge discontinuity feature map output from the shallow edge response layer and the anomaly pattern feature map output from the deep semantic abstraction layer are fused to generate a preliminary defect heatmap containing spatial location and anomaly degree. The fusion operation is achieved through element-wise addition. The edge discontinuity feature map emphasizes the discontinuity of defect boundaries, while the anomaly pattern feature map highlights internal texture anomalies. Data comparison shows that the fused heatmap exhibits both high edge response and high semantic anomaly values ​​in the scratch defect region.

[0025] See Figure 2In one embodiment of this application, data comparison shows that unregistered image sequences acquired using light sources with wavelengths of 450 nm, 550 nm, and 650 nm exhibit translational shifts of up to 5 pixels, while the shift is reduced to less than 0.5 pixels after registration. Images acquired under a central wavelength light source in the surface multispectral image sequence are selected as spatial reference images, for example, images acquired under a 550 nm green light source. For each grayscale image acquired under a non-central wavelength light source in the sequence, an image registration algorithm based on feature point detection and affine transformation is used to calculate the coordinate transformation parameters between each grayscale image acquired under a non-central wavelength light source and the spatial reference image. Feature point detection uses the SIFT algorithm to identify pad corners and silkscreen markings. The coordinate transformation parameters are used to perform a spatial transformation on each grayscale image under a non-central wavelength light source, aligning the pixels in each grayscale image under a non-central wavelength light source with the pixels in the spatial reference image one by one in spatial position. The spatial transformation includes the application of an affine transformation matrix to achieve rotation, scaling, and translation compensation. Grayscale images from different wavelength light sources, after pixel alignment, are stacked in order of their corresponding wavelengths to construct a multi-channel fused image with multiple spectral channels. The multi-channel fused image is a three-dimensional data array with dimensions of 1024 pixels multiplied by 1024 pixels multiplied by 3 spectral channels. In some embodiments, the normalized multi-channel fused image and the template image are uniformly divided into multiple non-overlapping local image blocks of the same size. The size of each local image block is fixed at 16 pixels multiplied by 16 pixels, thus dividing the entire image into 4096 local image blocks. For each pair of spatially corresponding local image blocks, the absolute intensity difference in different spectral channels is calculated pixel-by-pixel as the spectral channel difference, and the average of multiple spectral channel differences is taken as the comprehensive intensity difference value of the pixel. The formula for calculating the comprehensive intensity difference value is expressed as:

[0026] in: Represents position coordinates The overall strength difference value at the location, Indicates the total number of spectral channels. Indicates the multi-channel fused image at the 1st... Position coordinates on each spectral channel Pixel intensity value at that location, The template image is in the first position. Position coordinates on each spectral channel The pixel intensity value at each location. The overall intensity difference value of all pixels within each local image block is statistically analyzed, and the mean and variance of all overall intensity difference values ​​are calculated. The mean is used as the element value of the pixel intensity difference distribution map of the local image block region. Data comparison shows that the element value of the poor solder joint area in the pixel intensity difference distribution map exceeds 0.8, while the element value of the normal area is lower than 0.2.

[0027] Optionally, for each pair of local image patches in the multi-channel fused image and the template image, calculate their correlation coefficient and contrast difference in the gray-level co-occurrence matrix features, and use the combined measure of the correlation coefficient and contrast difference as the corresponding element value of the texture structure similarity matrix. The combined measure adopts a linear weighted sum form of texture structure similarity. ,in, This represents the weighted coefficient of the correlation coefficient in the features of the gray-level co-occurrence matrix. This represents the weighting coefficient for contrast differences in the features of the gray-level co-occurrence matrix. and These are pre-set positive weighting parameters, and their ratio can be adjusted according to the actual needs of printed circuit board defect detection. It is the correlation coefficient. It refers to the contrast difference. This can be understood as the gray-level co-occurrence matrix being calculated and averaged based on four directions (0 degrees, 45 degrees, 90 degrees, and 135 degrees) to achieve rotation invariance, with the correlation coefficient ranging from -1 to 1, while the contrast difference is a non-negative value.

[0028] In specific implementations, the pixel intensity difference distribution map and the texture structure similarity matrix are calculated in parallel. The pixel intensity difference distribution map highlights areas of intensity aberration, while the texture structure similarity matrix highlights areas of structural aberration. Data comparison shows that coating defects exhibit high values ​​in the pixel intensity difference distribution map but low values ​​in the texture structure similarity matrix. In some embodiments, the image registration algorithm uses the RANSAC algorithm to remove mismatched feature points. The affine transformation model includes six parameters: horizontal scaling factor, horizontal tilt factor, horizontal translation amount, vertical tilt factor, vertical scaling factor, and vertical translation amount. These six parameters can accurately describe rotation, scaling, shearing, and translation transformations.

[0029] Optionally, the pixel intensity values ​​of each spectral channel in the 3D data array of the multi-channel fused image are normalized to the range of 0 to 1 to eliminate illumination differences. It is understood that the division of local image patches uses a non-overlapping method to avoid redundant computation; the choice of local image patch size affects the feature granularity—smaller sizes improve positioning accuracy but increase computational burden.

[0030] In one embodiment of this application, data comparison shows that a scratch defect with a length of 200 micrometers on a printed circuit board causes significant intensity changes in the pixel intensity difference distribution map and moderate texture perturbation in the texture structure similarity matrix. The shallow edge response layer is composed of multiple edge detection operators of different scales connected in parallel. These edge detection operators include the Sobel operator and the Laplacian operator. The Sobel operator uses a 3×3 horizontal and vertical convolution kernel, while the Laplacian operator uses a 5×5 second-order differential convolution kernel. The pixel intensity difference distribution map is input to the shallow edge response layer, where each edge detection operator independently performs convolution operations to extract edge gradient and direction information at different scales. For example, the 3×3 Sobel operator is sensitive to subtle intensity steps, while the 5×5 Laplacian operator is sensitive to larger-scale intensity abrupt changes. The outputs of each edge detection operator are weighted and fused to generate an edge discontinuity feature map emphasizing the discontinuities of the difference regions. The weighted fusion operation is achieved by performing a pixel-level linear combination of the convolution outputs at different scales. The combination formula is expressed as:

[0031] in: This represents the final generated edge discontinuity feature map. This represents the total number of edge detection operators participating in the fusion. Indicates the first The fusion weight coefficients of the output feature maps of each edge detection operator. Indicates the first The feature map output after convolution operation of each edge detection operator is fused with weight coefficients. The value is preset and allocated based on the contribution of different operators to the detection of defect edges.

[0032] In some embodiments, the deep semantic abstraction layer consists of multiple cascaded convolutional and pooling layers. For example, the network structure may include three convolutional layers and two max-pooling layers arranged alternately. Each convolutional layer uses a 3×3 convolutional kernel followed by a ReLU activation function. The texture structure similarity matrix is ​​input to the deep semantic abstraction layer. Within the deep semantic abstraction layer, the texture structure similarity matrix undergoes multiple nonlinear transformations and feature dimensionality reduction to gradually abstract high-dimensional feature representations associated with typical defect patterns. The first convolutional layer extracts local texture patterns, while subsequent layers gradually combine these patterns to form more global semantic features. The final layer of the deep semantic abstraction layer outputs the high-dimensional feature representation as a feature map, i.e., an anomaly pattern feature map. Each pixel position in the anomaly pattern feature map is mapped to a high-dimensional feature vector to characterize the semantic anomaly attributes of the corresponding original image region.

[0033] In practice, the pixel intensity difference distribution map and texture structure similarity matrix are processed by the shallow edge response layer and the deep semantic abstraction layer, respectively. Data comparison shows that scratch defects present a clear linear high response band in the edge discontinuity feature map, while they present a diffuse medium response area in the abnormal pattern feature map.

[0034] Optionally, the Sobel operator output in the shallow edge response layer includes a horizontal gradient map and a vertical gradient map. These are combined into a single gradient magnitude map through a square root operation and then subjected to weighted fusion to generate the final feature map used to emphasize the discontinuity of the defect edge. This weighted fusion operation is achieved by performing a pixel-level linear combination of the outputs from different processing paths. Its general calculation formula can be expressed as:

[0035] in: Indicates the pixel position The value of the fused feature map generated at that location. This represents the total number of feature maps participating in the weighted fusion. Indicates the first Input feature maps The corresponding preset or learned fusion weight coefficients, Indicates the first Each input feature map at location The pixel value at that location.

[0036] It is understandable that the convolutional layers of the deep semantic abstraction layer learn a set of filter weights through training. These filter weights can produce specific responses to different types of abnormal patterns in the texture structure similarity matrix, such as spots, cracks, or stains.

[0037] In some embodiments, the edge detection operator may further include a Prewitt operator or a Canny operator edge detector, with different operators integrated in a parallel branching manner in the shallow edge response layer. Optionally, the pooling operation in the deep semantic abstraction layer adopts 2×2 max pooling with a stride of 2, which gradually reduces the feature map space size while preserving significant texture features.

[0038] See Figure 3 This is a multispectral reflectance intensity curve of different defect types and normal areas at different wavelengths. It is a key visualization result used for spectral consistency determination in the detection of micro-defects in electronic components.

[0039] The spectral curves of scratches and solder joint defects differ significantly from those of normal areas, each possessing a unique band response mode, which can be used to distinguish different defect types. When the multispectral reflectance curve of a defective area matches the standard spectral mode of the corresponding type with a degree exceeding a threshold, it can be confirmed as that type of defect; otherwise, it is marked as suspected or excluded. The spectral fluctuations of stain defects are large and irregular, and can be filtered or re-examined based on low match with the standard mode. By analyzing the spectral response ranges of different defects, the selection of the light source and camera parameters of the multispectral imaging system can be optimized in a targeted manner, allowing the inspection to focus more on key information and improving efficiency.

[0040] In one embodiment of this application, data comparison shows that after geometric morphology filtering, the number of candidate defect regions decreased from 50 in the initial defect heatmap to 12 in the refined defect region list. Typical geometric morphology descriptions for each type of defect are read one by one from the morphology rule library. These descriptions include aspect ratio range, area threshold, boundary curvature features, and principal axis direction. For example, the typical geometric morphology description for a scratch defect includes an aspect ratio greater than 5, an area between 100 and 5000 square pixels, a gentle change in boundary curvature, and a principal axis direction parallel to the circuit path.

[0041] On the preliminary defect heatmap, a connected component analysis algorithm is used to extract the contours of all potential defect regions whose anomaly intensities exceed a threshold of 0.7. The connected component analysis employs an 8-neighborhood connection method to determine the boundaries of each consecutive anomalous pixel region. For each extracted potential defect region contour, its actual geometric morphological parameters are calculated, and these parameters are compared one by one with the typical geometric morphological parameters of each type of defect in the morphological rule library. A long strip region with an aspect ratio of 8.2, an area of ​​1200 square pixels, and a boundary curvature standard deviation of 0.05 has a matching degree of 0.91 with the typical geometric morphological description of a scratch defect. Only potential defect regions whose actual geometric morphological parameters match the typical geometric morphological description of at least one type of defect with a matching degree exceeding a set threshold of 0.75 are retained. Their contour information and the matched defect type assumption are recorded in the refined defect region list. The recorded information includes the set of vertex coordinates of the contour polygon, the matched defect type assumption, and the matching degree score. In some embodiments, the morphological rule base is stored in the form of a structured data table, as shown in Table 1, which includes fields such as defect type, minimum aspect ratio, maximum aspect ratio, minimum area, maximum area, and maximum standard deviation of boundary curvature.

[0042] Table 1: Description of Typical Defect Geometric Morphology

[0043] In practical implementation, for each entry in the list of refining defect areas, its pixel position in the multi-channel fused image is located based on its recorded contour information. For example, the contour information of an entry assumed to be a scratch contains 50 vertex coordinates, which are mapped to a 1024x1024 pixel space in the multi-channel fused image. All pixel data within the contour area of ​​the multi-channel fused image are acquired. Each pixel contains reflection intensity values ​​under multiple wavelength light sources. For example, the scratch contour area contains 300 pixels, each recording reflection intensity values ​​at three wavelengths: 450 nm, 550 nm, and 650 nm. For each refining defect area, the regional average reflection intensity value of all pixels under each specific wavelength light source is calculated. The formula for calculating the average reflection intensity is as follows:

[0044] in: Indicates at wavelength The average reflection intensity of the refining defect region calculated under the light source. This indicates the total number of pixels contained within the outline of the refining defect region. Indicates the first within the contour area Each pixel at wavelength The reflected intensity value recorded under the light source.

[0045] Optionally, when calculating the regional average, only pixels within the contour are used, and boundary pixels can be excluded to avoid mixing with background information. It can be understood that by connecting the calculated regional average values ​​under multiple wavelength light sources in ascending order of wavelength, a curve is plotted with wavelength as the horizontal axis and average reflection intensity as the vertical axis. This curve is the multispectral reflection curve of the refined defect region. Data comparison shows that the multispectral reflection curve of a real scratch defect on the surface of a printed circuit board has a reflection intensity of 0.85 at 450 nm and 0.45 at 650 nm.

[0046] In some embodiments, the matching process of the morphological rule base adopts a weighted scoring mechanism. The deviation between the actual geometric morphological parameters and each constraint condition in the typical geometric morphological description is converted into a deduction item. The deduction items corresponding to all constraints are summarized to form a total deduction. The total matching score is 1 minus the total deduction. Optionally, after extracting the contour, the connected component analysis algorithm also calculates the minimum bounding rectangle, area, perimeter, centroid, and other geometric parameters of the contour for subsequent comparison. It can be understood that when extracting the multispectral reflectance curve for each entry in the refined defect region list, the multi-channel fused image used is a three-dimensional data array that has been registered and aligned.

[0047] See Figure 4This is a geometric parameter constraint map defined in the morphology rule library for typical defects on the surface of electronic components. It serves as the core basis for defect morphology matching and filtering. In the defect detection process, these geometric parameter constraints constitute the core content of the morphology rule library. By calculating the matching degree between the actual geometric parameters of potential defect regions extracted from the preliminary defect heatmap and these rules, abnormal regions that do not conform to any type of defect morphology can be effectively filtered out, significantly reducing the computational load of subsequent spectral analysis and improving detection efficiency and accuracy. The significant differences in geometric parameter constraints among different defects provide a clear classification basis for the algorithm, narrowing the hypothesis space of defect types before spectral analysis. Pre-screening using morphology rules avoids misjudging non-defect texture variations or lighting interference as defects, improving the reliability of detection results.

[0048] The preliminary defect heatmap is compared with a morphological rule library using pattern matching and geometric constraint verification. Abnormal regions that do not conform to any type of defect morphological rule are filtered out, generating a refined defect region list. The specific implementation process of pattern matching is as follows: On the preliminary defect heatmap, a connected component analysis algorithm is used to identify and extract all connected regions whose abnormal intensity exceeds a preset threshold, obtaining the contour information of each region. For each extracted potential defect region contour, its actual geometric morphological parameters are calculated, such as region area, aspect ratio, and boundary curvature. These calculated actual geometric morphological parameters are compared one by one with the predefined typical geometric morphological descriptions for each type of defect in the morphological rule library. If the matching degree between the actual geometric morphological parameters of a region and the typical geometric morphological description of at least one type of defect in the morphological rule library exceeds a set threshold, the region is determined to have passed the preliminary pattern matching, and its contour information and the assumed defect type are recorded in the refined defect region list; otherwise, it is filtered out. Geometric constraint verification is a process of further spatial logic verification of the potential defect regions that have passed the preliminary pattern matching. This verification is based on the known physical structural layout of the electronic component to be tested. For example, spatial constraints can be added to the morphological rule base for specific defect types, such as "poor solder joint" defects can only appear within the solder pad area, and "missing coating" defects should not appear in the substrate area outside the metal traces. During verification, the system reads the coordinate position of the current potential defect area in the image and compares it with the pre-stored component structure template for spatial relationship. If the calculated geometric morphological parameters of a region match the description of a certain type of defect, but its location violates the known physical spatial constraints of that type of defect, then the region is judged to be non-compliant in the geometric constraint verification and is thus filtered out, not entering the refined defect area list.

[0049] In one embodiment of this application, data comparison shows that the multispectral reflectance curve of a defect region assumed to be a scratch has reflectance intensities of 0.82, 0.76, and 0.43 at 450 nm, 550 nm, and 650 nm, respectively, while the standard spectral response curve of a scratch defect in the morphology rule library has reflectance intensities of 0.80, 0.78, and 0.45 at the corresponding wavelengths, respectively. From the morphology rule library, a standard spectral response curve corresponding to the assumed defect type of the current refined defect region is obtained. The standard spectral response curve is a reference curve that defines the trend of standard reflectance intensity with wavelength. The standard spectral response curve is an average spectral characteristic curve obtained by statistical learning from a large number of known defect samples under the same multispectral imaging conditions. The multispectral reflectance curve of the current defect region is aligned with the standard spectral response curve of the same type by sampling points. Sampling point alignment means selecting the same wavelength point sequence on the horizontal axis (wavelength) to ensure that the points compared on the two curves correspond to exactly the same wavelength values. For example, sampling alignment is performed at five wavelength points: 450 nm, 500 nm, 550 nm, 600 nm, and 650 nm. Calculate the difference in reflection intensity between all corresponding sampling points on the two curves, and then calculate the root mean square error of the reflection intensity difference. The formula for calculating the root mean square error is as follows:

[0050] in: This represents the calculated root mean square error. This represents the total number of sampling points aligned on the two curves. The multispectral reflectance curve of the current defect region is shown in the first... The reflection intensity value at each sampling wavelength point The standard spectral response curve is shown in the first... The standard reflection intensity value at each sampling wavelength point. The root mean square error is input into a preset conversion function and mapped to a value within a predetermined range. This value is the spectral fit, and a higher spectral fit value indicates a higher degree of matching between the two curves.

[0051] The preset conversion function is based on the principle of exponential decay of error, which can smoothly map the root mean square error to a matching degree between 0 and 1. The specific formula is as follows:

[0052] Where: S represents the spectral fit, with a value range of (0,1]. The root mean square error (RMSE) is calculated and its value is greater than or equal to 0. k represents the sensitivity constant (k>0), a pre-set positive parameter. The larger the value of k, the higher the spectral fit S. The faster the rate of descent, the stricter the matching criteria. When the two curves are perfectly aligned, ,but This indicates an exact match. With As the S-value increases, it decreases towards 0, indicating a lower degree of matching.

[0053] In practice, a spectral fit confirmation threshold and an anomaly confirmation threshold from the preliminary defect heatmap are set for each defect type. For example, the spectral fit confirmation threshold for scratch defects is set to 0.85, and the anomaly confirmation threshold is set to 0.75. For each entry in the list of refining defect areas, it is determined whether its calculated spectral fit reaches the spectral fit confirmation threshold for its assumed defect type. For example, the calculated spectral fit for a refining defect area assumed to be a scratch is 0.88, exceeding the scratch defect confirmation threshold of 0.85. Simultaneously, the average anomaly level of the refining defect areas is read from the preliminary defect heatmap and determined whether it reaches the anomaly confirmation threshold for the assumed defect type. For example, the average anomaly level of the same scratch defect area in the preliminary defect heatmap is 0.80, exceeding the scratch defect anomaly confirmation threshold of 0.75. Only when the average values ​​of spectral consistency and anomaly severity simultaneously reach or exceed their respective confirmation thresholds is the refining defect area definitively confirmed as a hypothetical defect type and classified as such. For example, the above item was definitively confirmed and classified as a "scratch" defect. If the average values ​​of spectral consistency and anomaly severity fail to simultaneously meet the corresponding anomaly severity confirmation thresholds, the item is removed from the refining defect area list or marked as a suspected defect awaiting re-examination. For example, if an item has a spectral consistency of only 0.70 and an average anomaly severity of 0.90, the item is marked as "awaiting re-examination" and is not included in the final inspection report.

[0054] See Figure 5 This is a comparison chart of multispectral reflectance curves for a scratch defect. The two curves show a high degree of consistency in their overall trends, both maintaining high reflectance intensity at short wavelengths and decreasing significantly at long wavelengths. The slight difference between the measured curve and the standard curve indicates that the spectral characteristics of this defect area are highly consistent with typical scratches. This high degree of consistency is one of the key bases for defect type confirmation. It provides a solid visual and data foundation for subsequent defect confirmation, making the scientific rigor and scientific nature of the technical solution immediately apparent. Different types of defects have unique spectral response patterns. By comparing the measured curve with the standard curve, the possibility of other defect types can be intuitively ruled out. When the detection system makes a misjudgment, by comparing the new measured curve with the standard pattern in this chart, the problem can be quickly located as to whether it lies in spectral acquisition, feature extraction, or the matching algorithm.

[0055] The above are merely preferred embodiments of this application and are not intended to limit this application in any other way. Any person skilled in the art may use the above-disclosed technical content to make changes or modifications to equivalent embodiments and apply them to other fields. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of this application without departing from the content of the technical solution of this application shall still fall within the protection scope of the technical solution of this application.

Claims

1. A method for detecting micro-defects in printed circuit boards based on image recognition, characterized in that, The method includes: A surface multispectral image sequence of the electronic component to be inspected is acquired, and a preliminary defect thermal map containing spatial location and degree of anomaly is generated based on the surface multispectral image sequence. Based on the known physical structure and material properties of the electronic component to be tested, a morphological rule library corresponding to the defect type is established; The preliminary defect heatmap is matched with the morphology rule base by class and geometric constraint verification to filter out abnormal areas that do not conform to any type of defect morphology rule, and a refined defect area list is generated. For each defect region in the refined defect region list, the corresponding multispectral reflectance curve is extracted from the multichannel fused image, and the spectral fit is calculated by combining the spectral response mode of the corresponding type of defect in the morphological rule library. Based on the calculation results of spectral fit and the degree of anomaly of the corresponding area in the preliminary defect heat map, each defect in the list of refining defect areas is finally confirmed and classified, and a test report containing defect type, coordinates and quantified severity level is output.

2. The method for detecting micro-defects in printed circuit boards based on image recognition according to claim 1, characterized in that, The generation of a preliminary defect thermal map containing spatial location and anomaly degree based on the surface multispectral image sequence includes: Grayscale images of the surface of the electronic component under test are acquired under multiple monochromatic light sources of different wavelengths to form a multispectral image sequence of the surface. The surface multispectral image sequence is registered in the same region and aligned with pixels to synthesize a multi-channel fused image of the electronic component to be detected. Each pixel in the multi-channel fused image contains reflection intensity information from multiple wavelength light sources. A template image matching the model of the electronic component to be detected is retrieved from the standard component database, and the multi-channel fused image and the template image are normalized in spatial scale and matched with reference feature points. Based on the matching correspondence, calculate the element values ​​of the pixel intensity difference distribution map and the corresponding element values ​​of the texture structure similarity matrix of each local image block in the multi-channel fused image and the corresponding region in the template image; A hierarchical defect feature extraction network is constructed, which includes a shallow edge response layer and a deep semantic abstraction layer. The element values ​​of the pixel intensity difference distribution map and the corresponding element values ​​of the texture structure similarity matrix are input into the hierarchical defect feature extraction network in parallel for multi-scale feature extraction. By fusing the edge discontinuity feature map output by the shallow edge response layer with the abnormal pattern feature map output by the deep semantic abstraction layer, a preliminary defect heatmap containing spatial location and degree of abnormality is generated.

3. The method for detecting micro-defects in printed circuit boards based on image recognition according to claim 2, characterized in that, The step of performing same-region registration and pixel alignment operations on the surface multispectral image sequence to synthesize a multi-channel fused image of the electronic component to be detected includes: The image acquired under the center wavelength light source in the surface multispectral image sequence is selected as the spatial reference image; For each grayscale image acquired under a non-central wavelength light source in the surface multispectral image sequence, an image registration algorithm based on feature point detection and affine transformation model is used to calculate the coordinate transformation parameters between each grayscale image acquired under a non-central wavelength light source in the surface multispectral image sequence and the spatial reference image. The coordinate transformation parameters are used to perform spatial transformation on each grayscale image under a non-central wavelength light source, so that the pixels in each grayscale image under a non-central wavelength light source are aligned one by one with the pixels in the spatial reference image in spatial position. In order of wavelength, grayscale images of all wavelength light sources that have completed pixel alignment are stacked to construct a multi-channel fused image.

4. The method for detecting micro-defects in printed circuit boards based on image recognition according to claim 3, characterized in that, The calculation of the element values ​​of the pixel intensity difference distribution map between each local image patch in the multi-channel fused image and the corresponding region in the template image, and the corresponding element values ​​of the texture structure similarity matrix, includes: The normalized multi-channel fused image and the template image are evenly divided into multiple local image blocks of the same size that do not overlap. For each pair of local image blocks corresponding to spatial locations, the absolute intensity difference in different spectral channels is calculated pixel by pixel as the spectral channel difference, and the average value of multiple spectral channel differences is taken as the comprehensive intensity difference value of the pixel. The combined intensity difference value of all pixels within each local image block is statistically analyzed, and the mean of all combined intensity difference values ​​is calculated as the element value of the pixel intensity difference distribution map of the local image block region. Meanwhile, for each pair of local image patches, the correlation coefficient and contrast difference on the gray-level co-occurrence matrix features are calculated, and the combined measure of the correlation coefficient and contrast difference is used as the corresponding element value of the texture structure similarity matrix.

5. The method for detecting micro-defects in printed circuit boards based on image recognition according to claim 2, characterized in that, The hierarchical defect feature extraction network is constructed, which includes a shallow edge response layer and a deep semantic abstraction layer. The element values ​​of the pixel intensity difference distribution map and the corresponding element values ​​of the texture structure similarity matrix are input in parallel into the hierarchical defect feature extraction network for multi-scale feature extraction, including: The shallow edge response layer is composed of multiple edge detection operators of different scales connected in parallel, including the Sobel operator and the Laplacian operator; The element values ​​of the pixel intensity difference distribution map are input to the shallow edge response layer, and each edge detection operator performs convolution operation independently to extract edge gradient and direction information at different scales; The edge gradient and direction information output by each edge detection operator are weighted and fused to generate the edge discontinuity feature map that emphasizes the discontinuity of edges in the difference region. The corresponding element values ​​of the texture structure similarity matrix undergo multiple nonlinear transformations and feature dimensionality reduction in the deep semantic abstraction layer, gradually abstracting high-dimensional feature expressions associated with typical defect patterns. The last layer of the deep semantic abstraction layer outputs the high-dimensional feature representation as a feature map, namely the abnormal pattern feature map.

6. The method for detecting micro-defects in printed circuit boards based on image recognition according to claim 1, characterized in that, The process involves performing class-by-class pattern matching and geometric constraint verification between the preliminary defect heatmap and the morphological rule base, filtering out abnormal regions that do not conform to any type of defect morphological rule, and generating a refined defect region list, including: The typical geometric shape description of each type of defect is read one by one from the morphological rule library. The typical geometric shape description includes aspect ratio range, area threshold, boundary curvature features and principal axis direction. On the preliminary defect heatmap, the contours of all potential defect regions with abnormal intensity exceeding the threshold are extracted using a connected component analysis algorithm; For each extracted potential defect region contour, its actual geometric morphological parameters are calculated, and the actual geometric morphological parameters are compared one by one with the typical geometric morphological description of each type of defect in the morphological rule library. Only the potential defect areas whose actual geometric parameters match the typical geometric description with a set threshold are retained, and their contour information and the matching defect type assumptions are recorded in the refined defect area list.

7. The method for detecting micro-defects in printed circuit boards based on image recognition according to claim 6, characterized in that, For each defect region in the refined defect region list, the corresponding multispectral reflectance curve is extracted from the multichannel fused image, including: For each entry in the list of refining defect areas, the pixel position of the refining defect area in the multi-channel fused image is located based on the contour information recorded for each entry. Acquire all pixel data of the multi-channel fused image within the contour region, where each pixel contains the reflection intensity value under all acquired wavelength light sources; For the refined defect area, calculate the regional average value of the reflection intensity of all pixels under each specific wavelength light source; The calculated regional average values ​​under multiple wavelength light sources are connected sequentially in order of wavelength from shortest to longest, and plotted as a curve with wavelength as the horizontal axis and average reflection intensity as the vertical axis. This curve is the multispectral reflection curve of the refining defect area.

8. The method for detecting micro-defects in printed circuit boards based on image recognition according to claim 7, characterized in that, The calculation of spectral fit, which combines the spectral response patterns of corresponding defect types in the morphological rule base, includes: Obtain the standard spectral response curve corresponding to the assumed defect type of the current refining defect region from the morphological rule library; Align the sampling points of the multispectral reflectance curve of the current defect region with the standard spectral response curve of the same type. Calculate the difference in reflection intensity between all corresponding sampling points on the two curves, and obtain the root mean square error of the difference in reflection intensity. The root mean square error is input into a preset conversion function to obtain the spectral fit.

9. The method for detecting micro-defects in printed circuit boards based on image recognition according to claim 8, characterized in that, Based on the calculation results of spectral fit and the degree of anomaly of the corresponding region in the preliminary defect thermal map, each defect in the refining defect region list is finally confirmed and classified, including: An anomaly level confirmation threshold is preset, and a spectral fit confirmation threshold is set for each defect type; For each entry in the list of refining defect areas, determine whether the calculated spectral fit reaches the confirmation threshold of the spectral fit. At the same time, the average value of the abnormality of the refining defect area is read from the preliminary defect heat map to determine whether it reaches the abnormality confirmation threshold of the assumed defect type. Only when the average value of the spectral fit and the degree of anomaly simultaneously reaches or exceeds their respective confirmation thresholds is the refining defect region finally confirmed as the hypothetical defect type, and it is classified and labeled as the hypothetical defect type. If the average value of the spectral consistency and the degree of anomaly fails to simultaneously reach or exceed their respective confirmation thresholds, the entry will be removed from the list of refining defect areas or marked as a suspected defect to be reviewed.

10. A micro-defect detection system for printed circuit boards based on image recognition, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the image recognition-based micro-defect detection method for printed circuit boards as described in any one of claims 1 to 9.

Images